[OpenAMP] 嘗試在 Linux user space 執行特定 cpu 的 bare-matal application

發表於

前言

這個標題可能有點繞口，但主要的動作就是，我們先關閉一顆 cpu，之後我們去 trace smp 相關的程式
之後針對 smp 的啟動程序去執行特定的位址，達到該 cpu 啟動後，會自動執行 baremetal application 的成果

實驗步驟

首先，我們先分析一下我們 dump 出來的運作流程
我們將 message 塞入 platsmp.c

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
[    0.126278] rockchip_smp_prepare_cpus: disable core1
[    0.126314] pmu_set_power_domain: rstc address: vir=ee88f7c0
[    0.126327] pmu_set_power_domain:(assert) status=ffffffda
[    0.126353] pmu_set_power_domain: has_pmu!, pmu addr-> vir:ee802a00, PMU_PWRDN_CON=8, result=0x00000002
[    0.126366] pmu_set_power_domain: pmu address: vir=ee802a00
[    0.126375] pmu_set_power_domain: IS_ERR(rtsc) entered
[    0.126385] rockchip_smp_prepare_cpus: disable core2
[    0.126415] pmu_set_power_domain: rstc address: vir=ee88f7c0
[    0.126427] pmu_set_power_domain:(assert) status=ffffffda
[    0.126444] pmu_set_power_domain: has_pmu!, pmu addr-> vir:ee802a00, PMU_PWRDN_CON=8, result=0x00000006
[    0.126457] pmu_set_power_domain: pmu address: vir=ee802a00
[    0.126466] pmu_set_power_domain: IS_ERR(rtsc) entered
[    0.126475] rockchip_smp_prepare_cpus: disable core3
[    0.126505] pmu_set_power_domain: rstc address: vir=ee88f7c0
[    0.126516] pmu_set_power_domain:(assert) status=ffffffda
[    0.126533] pmu_set_power_domain: has_pmu!, pmu addr-> vir:ee802a00, PMU_PWRDN_CON=8, result=0x0000000e
[    0.126545] pmu_set_power_domain: pmu address: vir=ee802a00
[    0.126554] pmu_set_power_domain: IS_ERR(rtsc) entered
[    0.126640] Setting up static identity map for 0x100000 - 0x100058
[    0.128998] rockchip_boot_secondary: cpu core1 start !!
[    0.129047] pmu_set_power_domain: rstc address: vir=ee89b2c0
[    0.129072] pmu_set_power_domain: has_pmu!, pmu addr-> vir:ee802a00, PMU_PWRDN_CON=8, result=0x0000000c
[    0.129085] pmu_set_power_domain: pmu address: vir=ee802a00
[    0.129094] pmu_set_power_domain: IS_ERR(rtsc) entered
[    0.129105] pmu_set_power_domain:(dessert) status=ffffffda
[    0.130295] CPU1: update cpu_capacity 430
[    0.130304] CPU1: thread -1, cpu 1, socket 5, mpidr 80000501
[    0.131072] rockchip_boot_secondary: cpu core2 start !!
[    0.131127] pmu_set_power_domain: rstc address: vir=ee89b640
[    0.131150] pmu_set_power_domain: has_pmu!, pmu addr-> vir:ee802a00, PMU_PWRDN_CON=8, result=0x00000008
[    0.131163] pmu_set_power_domain: pmu address: vir=ee802a00
[    0.131172] pmu_set_power_domain: IS_ERR(rtsc) entered
[    0.131183] pmu_set_power_domain:(dessert) status=ffffffda
[    0.132366] CPU2: update cpu_capacity 430
[    0.132375] CPU2: thread -1, cpu 2, socket 5, mpidr 80000502
[    0.133132] rockchip_boot_secondary: cpu core3 start !!
[    0.133188] pmu_set_power_domain: rstc address: vir=ee89b9c0
[    0.133211] pmu_set_power_domain: has_pmu!, pmu addr-> vir:ee802a00, PMU_PWRDN_CON=8, result=0x00000000
[    0.133224] pmu_set_power_domain: pmu address: vir=ee802a00
[    0.133233] pmu_set_power_domain: IS_ERR(rtsc) entered
[    0.133244] pmu_set_power_domain:(dessert) status=ffffffda
[    0.134427] CPU3: update cpu_capacity 430
[    0.134436] CPU3: thread -1, cpu 3, socket 5, mpidr 80000503
[    0.134566] Brought up 4 CPUs
[    0.134589] SMP: Total of 4 processors activated (192.00 BogoMIPS).
[    0.134599] CPU: All CPU(s) started in SVC mode.
[    0.136426] devtmpfs: initialized
[    0.158885] VFP support v0.3: implementor 41 architecture 3 part 30 variant d rev 0
[    0.159394] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.159430] futex hash table entries: 1024 (order: 4, 65536 bytes)
[    0.160754] pinctrl core: initialized pinctrl subsystem
[    0.162142] NET: Registered protocol family 16
[    0.164101] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.185908] cpuidle: using governor ladder
[    0.215947] cpuidle: using governor menu

  • rockchip_smp_prepare_cpus(): 這一部份是將 cpu disable，會執行 pmu_set_power_domain(cpu, false) 將除了 0 以外的 core 關閉

    1
    2
    3
    -> pmu_set_power_domain(1, false);
    -> pmu_set_power_domain(2, false);
    -> pmu_set_power_domain(3, false);

  • pmu_set_power_domain(int cpu , int on): 這部份是將 power domain 關閉或是開啟

    1
    2
    3
    4
    5
    6
    7
    8
    -> disable cpu N:
        -> assert reset control register (cpu3: set CRU_SOFTRST0_CON bit 3 to 1)
        -> set PMU_PWRDN_CON bit 1 2 3 to 1, 1 is off (0x0 -> 0x2 -> 0x6 -> 0xe)
        -> free reset control register address
    -> enable cpu N:
        -> set PMU_PWRDN_CON bit 1 2 3 to 0, 0 is on  (0xe -> 0xc -> 0x8 -> 0x0)
        -> disassert reset control register (cpu3: set CRU_SOFTRST0_CON bit 3 to 0)
        -> free reset control register address

  • rockchip_boot_secondary(): enable cpu N 並且將 boot entry 設定到指定位址

    1
    2
    3
    -> pmu_set_power_domain(N, true);
    -> sram_base + 8 = pc
    -> sram_base + 4 = DEADBEAF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
static int rockchip_boot_secondary(unsigned int cpu, struct task_struct *idle)
{
        /* ... get pmu base addresss */

        ret = pmu_set_power_domain(0 + cpu, true);
        if (ret < 0)
                return ret;

        if (read_cpuid_part() != ARM_CPU_PART_CORTEX_A9) {
                /*
                 * We communicate with the bootrom to active the cpus other
                 * than cpu0, after a blob of initialize code, they will
                 * stay at wfe state, once they are actived, they will check
                 * the mailbox:
                 * sram_base_addr + 4: 0xdeadbeaf
                 * sram_base_addr + 8: start address for pc
                 * The cpu0 need to wait the other cpus other than cpu0 entering
                 * the wfe state.The wait time is affected by many aspects.
                 * (e.g: cpu frequency, bootrom frequency, sram frequency, ...)
                 */
                mdelay(1); /* ensure the cpus other than cpu0 to startup */

                writel(virt_to_phys(secondary_startup), sram_base_addr + 8);
                writel(0xDEADBEAF, sram_base_addr + 4);
                dsb_sev();
        }
}

修改 baremetal code

然後我們修改一下 code

修改位址與 makefile

首先，我們使用的是 Firefly-RK3288 ，提供了 2G DDR 的空間，也就是 0x00000000 - 0x80000000

為了防止 kernel remap 我們無法預知的位址，因此我們在 uboot 中加入了 mem=960m 表示我們只提供 960M 的 memory 空間給 kernel 使用
也就是 0x00000000 - 0x3c0000000 使 kernel 可能會用到的位址
因此我們也將 entry point 修改成 0x40000000 避免 kernel 使用，但是我們還是可以透過 devmem 來寫入

備註：

1
2
3
以上是在 firefly RK3288 的公版上進行設置的，之後 HW 所提供的 DDR size 可能只有 1G
也就是從到 0x40000000 而已，如果使用以上設定會造成資料無法存取的狀況，因此之後要運行 application 的話
應該是要將存取位址設置成 0x3c000000

loadmetal 加入 RK3288 支援

沒有改什麼大部分的內容，但因為要使用 hardfloat compiler 才可以在 rootfs 使用，因此我們就照此辦理

1
2
source /usr/local/weintek-i686/environment-setup-armv7ahf-vfp-neon-weintek-linux-gnueabi
arm-weintek-linux-gnueabi-gcc -march=armv7-a -mfloat-abi=hard -mfpu=neon --sysroot=$SDKTARGETSYSROOT -o loadmetal main.c

這樣就可以 build 出可使用的 loadmetal binary
若要使用，請將 #define LOAD_RK3288 打開

Source code of loadmetal

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/mman.h>

#define LOAD_RK3288

#ifdef LOAD_RK3288
#define MEMADDR 0x3c000000
#else
#define MEMADDR 0x10000000
#endif

int main (int argc, char * argv [])
{
        if (argc != 2) {
                printf("Usage: %s xxx.elf(xxx.bin)\n", argv[0]);
                return 0;
        }

        char *filename = argv[1];

        printf("Open file %s\n", filename);

        FILE *file = fopen(argv[1],"rb");

        /* Get file length */
        fseek(file, 0, SEEK_END);
        unsigned long filelen = ftell(file);
        fseek(file, 0, SEEK_SET);
        printf("File lenght %d\n", filelen);

        /* Map Physical address of RAM to virtual address segment with Read/Write Access */
        printf("Open memory %x to read data\n", MEMADDR);
        int fd = open("/dev/mem", O_RDWR);
        void *address = mmap(NULL, filelen,PROT_READ|PROT_WRITE, MAP_SHARED, fd, MEMADDR);

        /* Read file contents */
        fread(address, filelen, 1, file);
        fclose(file);

        printf("Read file %s with length 0x%x to memory address 0x%x done!!\n", filename, filelen ,MEMADDR);
}

問題：open(“/dev/mem”) 會失敗: permission denied

原因是因為 CONFIG_STRICT_DEVMEM 被開啟，結果 user space 就只能access reserver memory 和kernel 使用的 system Ram 兩種
所以透過 menu -> Kernel Hacking 把 config 關閉後就可以了

這邊忘記針對 GPIO8 clock 進行設定，設定如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
void bootmain(void)
{
        /* This application for Firefly RK3288 LED toggle example */

        /* Setting LEDs
         *  WORK_LED : GPIO8_A1
         * POWER_LED : GPIO8_A2
         */

        /* set GRF_GPIO8A_IOMUX (default: gpio) */
        rk_clrreg(GRF_GPIO8A_IOMUX, (BIT(3) | BIT(2)));

        /* set pull-up GPIO8_A (defult: pull-up)
         * GPIO8_A1(bit3, bit2) = (0, 1)
         * GPIO8_A2(bit5, bit4) = (0, 1)
         */
        rk_clrsetreg(GRF_GPIO8A_P, (BIT(3)), (BIT(2)));

        /* set GPIO8 clock */
        rk_clrreg(CRU_CLKGATE14_CON, BIT(8));

        /* set GPIO8_A1 direction (default: input), 0:input, 1:output */
        rk_setreg(GPIO8_DIRECTION, (BIT(1)));

        /* set GPIO8_A1 data (default: 0x00000000)
         * Accroding to pull-up pin :
         *                            0: on
         *                            1: off
         */
}

補充：透過 uboot boot command maxcpus=3 進行測試

我們修改 include/config_distro_bootcmd.h

1
2
3
4
5
6
7
8
9
10
diff --git a/include/config_distro_bootcmd.h b/include/config_distro_bootcmd.h
index c0dcb4e..c09ddad 100644
--- a/include/config_distro_bootcmd.h
+++ b/include/config_distro_bootcmd.h
@@ -70,7 +70,7 @@

 #define BOOTENV_SHARED_SD \
        "sd_boot= "     \
-       "env set bootargs \"earlyprintk console=ttyS2,115200n8 loglevel=7 root=/dev/mmcblk0p2 rw rootfstype=ext4 rootwait\"; "  \
+       "env set bootargs \"earlyprintk console=ttyS2,115200n8 loglevel=7 root=/dev/mmcblk0p2 rw rootfstype=ext4 rootwait mem=960m maxcpus=3\"; "       \

之後進系統先查看 cpu 狀態

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
cat /proc/cpuinfo

....省略

processor    : 2
model name    : ARMv7 Processor rev 1 (v7l)
BogoMIPS    : 12.09
Features    : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm 
CPU implementer    : 0x41
CPU architecture: 7
CPU variant    : 0x0
CPU part    : 0xc0d
CPU revision    : 1

Hardware    : Rockchip (Device Tree)
Revision    : 0000
Serial        : 0000000000000000

可以看到 kernel 只啟動到 cpu 2
之後執行 loadmetal + insmod