blob: c8e2405cc0e242b34b18de326624e8e4fab73fbe [file] [log] [blame]
Dan Handley610e7e12018-03-01 18:44:00 +00001Trusted Firmware-A for Raspberry Pi 3
2=====================================
Antonio Nino Diaz8869b482017-12-01 11:11:26 +00003
4.. section-numbering::
5 :suffix: .
6
7.. contents::
8
9The `Raspberry Pi 3`_ is an inexpensive single-board computer that contains four
Dan Handley610e7e12018-03-01 18:44:00 +000010Arm Cortex-A53 cores, which makes it possible to have a port of Trusted
11Firmware-A (TF-A).
Antonio Nino Diaz8869b482017-12-01 11:11:26 +000012
Dan Handley610e7e12018-03-01 18:44:00 +000013The following instructions explain how to use this port of the TF-A with the
14default distribution of `Raspbian`_ because that's the distribution officially
15supported by the Raspberry Pi Foundation. At the moment of writing this, the
16officially supported kernel is a AArch32 kernel. This doesn't mean that this
17port of TF-A can't boot a AArch64 kernel. The `Linux tree fork`_ maintained by
18the Foundation can be compiled for AArch64 by following the steps in
19`AArch64 kernel build instructions`_.
Antonio Nino Diaz8869b482017-12-01 11:11:26 +000020
21**IMPORTANT NOTE**: This port isn't secure. All of the memory used is DRAM,
22which is available from both the Non-secure and Secure worlds. This port
23shouldn't be considered more than a prototype to play with and implement
24elements like PSCI to support the Linux kernel.
25
26Design
27------
28
29The SoC used by the Raspberry Pi 3 is the Broadcom BCM2837. It is a SoC with a
30VideoCore IV that acts as primary processor (and loads everything from the SD
31card) and is located between all Arm cores and the DRAM. Check the `Raspberry Pi
323 documentation`_ for more information.
33
34This explains why it is possible to change the execution state (AArch64/AArch32)
35depending on a few files on the SD card. We only care about the cases in which
36the cores boot in AArch64 mode.
37
38The rules are simple:
39
40- If a file called ``kernel8.img`` is located on the ``boot`` partition of the
41 SD card, it will load it and execute in EL2 in AArch64. Basically, it executes
42 a `default AArch64 stub`_ at address **0x0** that jumps to the kernel.
43
44- If there is also a file called ``armstub8.bin``, it will load it at address
45 **0x0** (instead of the default stub) and execute it in EL3 in AArch64. All
46 the cores are powered on at the same time and start at address **0x0**.
47
48This means that we can use the default AArch32 kernel provided in the official
Dan Handley610e7e12018-03-01 18:44:00 +000049`Raspbian`_ distribution by renaming it to ``kernel8.img``, while TF-A and
50anything else we need is in ``armstub8.bin``. This way we can forget about the
51default bootstrap code. When using a AArch64 kernel, it is only needed to make
52sure that the name on the SD card is ``kernel8.img``.
Antonio Nino Diaz8869b482017-12-01 11:11:26 +000053
54Ideally, we want to load the kernel and have all cores available, which means
55that we need to make the secondary cores work in the way the kernel expects, as
56explained in `Secondary cores`_. In practice, a small bootstrap is needed
Dan Handley610e7e12018-03-01 18:44:00 +000057between TF-A and the kernel.
Antonio Nino Diaz8869b482017-12-01 11:11:26 +000058
59To get the most out of a AArch32 kernel, we want to boot it in Hypervisor mode
60in AArch32. This means that BL33 can't be in EL2 in AArch64 mode. The
61architecture specifies that AArch32 Hypervisor mode isn't present when AArch64
62is used for EL2. When using a AArch64 kernel, it should simply start in EL2.
63
64Placement of images
65~~~~~~~~~~~~~~~~~~~
66
67The file ``armstub8.bin`` contains BL1 and the FIP. It is needed to add padding
68between them so that the addresses they are loaded to match the ones specified
Dan Handley610e7e12018-03-01 18:44:00 +000069when compiling TF-A.
Antonio Nino Diaz8869b482017-12-01 11:11:26 +000070
71The device tree block is loaded by the VideoCore loader from an appropriate
72file, but we can specify the address it is loaded to in ``config.txt``.
73
74The file ``kernel8.img`` contains a kernel image that is loaded to the address
75specified in ``config.txt``. The `Linux kernel tree`_ has information about how
76a AArch32 Linux kernel image is loaded in ``Documentation/arm/Booting``:
77
78::
79
80 The zImage may also be placed in system RAM and called there. The
81 kernel should be placed in the first 128MiB of RAM. It is recommended
82 that it is loaded above 32MiB in order to avoid the need to relocate
83 prior to decompression, which will make the boot process slightly
84 faster.
85
86There are no similar restrictions for AArch64 kernels, as specified in the file
87``Documentation/arm64/booting.txt``.
88
89This means that we need to avoid the first 128 MiB of RAM when placing the
Dan Handley610e7e12018-03-01 18:44:00 +000090TF-A images (and specially the first 32 MiB, as they are directly used to
91place the uncompressed AArch32 kernel image. This way, both AArch32 and
Antonio Nino Diaz8869b482017-12-01 11:11:26 +000092AArch64 kernels can be placed at the same address.
93
94In the end, the images look like the following diagram when placed in memory.
95All addresses are Physical Addresses from the point of view of the Arm cores.
96Again, note that this is all just part of the same DRAM that goes from
97**0x00000000** to **0x3F000000**, it just has different names to simulate a real
98secure platform!
99
100::
101
102 0x00000000 +-----------------+
103 | ROM | BL1
Ying-Chun Liu (PaulLiu)51b146f2018-07-04 02:32:27 +0800104 0x00020000 +-----------------+
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000105 | FIP |
106 0x00200000 +-----------------+
107 | |
108 | ... |
109 | |
110 0x01000000 +-----------------+
111 | Kernel |
112 +-----------------+
113 | |
114 | ... |
115 | |
116 0x02000000 +-----------------+
117 | DTB |
118 +-----------------+
119 | |
120 | ... |
121 | |
122 0x10000000 +-----------------+
123 | Secure SRAM | BL2, BL31
124 0x10100000 +-----------------+
Ying-Chun Liu (PaulLiu)64c6ff12018-06-13 20:53:08 +0800125 | Secure DRAM | BL32 (Secure payload)
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000126 0x10300000 +-----------------+
127 | Non-secure DRAM | BL33
128 0x11000000 +-----------------+
129 | |
130 | ... |
131 | |
132 0x3F000000 +-----------------+
133 | I/O |
134 0x40000000 +-----------------+
135
136The area between **0x10000000** and **0x11000000** has to be protected so that
Antonio Nino Diaz73ecb9f2018-06-13 13:34:46 +0100137the kernel doesn't use it. That is done by adding ``memmap=16M$256M`` to the
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000138command line passed to the kernel. See the `Setup SD card`_ instructions to see
139how to do it.
140
141The last 16 MiB of DRAM can only be accessed by the VideoCore, that has
142different mappings than the Arm cores in which the I/O addresses don't overlap
143the DRAM. The memory reserved to be used by the VideoCore is always placed at
144the end of the DRAM, so this space isn't wasted.
145
Dan Handley610e7e12018-03-01 18:44:00 +0000146Considering the 128 MiB allocated to the GPU and the 16 MiB allocated for
147TF-A, there are 880 MiB available for Linux.
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000148
149Boot sequence
150~~~~~~~~~~~~~
151
Dan Handley610e7e12018-03-01 18:44:00 +0000152The boot sequence of TF-A is the usual one except when booting an AArch32
153kernel. In that case, BL33 is booted in AArch32 Hypervisor mode so that it
154can jump to the kernel in the same mode and let it take over that privilege
155level. If BL33 was running in EL2 in AArch64 (as in the default bootflow of
156TF-A) it could only jump to the kernel in AArch32 in Supervisor mode.
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000157
158The `Linux kernel tree`_ has instructions on how to jump to the Linux kernel
159in ``Documentation/arm/Booting`` and ``Documentation/arm64/booting.txt``. The
160bootstrap should take care of this.
161
162Secondary cores
163~~~~~~~~~~~~~~~
164
165The kernel used by `Raspbian`_ doesn't have support for PSCI, so it is needed to
166use mailboxes to trap the secondary cores until they are ready to jump to the
167kernel. This mailbox is located at a different address in the AArch32 default
168kernel than in the AArch64 kernel.
169
Dan Handley610e7e12018-03-01 18:44:00 +0000170Also, this port of TF-A has another Trusted Mailbox in Shared BL RAM. During
171cold boot, all secondary cores wait in a loop until they are given given an
172address to jump to in this Mailbox (``bl31_warm_entrypoint``).
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000173
174Once BL31 has finished and the primary core has jumped to the BL33 payload, it
175has to call ``PSCI_CPU_ON`` to release the secondary CPUs from the wait loop.
176The payload then makes them wait in another waitloop listening from messages
177from the kernel. When the primary CPU jumps into the kernel, it will send an
178address to the mailbox so that the secondary CPUs jump to it and are recognised
179by the kernel.
180
181Build Instructions
182------------------
183
184To boot a AArch64 kernel, only the AArch64 toolchain is required.
185
186To boot a AArch32 kernel, both AArch64 and AArch32 toolchains are required. The
187AArch32 toolchain is needed for the AArch32 bootstrap needed to load a 32-bit
188kernel.
189
Dan Handley610e7e12018-03-01 18:44:00 +0000190First, clone and compile `Raspberry Pi 3 TF-A bootstrap`_. Choose the one
191needed for the architecture of your kernel.
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000192
Dan Handley610e7e12018-03-01 18:44:00 +0000193Then compile TF-A. For a AArch32 kernel, use the following command line:
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000194
195.. code:: shell
196
197 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \
198 RPI3_BL33_IN_AARCH32=1 \
199 BL33=../rpi3-arm-tf-bootstrap/aarch32/el2-bootstrap.bin \
200 all fip
201
202For a AArch64 kernel, use this other command line:
203
204.. code:: shell
205
206 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \
207 BL33=../rpi3-arm-tf-bootstrap/aarch64/el2-bootstrap.bin \
208 all fip
209
210Then, join BL1 and the FIP with the following instructions (replace ``release``
211by ``debug`` if you set the build option ``DEBUG=1``):
212
213.. code:: shell
214
215 cp build/rpi3/release/bl1.bin bl1.pad.bin
Ying-Chun Liu (PaulLiu)51b146f2018-07-04 02:32:27 +0800216 truncate --size=131072 bl1.pad.bin
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000217 cat bl1.pad.bin build/rpi3/release/fip.bin > armstub8.bin
218
219The resulting file, ``armstub8.bin``, contains BL1 and the FIP in the place they
Dan Handley610e7e12018-03-01 18:44:00 +0000220need to be for TF-A to boot correctly. Now, follow the instructions in
221`Setup SD card`_.
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000222
223The following build options are supported:
224
225- ``PRELOADED_BL33_BASE``: Specially useful because the file ``kernel8.img`` can
226 be loaded anywhere by modifying the file ``config.txt``. It doesn't have to
227 contain a kernel, it could have any arbitrary payload.
228
229- ``RESET_TO_BL31``: Set to 1 by default. If using a 32-bit kernel like
230 `Raspbian`_, the space used by BL1 can overwritten by the kernel when it is
231 being loaded. Even when using a AArch64 kernel the region used by
232 BL1 isn't protected and the kernel could overwrite it. The space used by BL31
233 is reserved by the command line passed to the kernel.
234
235- ``RPI3_BL33_IN_AARCH32``: This port can load a AArch64 or AArch32 BL33 image.
Dan Handley610e7e12018-03-01 18:44:00 +0000236 By default this option is 0, which means that TF-A will jump to BL33 in EL2
237 in AArch64 mode. If set to 1, it will jump to BL33 in Hypervisor in AArch32
238 mode.
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000239
Ying-Chun Liu (PaulLiu)64c6ff12018-06-13 20:53:08 +0800240- ``BL32``: This port can load and run OP-TEE. The OP-TEE image is optional.
241 Please use the code from `here <https://github.com/OP-TEE/optee_os>`__.
242 Build the Trusted Firmware with option ``BL32=tee-header_v2.bin
243 BL32_EXTRA1=tee-pager_v2.bin BL32_EXTRA2=tee-pageable_v2.bin``
244 to put the binaries into the FIP.
245
Ying-Chun Liu (PaulLiu)4176e0f2018-07-05 14:55:21 +0800246- ``TRUSTED_BOARD_BOOT``: This port supports TBB. Set this option
247 ``TRUSTED_BOARD_BOOT=1`` to enable it. In order to use TBB, you might
248 want to set ``GENERATE_COT=1`` to let the contents of the FIP automatically
249 signed by the build process. The ROT key will be generated and output to
250 ``rot_key.pem`` in the build directory. It is able to set ROT_KEY to
251 your own key in PEM format.
252 Also in order to build, you need to clone mbedtls from
253 `here <https://github.com/ARMmbed/mbedtls>`__.
254 And set MBEDTLS_DIR to mbedtls source directory.
255
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000256The following is not currently supported:
257
Dan Handley610e7e12018-03-01 18:44:00 +0000258- AArch32 for TF-A itself.
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000259
260- ``EL3_PAYLOAD_BASE``: The reason is that you can already load anything to any
261 address by changing the file ``armstub8.bin``, so there's no point in using
Dan Handley610e7e12018-03-01 18:44:00 +0000262 TF-A in this case.
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000263
264- ``LOAD_IMAGE_V2=0``: Only version 2 is supported.
265
Antonio Nino Diaz1f470022018-03-27 09:39:47 +0100266- ``MULTI_CONSOLE_API=0``: The multi console API must be enabled. Note that the
267 crash console uses the internal 16550 driver functions directly in order to be
268 able to print error messages during early crashes before setting up the
269 multi console API.
270
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000271AArch64 kernel build instructions
272---------------------------------
273
274The following instructions show how to install and run a AArch64 kernel by
275using a SD card with the default `Raspbian`_ install as base. Skip them if you
276want to use the default 32-bit kernel.
277
278Note that this system won't be fully 64-bit because all the tools in the
279filesystem are 32-bit binaries, but it's a quick way to get it working, and it
280allows the user to run 64-bit binaries in addition to 32-bit binaries.
281
2821. Clone the `Linux tree fork`_ maintained by the Raspberry Pi Foundation. To
283 speed things up, do a shallow clone of the desired branch.
284
285.. code:: shell
286
287 git clone --depth=1 -b rpi-4.14.y https://github.com/raspberrypi/linux
288 cd linux
289
2902. Configure and compile the kernel. Adapt the number after ``-j`` so that it is
291 1.5 times the number of CPUs in your computer. This may take some time to
292 finish.
293
294.. code:: shell
295
296 make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig
297 make -j 6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
298
2993. Copy the kernel image and the device tree to the SD card. Replace the path
300 by the corresponding path in your computers to the ``boot`` partition of the
301 SD card.
302
303.. code:: shell
304
305 cp arch/arm64/boot/Image /path/to/boot/kernel8.img
306 cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb /path/to/boot/
307
3084. Install the kernel modules. Replace the path by the corresponding path to the
309 filesystem partition of the SD card on your computer.
310
311.. code:: shell
312
313 make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
314 INSTALL_MOD_PATH=/path/to/filesystem modules_install
315
3165. Follow the instructions in `Setup SD card`_ except for the step of renaming
317 the existing ``kernel7.img`` (we have already copied a AArch64 kernel).
318
319Setup SD card
320-------------
321
322The instructions assume that you have an SD card with a fresh install of
323`Raspbian`_ (or that, at least, the ``boot`` partition is untouched, or nearly
Antonio Nino Diaz1f470022018-03-27 09:39:47 +0100324untouched). They have been tested with the image available in 2018-03-13.
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000325
3261. Insert the SD card and open the ``boot`` partition.
327
3282. Rename ``kernel7.img`` to ``kernel8.img``. This tricks the VideoCore
Dan Handley610e7e12018-03-01 18:44:00 +0000329 bootloader into booting the Arm cores in AArch64 mode, like TF-A needs,
330 even though the kernel is not compiled for AArch64.
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000331
3323. Copy ``armstub8.bin`` here. When ``kernel8.img`` is available, The VideoCore
333 bootloader will look for a file called ``armstub8.bin`` and load it at
334 address **0x0** instead of a predefined one.
335
Antonio Nino Diaz73ecb9f2018-06-13 13:34:46 +01003364. Open ``cmdline.txt`` and add ``memmap=16M$256M`` to prevent the kernel from
Dan Handley610e7e12018-03-01 18:44:00 +0000337 using the memory needed by TF-A. If you want to enable the serial port
338 "Mini UART", make sure that this file also contains
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000339 ``console=serial0,115200 console=tty1``.
340
341 Note that the 16 MiB reserved this way won't be available for Linux, the same
342 way as the memory reserved in DRAM for the GPU isn't available.
343
3445. Open ``config.txt`` and add the following lines at the end (``enable_uart=1``
345 is only needed to enable debugging through the Mini UART):
346
347::
348
349 enable_uart=1
350 kernel_address=0x01000000
351 device_tree_address=0x02000000
352
353If you connect a serial cable to the Mini UART and your computer, and connect
354to it (for example, with ``screen /dev/ttyUSB0 115200``) you should see some
355text. In the case of an AArch32 kernel, you should see something like this:
356
357::
358
359 NOTICE: Booting Trusted Firmware
360 NOTICE: BL1: v1.4(release):v1.4-329-g61e94684-dirty
361 NOTICE: BL1: Built : 00:09:25, Nov 6 2017
362 NOTICE: BL1: Booting BL2
363 NOTICE: BL2: v1.4(release):v1.4-329-g61e94684-dirty
364 NOTICE: BL2: Built : 00:09:25, Nov 6 2017
365 NOTICE: BL1: Booting BL31
366 NOTICE: BL31: v1.4(release):v1.4-329-g61e94684-dirty
367 NOTICE: BL31: Built : 00:09:25, Nov 6 2017
368 [ 0.266484] bcm2835-aux-uart 3f215040.serial: could not get clk: -517
369
370 Raspbian GNU/Linux 9 raspberrypi ttyS0
371 raspberrypi login:
372
373Just enter your credentials, everything should work as expected. Note that the
374HDMI output won't show any text during boot.
375
376.. _default Arm stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub7.S
377.. _default AArch64 stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub8.S
378.. _Linux kernel tree: https://github.com/torvalds/linux
379.. _Linux tree fork: https://github.com/raspberrypi/linux
380.. _Raspberry Pi 3: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/
Dan Handley610e7e12018-03-01 18:44:00 +0000381.. _Raspberry Pi 3 TF-A bootstrap: https://github.com/AntonioND/rpi3-arm-tf-bootstrap
Antonio Nino Diaz8869b482017-12-01 11:11:26 +0000382.. _Raspberry Pi 3 documentation: https://www.raspberrypi.org/documentation/
383.. _Raspbian: https://www.raspberrypi.org/downloads/raspbian/