64bit kernel on Raspberry Pi 4 (quirks)

by Andrei Gherzan 2019-07-04 Reading time: 4 minutes

TL;DR

this braindump is based on a more general one available here
it adds specific quirks currently needed for Raspberry Pi 4

There are a couple of things we discovered while adding support in Yocto/meta-raspberrypi for the new Raspberry Pi 4 and I realized that they can be a good guide for anyone trying to boot a 64bit kernel for this board with or without a 64bit user-space. I will try to keep this post updated as further development will invalidate items discussed below.

A. Raspberry Pi firmware

If you are working on top of a 32bit Raspbian distribution (at least 2019-06-20), you will have all the needed blobs in the boot partition. If you build a distro from scratch or working on a different distro, you will need to include the boot files from a raspberrypi-firmware release version greater or equal to 1.20190620.

B. Compile aarch64 toolchain

This step is only needed if you need to build a custom armstub (see below). Initially, this guide included the instructions for building a toolchain and the kernel. That was split out of it because it was useful for other boards as well (see Raspberry Pi 3). Also, all these quirks will be eventually obsolete. That being said, for now, follow the guide here to build a 64bit kernel for Raspberry Pi 4 (be aware to pick up the right defconfig.

C. arm8 stub with `BCM2711` support

A new Raspberry Pi firmware was released which includes the needed armstubs support for bcm2838 (bcm2711). It means that you can skip this entire step by just using a newer firmware version. The minimum version needed to avoid having custom stubs is 2019-07-09. This version is included in Raspbian 2019-07-10.

If you followed the braindump above, you have a toolchain and a kernel compiled. I assume that the exported variables (I define some paths to be able to reference them later) are in place. Otherwise feel free to tweak the instructions below as needed.

The armstub is the code that the GPU loads and sets the ARM CPU before running the kernel. This support for BCM2711 was not included in the current latest firmware (see above) so you will need to compile and configure the board to use the appropriate one.

git clone https://github.com/raspberrypi/tools.git rpi-tools
cd rpi-tools/armstubs
git checkout 7f4a937e1bacbc111a22552169bc890b4bb26a94
PATH=$PATH:$TOOLCHAIN/bin make armstub8.bin # See B above for the TOOLCHAIN path
export ARMSTUB=`realpath armstub8.bin` # used if you want to deploy it to raspbian, ignore otherwise

D. Tweak NVRAM configuration for `brcmfmac43455`

Raspbian >= 2019-06-20 includes this change by default so skip if that is what you are working on. Otherwise, the current firmware-nonfree uses a NVRAM configuration which renders the WiFi interface unusable. You will need to change bootflags3 as it follows:

boardflags3=0x44200100

E. Switch Raspbian `2019.06.20` to 64bit kernel

Let’s deploy now all the needed quirks for a Raspberry Pi 4 running a 64bit kernel on Raspbian.

The following commands assume that the boot partition mount point is /run/media/me/boot. Also, the paths rely on the exports set throughout this braindump and the one described in B.

The needed armstub is now included from Raspbian 2019-07-10 (the packaged firmware includes the armstubs for bcm2711). Deploying and using a custom one is only needed if you are working (for whatever reason) on an older version:

cp $ARMSTUB /run/media/me/boot
echo "armstub=armstub8-gic.bin" >> /run/media/me/boot/config.txt
echo "enable_gic=1" >> /run/media/me/boot/config.txt 

Kernel on 64bit had a memory size limitation due to the fact that only the first 1Gb can be used for DMA. This was mitigated in the upstream Raspberry Pi linux fork. If you are using an older version of the kernel, you will need to limit the memory which in turn will limit the memory for DMA:

echo "total_mem=1024" >> /run/media/me/boot/config.txt

Even in the current kernel there seems to be a bug that the USB ports are not working with 4 GB of RAM but the 64 bit kernel works fine with just 3 GB memory, see this issue. So we should limit the ram to 3 GB by

echo "total_mem=3072" >> /run/media/me/boot/config.txt

The last piece in the jigsaw is forcing the firmware to set the arm in 64bit mode. This should happen automatically based on the kernel image filename. Currently that doesn’t seem to be stable so we will force it in the config.txt:

echo "arm_64bit=1" >> /run/media/me/boot/config.txt

And that is it.

sync
umount /run/media/me/boot

In conclusion, if you are working on the latest Raspbian and the latest revision 4.19.y kernel, all you need to do is force the arm in 64bit mode and an 3 GB memory limit as describe above.