I have learnt that 32-bit and 64-bit ARM are radically different, down to having different instruction sets. And that means that all the 32-bit examples are not directly useful to me: I can extract information from them and use their resources but I cannot reuse any of the assembler code, and much of any C code may have references to addresses or layouts that are not compatible with the 64-bit model.
Furthermore, although I think it's probably a good idea in general, I don't want to fiddle with GPIO pins and serial UART ports - I want to use the framebuffer and HDMI. And I want to write my code in Rust, although I will happily port code written in C. So of the possible examples I have, it seems that Koltan Baldaszti's repository is the best starting point. So, I'm going to copy the code from tutorial 9 and adapt it somewhat. This tutorial sets out to display an image of Homer Simpson on the display during boot.
Get it to Work
I have a sample, but I still need to actually reproduce the results myself. And before I can do anything else, I want to install the Pi Emulator.Fortunately, there is a Linux package for that.
sudo apt install qemu-system-armWe already have a suitable C compiler from our previous cross-compilation of Rust. One of my modifications is to update the Makefile to use that rather than aarch64-elf-gcc. After this, make will build our application and make run will run it in the emulator. So far, so easy.
[sudo] password for gareth:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
ibverbs-providers ipxe-qemu ipxe-qemu-256k-compat-efi-roms libcacard0 libdaxctl1 libfdt1 libgfapi0 libgfrpc0 libgfxdr0 libglusterfs0
libibverbs1 libiscsi7 libndctl6 libpmem1 libpmemobj1 librados2 librbd1 librdmacm1 libslirp0 libspice-server1 libusbredirparser1
libvirglrenderer1 qemu-block-extra qemu-efi-aarch64 qemu-efi-arm qemu-system-common qemu-system-data qemu-system-gui qemu-utils
Suggested packages:
gstreamer1.0-libav gstreamer1.0-plugins-ugly samba vde2 debootstrap
The following NEW packages will be installed
ibverbs-providers ipxe-qemu ipxe-qemu-256k-compat-efi-roms libcacard0 libdaxctl1 libfdt1 libgfapi0 libgfrpc0 libgfxdr0 libglusterfs0
libibverbs1 libiscsi7 libndctl6 libpmem1 libpmemobj1 librados2 librbd1 librdmacm1 libslirp0 libspice-server1 libusbredirparser1
libvirglrenderer1 qemu-block-extra qemu-efi-aarch64 qemu-efi-arm qemu-system-arm qemu-system-common qemu-system-data qemu-system-gui
qemu-utils
0 to upgrade, 30 to newly install, 0 to remove and 31 not to upgrade.
At some point, I need to write a full post on the boot sequence for the Pi, but for now, the important thing to know is that the Pi looks for an SD Card with a "boot" partition which needs to be formatted for FAT32. On that card, there need to be some "standard" firmware files and a kernel7.img or kernel8.img file. All of this happens automatically when you create a standard Pi SD card, so I'm just reusing one of these. The difference between kernel7.img and kernel8.img is the architecture that the system will boot into: if it finds kernel8.img, it will use ARMv8 (64-bit) whereas if it finds kernel7.img, it will use ARMv7 (which is the 32-bit version). All very clever.
So to deploy this program on my actual Pi box, it is simply necessary to copy the kernel8.img file produced by make into the "boot" partition of an SD card which was previously formatted to run the standard OS. In spite of the name, this file doesn't need to be an OS kernel as such, just a program that can run without an existing operating system. Obviously, it is possible to start from scratch with a completely vanilla SD card and install all the necessary files: I'm sure that at some point I will do this.
When this card is put into the Pi and it is booted, a picture of Homer appears on the screen.
Excellent.
Introducing Rust
I'm now going to go back and start again with a completely new Rust project based on the Rust Bare Bones Tutorial which I have to admit I don't understand. Since I don't like just following instructions I don't understand, I'm going to try and skip most of the steps and see what does and doesn't work. If necessary, I'm going to come back around and fill in the blanks.The objective of this tutorial is just to get something to build using Rust that will run on the emulator and on the Pi.
So, first off, I'm going to do the obvious and use cargo to create my new project:
$ cargo new homer_rustI'm then going to follow the instruction to handle panic as abort in Cargo.html:
Created binary (application) `homer_rust` package
[profile.dev]And then I'm going to put this in src/lb.rs:
panic = "abort"
#![no_std]This implies to me that we don't want the src/main.rs which was automatically generated by cargo, so I'm going to delete it.
#[no_mangle]
pub extern fn kernel_main() {}
And then I'm going to build it using a cross-compiling cargo target:
$ cargo build --target=aarch64-unknown-linux-gnuWe now have a target directory and, inside this, we have a subdirectory for our cross target aarch64-unknown-linux-gnu, and then, within that a debug directory. In here are a number of files which I don't quite understand, but some judicious investigation suggests that the library has been built into homer_rust.rlib where the extension rlib is just an alias for an ar archive with compiled rust files in it (ar is the program used by the C compiler and other systems programming languages such as as for building archive libraries).
Compiling homerrust v0.1.0 (/home/gareth/Projects/homerrust)
Finished dev [unoptimized + debuginfo] target(s) in 0.03s
This library can now be linked together with a boot.S assembler script using an appropriate linker script to build an entire kernel image.
The instructions I'm following assume that you are familiar with the C version of the tutorial so I will go there to pull in the relevant files.
For now, just follow along. When we move on from just trying to copy things and get them to work, I'll come back and either write updated versions of these files or keep the same versions and explain how and why they work.
We need a linker.ld script which is appropriate for the Aarch64 architecture:
ENTRY(_start)And we also need the appropriate boot.S taken from the same source.
SECTIONS
{
/* Starts at LOADER_ADDR. */
. = 0x80000;
__start = .;
__text_start = .;
.text :
{
KEEP(*(.text.boot))
*(.text)
}
. = ALIGN(4096); /* align to page size */
__text_end = .;
__rodata_start = .;
.rodata :
{
*(.rodata)
}
. = ALIGN(4096); /* align to page size */
__rodata_end = .;
__data_start = .;
.data :
{
*(.data)
}
. = ALIGN(4096); /* align to page size */
__data_end = .;
__bss_start = .;
.bss :
{
bss = .;
*(.bss)
}
. = ALIGN(4096); /* align to page size */
__bss_end = .;
__bss_size = __bss_end - __bss_start;
__end = .;
}
// To keep this in the first portion of the binary.We now need to assemble this, and then link everything together:
.section ".text.boot"
// Make _start global.
.globl _start
.org 0x80000
// Entry point for the kernel. Registers:
// x0 -> 32 bit pointer to DTB in memory (primary core only) / 0 (secondary cores)
// x1 -> 0
// x2 -> 0
// x3 -> 0
// x4 -> 32 bit kernel entry point, _start location
_start:
// set stack before our code
ldr x5, =_start
mov sp, x5
// clear bss
ldr x5, =__bss_start
ldr w6, =__bss_size
1: cbz w6, 2f
str xzr, [x5], #8
sub w6, w6, #1
cbnz w6, 1b
// jump to C code, should not return
2: bl kernel_main
// for failsafe, halt this core
halt:
wfe
b halt
aarch64-linux-gnu-gcc -Wall -O2 -ffreestanding -fno-stack-protector -nostdinc -nostdlib -nostartfiles -c boott.S -o boot.oThis produces a "standard binary", that is, one which would load and run under Linux. But we are trying to execute something on bare metal, so we cannot expect to have a relocating loader to hand. So we need to extract all the code and build an image file that can be directly mapped into memory.
aarch64-linux-gnu-ld -nostdlib -T linker.ld boot.o target/aarch64-unknown-linux-gnu/debug/libhomer_rust.rlib -o kernel8.elf
aarch64-linux-gnu-objcopy -O binary kernel8.elf kernel8.imgSo this is the kernel image we will want to load. And then we can run this in the emulator using the same command as before:
qemu-system-aarch64 -M raspi3b -kernel kernel8.img -serial stdioThe emulator starts up and doesn't complain, so I'm going to consider that a moral victory.
Adding some actual code
This "kernel" is the absolutely minimal bare metal Rust program - it does nothing. It's probably possible to connect to the emulator in some way to see that the boot.S code at least is running. But I'm going to assume that everything is OK and press on to writing some more serious code.At this point, the Rust bare bones tutorial uses the UART port (which is mapped to the console in the emulator) to write the string "Hello, Rust kernel world". We can copy this code into our example and see if we can see this in the emulator. Since I don't have the UART actually connected to my physical hardware, this won't work on the Pi box itself.
There is a lot of complex code to actually drive the UART, which I'm not interested in, but ends up being wrapped up in a function write. So the kernel_main function becomes:
#[no_mangle]With this in place, we can re-link and re-run the code:
pub extern fn kernel_main() {
write("Hello Rust Kernel world!");
loop {
writec(getc())
}
}
aarch64-linux-gnu-ld -nostdlib -T linker.ld boot.o target/aarch64-unknown-linux-gnu/debug/libhomer_rust.rlib -o kernel8.elfSadly, it doesn't work.
aarch64-linux-gnu-objcopy -O binary kernel8.elf kernel8.img
qemu-system-aarch64 -M raspi3b -kernel kernel8.img -serial stdio
It's not entirely clear why not. This is why we have debuggers.
It is possible to start the emulator QEMU with a couple of flags that say "don't start running the executable until the debugger says so" (-S) and "open a port to allow a debugger to connect" (-s). The difference between these two flags is the case that they are in.
qemu-system-aarch64 -M raspi3b -S -s -kernel kernel8.img -serial stdioBut what debugger do we have? There is a multi-architecture (i.e. cross) debugger available on linux:
$ sudo apt install gdb-multiarchWe obviously need to open a new terminal for this (the emulator is running in a terminal and is waiting for us to connect) and then run the debugger giving it information about the symbols in the program:
[sudo] password for gareth:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed
gdb-multiarch
0 to upgrade, 1 to newly install, 0 to remove and 31 not to upgrade.
Need to get 4,589 kB of archives.
After this operation, 18.2 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 gdb-multiarch amd64 12.1-0ubuntu1~22.04 [4,589 kB]
Fetched 4,589 kB in 1s (3,238 kB/s)
Selecting previously unselected package gdb-multiarch.
(Reading database ... 620618 files and directories currently installed.)
Preparing to unpack .../gdb-multiarch_12.1-0ubuntu1~22.04_amd64.deb ...
Unpacking gdb-multiarch (12.1-0ubuntu1~22.04) ...
Setting up gdb-multiarch (12.1-0ubuntu1~22.04) ...
$ gdb-multiarch -e kernel8.elfOnce the debugger starts, we tell it that the running process has opened the port 1234 (the default for -s) and then say that we want the correct layout for assembler, since we want to start by stepping through the boot.S file.
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) target remote :1234When we do this, we can see that the processor starts executing at address 0x0 and there is some "default" code there which executes and then transfers control to 0x80000. When we get there, we find that the code there is not the code from boot.S that we are expecting.
Remote debugging using :1234
0x0000000000000000 in ?? ()
(gdb) layout asm
What can we do to find our code? Well, objdump is a low-level introspector (compare to javap if you have a Java background). It has a whole bunch of features, including the (-d) option to "disassemble" the code in a file, so let's see what we've got:
$ aarch64-linux-gnu-objdump -d kernel8.elfOK, well that at least explains what is going on: _start has been correctly bound to 0x80000. But then the first instruction doesn't actually come along until 0x100000. Why not? I have to confess that it took me a while to figure this out, including fiddling with different things, until I realized that 0x100000 is exactly double 0x80000. What's happened is that we have applied two offsets of 0x80000 before the first instruction. Why? I'm not entirely sure, but I suspect that the instructions and code we have used have not all been entirely consistent. Looking at the linker file, we have this:
kernel8.elf: file format elf64-littleaarch64
Disassembly of section .text:
0000000000080000 <_start>:
...
100000: 58000185 ldr x5, 100030 <halt+0xc>
100004: 910000bf mov sp, x5
100008: 58000185 ldr x5, 100038 <halt+0x14>
10000c: 18000106 ldr w6, 10002c <halt+0x8>
100010: 34000086 cbz w6, 100020 <_start+0x80020>
100014: f80084bf str xzr, [x5], #8
100018: 510004c6 sub w6, w6, #0x1
10001c: 35ffffa6 cbnz w6, 100010 <_start+0x80010>
100020: 94000067 bl 1001bc <kernel_main>
. = 0x80000;while in the assembler file boot.S, we have:
__start = .;
__text_start = .;
.text :
{
KEEP(*(.text.boot))
*(.text)
}
.globl _startAnd I think both of these are being applied, rather than being seen as saying the same thing. So the obvious thing is to comment out the .org in the boot.S file.
.org 0x80000
_start:
.globl _startAnd with that fix, everything works!
// .org 0x80000
_start:
$ qemu-system-aarch64 -M raspi3b -kernel kernel8.img -serial stdioVery good. Let's check that in for posterity, and tag it RUST_BARE_METAL_MINIMAL.
Hello Rust Kernel world!
Making that reproducible
I'm not happy with either the file layout I've ended up with, or with the number of steps that it takes to build it. As far as I can tell, there is currently no standard way to perform "post-build" steps in a Rust project, such as the custom linking we are doing here. So I'm going to use a shell script to coordinate the cargo build of the Rust project with make doing the final assembly. To facilitate that, I'm going to copy the Makefile from my previous project and use that to assemble boot.S, do the linking and generate the final image. It also provides the run target to run the resultant image in the emulator.The Makefile ends up looking like this:
# Allow run to connect to GDB by specifying GDB flagsAnd then the build script which first does the Rust compilation, then calls make, looks like this:
# for example, -s -S
GDB =
DEBUG_WITH_GDB = -s -S
LIB = ../target/aarch64-unknown-linux-gnu/debug/libhomer_rust.rlib
all: kernel8.img
boot.o: boot.S
aarch64-linux-gnu-gcc -Wall -O2 -ffreestanding -fno-stack-protector -nostdinc -nostdlib -nostartfiles -c boot.S -o boot.o
kernel8.img: boot.o $(LIB)
aarch64-linux-gnu-ld -nostdlib boot.o $(LIB) -T linker.ld -o kernel8.elf
aarch64-linux-gnu-objcopy -O binary kernel8.elf kernel8.img
clean:
rm -f kernel8.elf kernel8.img *.o
run: kernel8.img
qemu-system-aarch64 $(GDB) -M raspi3b -kernel kernel8.img -serial stdio
cargo build --target aarch64-unknown-linux-gnuAt the same time, I'm going to move all the non-rust code down into a new directory called asm. And wrap everything in a single script to build the Rust and assembler code and link it all together.
cd asm
make
This is now available in RUST_BARE_METAL_MAKEFILE
Porting Homer from C to Rust
At this point, I have derived some code from two separate tutorials and have them both working. One is written in C and does what I want: displays a picture of Homer on the monitor. The other uses Rust but does none of this.So what I want to do now is to port the C code to Rust and ultimately have a program which is written in Rust, runs on actual Pi hardware and displays Homer's picture. I'm actually not sure how hard that is going to be to do.
The various parts of this code varies from the trivial to the hairy, so, in spite of the fact that nothing really works until we have some of the hairy code done, I am going to start with the trivial. Homer's image in the tutorial is encoded in something called The Gimp Header File Format which, it turns out, is just a way of encoding 24-bit RGB values as four printable characters.
So I'm going to start by copying across the actual data into Rust, and then porting the decoding function and then write the resulting image to the UART in hex.
In order to do this, I am going to first write a function (or two) to convert 24-bit RGB values (in a u32) to a "string". But because we are working in a very constrained environment, it's hard to create Strings or Vecs, so I'm using a static bufffer of u8s. Sadly, because of the limitations of Rust, this appears to be unsafe. Also, in order not to have panics when doing arithmetic, I have also switched to using a release build, rather than a debug build.
I'm slowly starting to realise why the various steps that I skipped in the "Bare Bones Rust tutorial" were there. In order to be able to use the standard library, it's necessary to have a build of Rust you can modify at runtime, so that you can use your own calls to replace the underlying calls (such as memory management) that are normally found in the operating system. For now, I'm going to continue to resist the temptation, but one day it will become inevitable.
static mut BUF:[u8;6] = [0;6];And we can test that this works in the lib.rs file:
pub fn hex(n : u32) -> &'static[u8;6] {
unsafe {
BUF[0] = digit((n >> 20) & 0xf);
BUF[1] = digit((n >> 16) & 0xf);
BUF[2] = digit((n >> 12) & 0xf);
BUF[3] = digit((n >> 8) & 0xf);
BUF[4] = digit((n >> 4) & 0xf);
BUF[5] = digit(n & 0xf);
&BUF
}
}
fn digit(n: u32) -> u8 {
if n < 10 {
((48 + n) & 0xff) as u8
} else {
(55 + n) as u8
}
}
pub extern fn kernel_main() {With that in place for debugging, we can go back and convert Homer into RGB format by converting each group of four characters into 24-bit RGB pixel values.
write_chars(hex(0x9a3cb0));
...
}
pub fn read_homer(homer: &str) -> &'static[u8;HOMER_BYTES] {This code is tagged as RUST_BARE_METAL_CONVERT_IMAGE.
unsafe {
let mut pos : usize = 0;
while pos < homer.len() {
let c0 = homer.as_bytes()[pos];
let c1 = homer.as_bytes()[pos+1];
let c2 = homer.as_bytes()[pos+2];
let c3 = homer.as_bytes()[pos+3];
IMAGE[pos] = 0;
IMAGE[pos+1] = ((c0-33) << 2) | ((c1-33) >> 4);
IMAGE[pos+2] = ((c1-33) << 4) | ((c2-33) >> 2);
IMAGE[pos+3] = ((c2-33) << 6) | ((c3-33));
pos+=4;
}
&IMAGE
}
}
Framebuffer Initialization
Before we can draw Homer on the screen, we need access to a "framebuffer". If you've not come across the word before, a framebuffer is a low-level block of memory that is mapped onto the screen by the GPU. So we need to ask the GPU to create a framebuffer for us. This requires us to use a "mailbox" to talk to the GPU.A mailbox is a way in which the CPU can communicate with the hardware using a combination of Memory-Mapped I/O and DMA. That is, a program can create a request in a buffer in memory, and then pass the address of that buffer to a specific memory-mapped address (the "mailbox") which is then read by the GPU, which then reads and processes the request from the buffer at that address. A further memory-mapped status address is then used to determine when the operation has completed and the response has been written to the same buffer.
It seems to me that there is not a single source you can go to in order to find out all the information you need in order to do this, but rather there are a number of different reference documents which you need to consult to put the entire picture together.
Probably the best documentation for this I have come across so far is the Raspberry Pi firmware wiki which has a page describing the various mailboxes. In fact, there are only two actual "mailboxes", one to send messages from the ARM to the GPU, and one for the GPU to send messages to the ARM. But the mailbox has within it a set of "channels", where each channel is essentially a different mailbox.
The other key source of information is other people's code, which has the advantage of having been tested. Unfortunately, it is often the case that this code worked in a different environment or on a different box to the one that we are using. In this case, we have some C code which does exactly what we want and that we know works on our box in 64-bit mode. So, first, let's look at that code. For now, we are just going to try and initialize the frame buffer.
The first thing is that there is a declaration of the message array. This is declared to be 36 4-byte words (144 bytes). This magic number is the size of the request, as we will see in a moment.
extern volatile unsigned int mbox[36];Among the challenges of working with devices on the Raspberry Pi is that everything is different everywhere depending on the specific device you have and how you are using it. I will try to be repeatedly clear that I am using a 3B+ in 64-bit (aarch64) mode. In this case, it would seem that the base address for all the memory-mapped devices is 0x3F000000. I don't have a definitive reference for this, but I have seen it used a number of times (including in this project) and it is the accepted answer in this question on the forum.
This is defined in gpio.h:
#define MMIO_BASE 0x3F000000Separately, from somewhere has come the knowledge that the mailbox for the Videocore GPU is to be found at offset 0xB880 in this range - that is at 0x3f00b880. This is defined in mbox.c:
#define VIDEOCORE_MBOX (MMIO_BASE+0x0000B880)As yet, I have been unable to find any original source for this, just examples of people using it.
The next step is to build a request in the mbox message defined earlier. This consists of four set operations and two get operations. These requests I can track back at least as far as the Wiki, but I'm not sure where they get their information from.
The first two words (remember, each of the entries in the mbox array is a 4-byte word) are the number of bytes in the request (not including this length) and a signal that this is a request (which is a word consisting of all zeros). The same buffer is used for the request AND the reply, so this distinguishes between the two. Responses all have a leading '1' (ie. 0x8xxxxxxx).
mbox[0] = 35*4;The first request is to set the physical width and height of the framebuffer. "Physical" refers to how big we think the actual screen is in pixels.
mbox[1] = MBOX_REQUEST;
0x48003 is the magic code which says that this is the operation we want to perform. The next word contains the space allocated for the value in bytes. We are going to be sending two words (a width and a height), so that is 8. As I read it, the following value is for flags sent with the request, which must have its top bit (b31) clear, and all the rest of the fields are reserved; I would use 0 myself (and will do later unless it doesn't work), but this code has repeated 8. When the request is overwritten by the response, this field will hold the length of the response value, again in bytes.
mbox[2] = 0x48003; //set phy whTo set the virtual width and height, we use the code 0x48004. This dictates how much memory is to be allocated for the display. In this case, the code has simply requested the same size as the physical screen, but it is common to request double the height to allow for double buffering of the display.
mbox[3] = 8;
mbox[4] = 8;
mbox[5] = 1024; //FrameBufferInfo.width
mbox[6] = 768; //FrameBufferInfo.height
mbox[7] = 0x48004; //set virt whGiven that there are both physical and virtual framebuffers, it is necessary to specify where in the virtual framebuffer the origin of the physical framebuffer is to be found. Tag 0x48009 performs this operation. Obviously, since our physical and virtual framebuffers are the same size, the offset must be (0,0).
mbox[8] = 8;
mbox[9] = 8;
mbox[10] = 1024; //FrameBufferInfo.virtual_width
mbox[11] = 768; //FrameBufferInfo.virtual_height
mbox[12] = 0x48009; //set virt offset0x48005 sets the depth of the framebuffer. This is the number of bits that are used to represent each pixel. This is set as 32. I'm not quite sure what that actually means - is it 24 rounded up to 32 or is it RGBA? Note that we only have 4 bytes of data with this tag.
mbox[13] = 8;
mbox[14] = 8;
mbox[15] = 0; //FrameBufferInfo.x_offset
mbox[16] = 0; //FrameBufferInfo.y.offset
mbox[17] = 0x48005; //set depthThe Videocore apparently allows both RGB and BGR orderings on the pixels, so we have to request the one that we want (if available). The 0x48006 request chooses a particular order (0 = BGR, 1 = RGB).
mbox[18] = 4;
mbox[19] = 4;
mbox[20] = 32; //FrameBufferInfo.depth
mbox[21] = 0x48006; //set pixel orderNow we need a memory address for the framebuffer. The allocate buffer request allocates a framebuffer and responds with the address. This code does not seem to exactly match the specification, which says the request value size should be 4. On the other hand, the response value size is expected to be 8, so it makes sense if the response is going to be written in the same place that we need 8 bytes here.
mbox[22] = 4;
mbox[23] = 4;
mbox[24] = 1; //RGB, not BGR preferably
The value that is passed here (4096) is the alignment, written as a modulus rather than a bit count or a mask. I'm assuming that's right. So whatever address we get back should be aligned to 4096, i.e. the bottom 14 bits will all be 0s.
mbox[25] = 0x40001; //get framebuffer, gets alignment on requestThe description of the Get Pitch tag says that it is the number of bytes per line. I would expect this to be four times the width in pixels (so 4096), but hardware being hardware, I can understand why it is good to check.
mbox[26] = 8;
mbox[27] = 8;
mbox[28] = 4096; //FrameBufferInfo.pointer
mbox[29] = 0; //FrameBufferInfo.size
mbox[30] = 0x40008; //get pitchMBOX_TAG_LAST (which has the value 0) says that there are no more tags in this request. Which is good, because we have run out of bytes in which to place them.
mbox[31] = 4;
mbox[32] = 4;
mbox[33] = 0; //FrameBufferInfo.pitch
mbox[34] = MBOX_TAG_LAST;OK. So now we know what we want the code to do, how do we write that in Rust?
Well, it's a little complicated, I think because we don't have proper memory allocation. But a mutable buffer seems able to handle the construction of the message.
let mut buf: [u32;36] = [0; 36];So now we need to actually send that whole message to the GPU for processing.
// The header of the message has a length and a status (0 = REQUEST; 0x8000xxxx = RESPONSE)
buf[0] = 35 * 4; // the buffer has 35 4-byte words
buf[1] = 0; // we indicate we are sending a MBOX_REQUEST as 0
// Now each of the tags
// First, set the physical size of the framebuffer to 1024 x 768
buf[2] = 0x48003;
buf[3] = 8; // the number of bytes in the request value
buf[4] = 0; // reserved in request - will be used for the length of the response value
buf[5] = 1024; // the requested width
buf[6] = 768; // the requested height
// Now, set the virtual size of the framebuffer
// This must be at least as big as the physical framebuffer, but can be bigger, e.g. to support double buffering
// or to support scrolling
buf[7] = 0x48004;
buf[8] = 8;
buf[9] = 0;
buf[10] = 1024; // the requested virtual width
buf[11] = 768; // the requested virtual height
// Now specify where the physical framebuffer is in the virtual framebuffer
buf[12] = 0x48009;
buf[13] = 8;
buf[14] = 0;
buf[15] = 0;
buf[16] = 0;
// Now set the depth of the framebuffer in bits. This is 32, presumably to get 24 bits aligned nicely
buf[17] = 0x48005;
buf[18] = 4;
buf[19] = 0;
buf[20] = 32; // allocate 32 bits per pixel
// choose RGB over BGR
buf[21] = 0x48006;
buf[22] = 4;
buf[23] = 0;
buf[24] = 1; // 1 = RGB, 0 = BGR
// Now we are ready to allocate the buffer.
buf[25] = 0x40001;
buf[26] = 8; // we are only sending 4 bytes, but we want 8 bytes back
buf[27] = 0;
buf[28] = 4096; // the modulus of the alignment we want (i.e. only the top 20 bits are significant)
buf[29] = 0;
// And we want to check that we were given RGB (or not)
buf[30] = 0x40008;
buf[31] = 4;
buf[32] = 0;
buf[33] = 0; // this is just a placeholder for the value to come back
// We have no more tags
buf[34] = 0;
Sadly, before we can do that, it seems that we need to add a spin loop to avoid a SEGV in the emulator qemu (this saddens me, but such is life). This doesn't do anything, it just wastes time. The mmio_read is just there to stop the compiler from optimizing the entire loop away.
// avoid a SEGV in the emulatorSo now we can write to the mbox. We call a function mbox_send to make it clear where the separation is between creating the request and dealing with the hardware.
let mut y = 0;
while y < 1000000 {
y = y + 1;
mmio_read(MBOX_STATUS); // do something to waste time
}
mbox_send(8, &mut buf);This, of course, is where the magic happens. The "magic number" 8 here is the "channel" number that we want to send the message on. Again, this is poorly documented; I obtained it mainly from examples, but I would cite as a "definitive" source the raspberry pi wiki.
This method starts by polling the MBOX_STATUS port until it stops being "busy".
fn mbox_send(ch: u8, buf: &mut[u32; 36]) {The message is passed in as a strongly typed array, but right now we are about to descend into the dark world of systems programming, and deal with "unsafe" raw pointers and integers, so that we can manipulate them. Note that this code does not need to be wrapped in an unsafe block, because converting pointers to integers is not inherently unsafe: it is taking a "random" integer and using it as a pointer which is unsafe.
while mmio_read(MBOX_STATUS) & MBOX_BUSY != 0 {
}
// obtain the address of buf as a raw pointerPlease pay attention! I am going to describe this next bit as "the most important thing to note here" for two reasons: first, I didn't do this right when I first wrote this code and spent about an hour trying to figure out why nothing happened; and, secondly, it is so counterintuitive to me.
let volbuf = buf as *const u32;
// then convert that to just a plain integer
let ptr:u32 = volbuf as u32;
The pointer is a "word" pointer and thus the bottom 4 bits are all zeros. The mailbox interface wants to just read one word for ease, so it re-purposes those four bits to send the channel number. If you don't merge the channel number in here, it will be sent to channel 0, which (at least in this case) won't do what you expect. I don't actually know what it will do, but it definitely won't initialize the framebuffer.
// what we pass to the mailbox is that in the top 28 bits and the channel number (ch) in the bottom 4 bitsNow we write this to the MBOX_WRITE address, which is a memory mapped device address, so it immediately causes the GPU (which is listening on the other end) to process our message. When it has finished processing, it will put a response back on the MBOX_STATUS address, so we wait for that to happen.
let addr = (ptr & !0x0F) | ((ch as u32) & 0x0f);
// send to the mailbox write addressWe can then write the output to the tty, which will come out on the terminal screen when running in the emulator.
mmio_write(MBOX_WRITE, addr);
// wait until we have a response from the GPU
while mmio_read(MBOX_STATUS) & MBOX_PENDING != 0 {
}
// show the returned buffer contentsWe now want to process the response. We know that the response is written back into the same buffer that we sent, so we should just be able to use buffer accesses. I ran into a problem that this didn't seem to be happening, and concluded that the compiler had decided it shouldn't have changed and returned the constants I had put there. I'm not sure that was true, but I added a read_volatile anyway. Once we have the response in place, we can check that all the things that should have happened in a particular way did, in fact, happen in that way.
write("returned buffer contents:\n");
let mut x = 0;
while x < 36 {
write_8_chars(hex32(buf[x]));
write("\n");
x = x + 1;
}
// The compiler optimizes away (or something) reads into buf and returns what we wroteWe can now run this in the emulator by using make:
// We need to be sure we read what was written
let stat = unsafe { read_volatile(volbuf.add(1)) };
// test that we received valid data
if stat != 0x80000000 {
write("error returned from getfb ");
write_8_chars(hex32(stat));
write("\n");
return;
}
let pixdepth = unsafe { read_volatile(volbuf.add(20)) };
if pixdepth != 32 {
write("pixel depth is not 32");
return;
}
let alignment = unsafe { read_volatile(volbuf.add(28)) };
if alignment == 0 {
write("alignment is zero");
return;
}
}
cd asmNothing very exciting happens, but we do get a bunch of numbers coming out confirming the buffer response and also showing us the converted RGB pixels of the image.
make run
I have checked this in as RUST_BARE_METAL_INITIALIZE_FRAMEBUFFER. So now it's time to move on to actually rendering the image on the screen.
Rendering the Image
So now we have an image in a buffer that is in basic RGB format, and we have a connection to the framebuffer. We need to extract the address of the framebuffer from the mailbox reply, and then start copying our image data from the decoded buffer to the framebuffer.Again, we have example C code to follow, and it's just a question of trying to convert that into Rust. I'm not going to reproduce all the C code here, although you can look at it in the relevant tutorial if you want.
The code to render Homer is fundamentally simple, so I'm going to present the function I ended up with all in a single slab here, and then go back and show the changes I had to make elsewhere in order to make everything fit together.
Basically, it is just a question of copying the array of pixels that make up the image of Homer into the block of memory we have been allocated as the framebuffer. The only caveat is that we need to copy it into the centre of the screen on a line-by-line basis.
fn show_homer(fb : &FrameBufferInfo, homer : &[u8; HOMER_BYTES]) {So the first thing we do is to figure out how much blank space we need at the top and the left.
// Because we want to put Homer in the middle of the screen, we need to first figure out where
// he should go. We have a framebuffer width and height, and a homer width and height.
// So we need to put half of the distance in each direction as an initial value for x and y
let xoff = fb.width/2 - HOMER_WIDTH/2;
let yoff = fb.height/2 - HOMER_HEIGHT/2;
// Now we want to go over each of the scan lines, having the ptr be the base (from fb.base_addr)Each row is just copied sequentially from the source image (using homer_index) into a single scan line. We figure out where to put this starting at the base address of the framebuffer PLUS the number of bytes per line multiplied by the top margin added to the current line, and then finally a left indent of 4 (the number of bytes per pixel) multiplied by the left margin. It is important to remember here that both the framebuffer pointer and ptr are pointers to bytes.
// plus the current y multiplied by the pitch, plus the above xoff
let mut homer_index: usize = 0;
let mut y = 0;
while y < HOMER_HEIGHT {
let ptr = fb.base_addr + (yoff + y) * fb.pitch + xoff*4;
let mut x: u32 = 0;Finally, we can copy across one row of pixels, and then increase the x position by one (which will be multiplied by 4 when computing the start location of the next pixel) and homer_index by 4. At the end of a row, we increase the line count by 1.
while x < HOMER_WIDTH {
unsafe { *((ptr + x*4 + 0) as *mut u8) = homer[homer_index + 0]; }
unsafe { *((ptr + x*4 + 1) as *mut u8) = homer[homer_index + 1]; }
unsafe { *((ptr + x*4 + 2) as *mut u8) = homer[homer_index + 2]; }
unsafe { *((ptr + x*4 + 3) as *mut u8) = homer[homer_index + 3]; }
x += 1;
homer_index += 4;
}
y += 1;
}
}
So far, so good, except that doesn't compile for a number of reasons. Most importantly, where did that new argument type FrameBufferInfo come from?
In order to group together all of the information about the framebuffer when collecting it from the return from the mailbox call to initialize the framebuffer and pass it to the render function, I declared a struct.
struct FrameBufferInfo {The four pieces of information we need here are the physical width and height of the framebuffer (which may be set differently to what we asked for), the pitch (the number of bytes in each line) and the allocated base address of the framebuffer.
width : u32,
height : u32,
pitch: u32,
base_addr: u32
}
Less obviously, we have reused HOMER_HEIGHT and HOMER_WIDTH which were defined in ghff.rs when we were reading the file. These are inaccessible due to protection, so we have made them pub.
pub const HOMER_HEIGHT : u32 = 64;Furthermore, in the file where we are referencing them, they need to be explicitly imported:
pub const HOMER_WIDTH : u32 = 96;
use crate::ghff::HOMER_WIDTH;Of course, we need to actually hook this code into the main flow somewhere, otherwise it will just be so much dead code.
use crate::ghff::HOMER_HEIGHT;
So we add a call to it in kernel_main:
let mut fb = FrameBufferInfo{width: 0, height: 0, pitch: 0, base_addr: 0};Of course, that's never quite enough: in order to pass in a FrameBufferInfo, we need to first declare it and then populate it, which requires us to pass it to lfb_init.
lfb_init(&mut fb);
let homer: &[u8; HOMER_BYTES] = read_homer(HOMER_DATA);
show_homer(&fb, &homer);
This, in turn, populates it when it has the answers back from the mailbox:
let volbuf = &mut buf as *mut u32;OK, so we are ready to give this a go. But I'm bored of running multiple commands to build and run in the emulator, so I'm adding a second script, run.sh which combines both.
fb.width = unsafe { read_volatile(volbuf.add(5)) };
fb.height = unsafe { read_volatile(volbuf.add(6)) };
fb.pitch = unsafe { read_volatile(volbuf.add(33)) };
fb.base_addr = unsafe { read_volatile(volbuf.add(28)) } & 0x3fffffff;
set -eSadly, when we run this, bad things happen:
`dirname $0`/build.sh
cd asm
make run
Compiling homer_rust v0.1.0 (/home/gareth/Projects/IgnoranceBlog/homer_rust)It would seem that the problem here is that some part of the render code uses an array reference which is automatically bounds checked by Rust. In doing so, it calls panic_bounds_check if the index is out of bounds. But this is part of the standard library which we are NOT including.
Finished release [optimized] target(s) in 0.20s
aarch64-linux-gnu-ld -nostdlib boot.o ../target/aarch64-unknown-linux-gnu/release/libhomer_rust.rlib -T linker.ld -o kernel8.elf
aarch64-linux-gnu-ld: ../target/aarch64-unknown-linux-gnu/release/libhomer_rust.rlib(homer_rust-2e5114238dd615aa.homer_rust.f56f16d2546ffac4-cgu.0.rcgu.o): in function `kernel_main':
homer_rust.f56f16d2546ffac4-cgu.0:(.text.kernel_main+0x9e4): undefined reference to `core::panicking::panic_bounds_check'
aarch64-linux-gnu-ld: homer_rust.f56f16d2546ffac4-cgu.0:(.text.kernel_main+0x9fc): undefined reference to `core::panicking::panic_bounds_check'
aarch64-linux-gnu-ld: homer_rust.f56f16d2546ffac4-cgu.0:(.text.kernel_main+0xa18): undefined reference to `core::panicking::panic_bounds_check'
aarch64-linux-gnu-ld: homer_rust.f56f16d2546ffac4-cgu.0:(.text.kernel_main+0xa34): undefined reference to `core::panicking::panic_bounds_check'
make: ** [Makefile:13: kernel8.img] Error 1
The simplest solution is to define it, but how and where? If we can find the mangled name (which has been demangled for the purposes of this error message), we can simply define it in boot.S.
objdump is our friend again, and we can use it to dump all the symbols found or referenced in our library:
aarch64-linux-gnu-objdump -t target/aarch64-unknown-linux-gnu/release/libhomer_rust.rlibYes, that was my reaction too. Not the world's most obvious name. But we can take it and run with it.
In archive target/aarch64-unknown-linux-gnu/release/libhomer_rust.rlib:
...
homer_rust-2e5114238dd615aa.homer_rust.f56f16d2546ffac4-cgu.0.rcgu.o: file format elf64-littleaarch64
SYMBOL TABLE:
...
0000000000000000 *UND* 0000000000000000 _ZN4core9panicking18panic_bounds_check17h75f9d87f2a814b7bE
.globl _ZN4core9panicking18panic_bounds_check17h75f9d87f2a814b7bESo, let's try running it again. And there he is! Homer appears in the middle of the screen.
_ZN4core9panicking18panic_bounds_check17h75f9d87f2a814b7bE:
b halt
But ... he looks a little weird. Sort of off-colour. I'm not 100% sure what, if any, tools can tell you what I've done wrong here, but given how close it is, it's fairly obvious the pixels are in the wrong places. Exhaustive testing would probably indicate the problem, but I did random trial-and-error and then went back and compared what I'd done to the original.
The problem is that we have three significant bytes (red, green and blue) in a 32-bit word (4 bytes) for each pixel. One of them has to be zero. I thought, given that it was really a 24-bit value, the most significant 8 bits would be zero and the three data bytes would be packed into the low 24 bits. But apparently not, the first three are significant and the fourth is zero. Thus, I need to go back and rework the code that expands the GHFF into bytes for HOMER_DATA.
IMAGE[pos] = ((c0-33) << 2) | ((c1-33) >> 4);And just like that, amazingly, we have Homer in the middle of the screen. While it's working, let's check it in as RUST_BARE_METAL_HOMER_APPEARS.
IMAGE[pos+1] = ((c1-33) << 4) | ((c2-33) >> 2);
IMAGE[pos+2] = ((c2-33) << 6) | ((c3-33));
IMAGE[pos+3] = 0;
I say amazingly because, knowing what I know now, it is amazing that this works. More luck than judgment, some would say. I tried removing all the trace statements and it stopped working. I tried it in a real device: no joy. I tried this and tried that, none of them worked. It is amazing how fragile it is. This does not seem like what we would expect.
Conclusion
We have succeeded in building a bare metal program which runs, at least under certain conditions, in the emulator. It does not run under all conditions and not at all on real hardware.It is a mess. As I looked over the code while writing up the description of it, I was appalled at how quickly it has spiralled out of control.
I am obviously not very experienced with Rust, and it shows: I am using all kinds of odd features in ways that I don't think the designers intended. We need to pull that back in.
I don't feel I have a stable methodology for development here.
All, of that, of course, is considered acceptable on this blog: here we support ignorance and the ability to learn from that and to grow and discover things. The big problem is deciding which order we are going to deal with things in. There is so much to do here, and so little time.
For me (at least; speak up if you feel differently), the most important thing is stability: I want it to behave predictably. Most importantly, in a given environment (in this case, the emulator) it should not be influenced by such trivia as whether or not there are debugging statements in the code. Then I want to get it working on real hardware, which I think will involve getting the real UART to work, and only then will I be able to consider what an appropriate factoring might be.
