An Operating System?

Yes, this blog represents the start of a project to build an operating system to run on a Raspberry Pi 3B+.

I hear a chorus of "Why?" and "WTF?" and "Are you serious right now?". Yes, I'm serious, for a number of reasons.

• First off, it comes under the category of "lifetime ambitions". That is, ever since I first understood how computers actually work (which was about three months after I first owned one), I wanted to: write a compiler; write an operating system; build my own computer from components; and design my own circuitry. I have long since done the first and third; while the final one is probably outside my actual competence, never mind the time and expense.
• At University, this desire was reinforced by a course on the subject which featured Andy Tanenbaum's MINIX book; this has influenced my thought process at a deep level since. In particular, the concept of the microkernel.
• The existence of the Raspberry Pi with an ARM v8 A64 architecture using a (removable) SD card for all its storage means that the barrier to entry of building an OS has never felt so low in terms of cost, size, commitment.
• While reading up on building an OS for the Raspberry Pi, I came across an article on building an OS using Rust rather than C. I've wanted to play with Rust for a while, but haven't had a project to play with. This seemed like a good fit, especially since, while I wouldn't say C scares me, it seems very 19th century.
• I generally view myself as a "server side" developer, and often think of the code I write in terms that are best used in discussing operating systems. Thus to build one is a logical next step.
• There are a range of topics I would like to experiment with which require you to be "down in the weeds" of the software stack and that (to me) means starting right at the bottom.
• I generally view the world differently to most developers, and I would like to have an operating system that followed my general philosophy: there are a number of things I want to experiment with in an OS.

As to why I'm thinking of kicking this off right now, a number of factors come into play:

• The fact that it has been a lifetime ambition means that from time to time it crops up and I mull it over; this time when it cropped up, it felt like the stars had aligned.
• I have just acquired a new Linux laptop, which is a much better fit for trying to do these things than it is on a Mac.
• I have a bunch of Raspberry Pis lying around doing nothing, and I was messing with them at a little lower level than usual which triggered the thought of building an OS on one.

I read up on the subject and thought "I can do this".

An OS with a Difference

So, I want to do something different, huh? Like what?

• First and foremost, as you may know, I'm big on event-driven programming. But that usually stops somewhere down near the operating system, where system calls are synchronous and "block". I want to build a completely event-driven, non-blocking OS. One consequence of this is that the kernel doesn't need to be "pre-emptive" as such, because code will never block, but it will frequently "yield" as it has done all it can and the thread of execution will end. The only threads that would need to be pre-empted are ones that are genuinely out of control.
• I would like to make the kernel as small as possible. The plan here is to move as much of the "kernel" code as possible back into user space in "privileged processes" which do all the hard work and only delegate the actual tasks of managing memory and processes along with interacting with devices to the actual kernel running in supervisor mode. I think this is the definition of a microkernel, but if not, I want to build a nanokernel. I want anything that can possibly be running in user space to run in user space.
• I don't want a "file system" as such. I want something that more resembles a run-time memory heap. No, this is not a truly original idea (I used ObjectStore back in the '90s) but I have not seen it done in at the OS level.
• I don't respect the idea of processes as first class objects. Instead, I want to break the server-side code up into small interacting components (dare I say microservices?) and each "request" or "event" (for want of a better word) is executed in its own, short-lived execution context bound to a CPU. The vestigial remnant of the "process" is its state, which is all that exists between requests.
• I have a radical security model in mind. First off, because this is primarily a server context, that model is based on Web Authentication (OAuth) rather than some kind of local password file. And then the security system I plan to build on top of that is radically different too.

Of course, the moment I say "security" (and especially OAuth) you say TLS and it dawns on both of us how much work there is to do here, especially if, as I say, I am planning on writing the core code in Rust, for which I am not sure what code I can rip off. But hopefully at some point in the next few years, I will have a sudden enthusiasm to write some TLS code.

To be clear, I view this as the start of a long, on-again, off-again, enthusiasm-driven project. I'll probably pick it up and put it down over the course of several years, write five blog posts as I implement (or re-implement) a feature, and then let it go dormant again.

Although it would be entirely possible to extend this as far as interacting with the user, for now (say the next five years), I'm only planning on building a server-side operating system; basically, something that accepts IP connections, processes them and sends responses. There are a number of reasons for this, but the main ones are that the HID code does not interest me (beyond literally writing the drivers for keyboard, mouse and display) and the stack to go from device driver to graphical user interface to HTML rendering (not to mention implementing or porting JavaScript) is massive and uninteresting. So maybe along the way I will build a single-user console interface, but a full-fledged GUI is not on the roadmap.

It's important to note that while I will be at liberty to borrow/steal any open source code I can find when I need something, given the set of choices I am making (a completely different core OS, so no system calls are the same; no blocking system calls at all; the code is in a different language to most open source software), I am basically going to have to write everything from scratch or at least do a significant job of cutting/pasting/modifying.

As someone who thinks in terms of Agile development, and in Tell-Dont-Ask in particular, it will be interesting to see how much of my code will end up being testable :-) Especially given how much of an experiment most of this code will be.

Inspiration Projects

I am not the first person to go down this path. As I said, at University I was highly influenced by Andy Tanenbaum's "Operating Systems: Design and Implementation" which spawned MINIX, which, I believe, in turn, heavily influenced the implementation of Linux. Since then, there have been more such projects.

In doing my research for this project, I have come across a number of other projects where people have gone down similar lines and have explained things well and/or have code to share. In fact, if you've come here for inspiration on getting started writing an operating system, they may in fact be more interesting things to read because they are more vanilla in their approach.

• Minix is a fully developed microkernel architecture UNIX-like kernel.
• RPiOS is a starter project (much like this one) which is basically designed around rebuilding the Linux kernel step by step
• Raspberry Pi Bare Bones is an outline of how to go about building an OS on a Raspberry Pi
• Raspberry Pi Bare Bones Rust is the same thing, but - conveniently - in Rust.

These last two projects come from a site which appears to have a lot of useful information about the Pi, although I haven't dug into it fully yet - indeed, it may contain answers to questions I have been wondering about.

• The official Raspberry Pi documentation is, of course, always useful.
• The firmware wiki contains interesting information about a lot of the low level behaviour of the Raspberry Pi, which is, to say the least, a little bizarre. I will discuss this in a later post.
• There are also tools on the same website.
• Mike Krinkin has a number of posts about Raspberry Pi architecture, including this one on how to move between the Operating System and User Space using Exception Levels.
• Cambridge University has a tutorial on how to build bare metal projects. Sadly, it is designed for older models of the Raspberry Pi using the ARM 32 architecture while I am using the 64 bit architecture.
• Brian Sidebotham has put a bunch of interesting information on his website related to the Raspberry Pi, although again it is relatively out of date for what I want to do.
• Another project that appears to be 32 bit but again has a lot of information that I have read and internalized and may well reference in the course of my project. For instance, it has good information on wrangling memory.
• Tiziano Santoro has written about cross-compiling Rust for the Raspberry Pi, most of which is not directly relevant to what we are doing (especially since I have solved this in a previous blog post).
• bzt has a repository which has code for the 64-bit Raspberry Pi 3, which is exactly what I want to do, so an excellent resource, although in C rather than Rust.
• Somewhat late in the game, I discovered that these had been the inspiration for a set of Rust tutorials for the 64-bit architecture, which I hope will be most useful, although so far their complexity - and lack of clear explanation have put me off.
• The QEMU emulator allows you to run resulting programs on your host machine rather than having to write to an SD card and put it in an actual Raspberry Pi. In addition, the emulator has builtin support for the TTY serial port, which makes it a lot easier to follow what is happening. Moreover, it is possible to connect GDB to it.
• GDB comes in a multi-arch version which enables it to support AArch64 code.
• Circle is a C++ project which is intended as a library for use with bare-metal projects. I have to say I'm confused by a lot of how it works, but that won't stop me from stealing information from it where it is useful. As you can see, I was particularly interested in how it reads from the SD card, although it appears to keep delegating the actual work to another class and another project (one of the reasons why I hate C++, and, to a lesser extent, OO in general: it's very hard to figure out what is going on when you are reading code).

And, of course, in order to learn Rust, I read the Rust programming language.

Again, somewhat later into doing my own work, I discovered that there was an explicit book on embedded and bare-metal Rust programming, the The Embedded Rust Book.

The hardcore information about the system and architecture seems to be sadly lacking or inconsistent, although the sample projects often seem to agree on a set of information that they seem to have dreamt up from thin air. Sometimes the information provided is out of date, or contradictory, or has data in it that seems wrong. I am sure that everything is ultimately consistent, but I am missing one or more keys.

Some of these documents include:

• The official guide to the ARM peripherals by Broadcom, which specifically relates to the BCM2835, which is not the board in the Raspberry Pi 3B+, but is basically the same. But it gives a memory base address as 7E00000, when it seems to be generally agreed that it is 3F000000. Presumably this is something to do with memory mapping.
• The guide to the BCM2711 has an overview of the architecture for the various boards, including three different views of the memory layout, which presumably has something to do with the memory addresses.
• The official site also has a hardware guide with lots of useful information about the hardware.

Note that as I find more projects and references over the course of time, I will come back here and update this list and re-post this blog entry.

Bare Metal vs Operating System

Just as a small note on terminology: you will see me (and others) using the phrases "bare metal" and "operating system" and it may seem that they are interchangeable. In some ways, they are, but they are distinct.

An operating system is almost always a bare metal program; but there are bare metal programs that are not operating systems. A bare metal program is one that runs directly from boot on the host system. It does not depend on, expect, or co-exist with an operating system. It expects to be able to interact directly with the hardware using physical addresses and memory-mapped devices. It can read and write any portion of memory and does not need to "allocate" it in any way. These are not features that are usually accessible by ordinary programs, which need the operating system to be an intermediary.

What's it called then?

A lot of the things I'm going to be experimenting with here are concepts taken from my "day job" (I'm no longer where this sits on a scale of "huge hobby project" to "reimplement the web") called Ziniki, so since this is, in many ways, "Ziniki on Bare Metal", I'm going to refer to it as "ZinikiOS" or "ZiOS" for short, which has the benefit of sounding a little like a Greek god :-)

My approach

I am not particularly interested in the normal things that can be done with operating systems, so I am not going to dwell too long on them. I am very much more interested in the things that are different or unique here, and, much like everything else I do on this blog, I will be looking at quick spikes into this and that to see what I can get working and what I can test. As always, I am much more interested in experimenting, recording and learning than I am in getting anything actually done.

Obviously, a lot of what I will be doing is going to be very low level - everything at a low level in computers is incredibly tedious. Especially when it needs to be written in assembler. Hopefully I can manage to at least make that interesting if not compelling.

Apart from all the exploration we will be doing of operating system concepts, there will be a lot of other unfamiliar material here - from Rust to assembler to ARM architecture. Some of this may be completely foreign to you (I'm aware of most of the concepts, but not the details). I'll try and make sure that I explain things as best I can as we go along.

When I'm doing totally normal things (such as device drivers), I will try to point that out (along with other places where people have done the same completely normal things and where to read more about that). When I am doing things in a bizarre and different way, I will attempt to explain both what I am doing and how that is different from a more "normal" way of doing it, and hopefully point you to a relevant example of the similar thing done in the normal way.