Today marks the day I start my journey of writing an OS entirely in Go. I won't explain why I chose to use Go in this post. However, I may dedicate a future post to explaining my plans more in-depth for the OS and why Go is a good choice for it.
Until then, I start my adventure at the beginning of any project. This is the Hello World of Operating System development.
History
An Operating System like Windows, macOS, or Linux loads and runs your favorite video game or word processor but has it ever occurred to you what loads the OS? The answer is the bootloader. But then I ask what loads the bootloader? There are two answers to this. When the computer starts up there are two major pieces of software that will run either BIOS or UEFI. To be clear, neither of these are bootloaders themselves but they do load the bootloader which eventually loads the OS which then runs the web browser you are reading this on.
BIOS is the grandparent interface that stands for Basic Input/Output System. It was originally developed roughly 40 years ago to standardize the startup of the PC so that operating systems could be more cross-platform. Back then was the wild west of computing where each year a new CPU and OS were created. Now things have mostly calmed down. Most commercial PCs run with an amd64 CPU and Windows or Linux for desktop OSs. Of course, there is macOS which has an arm64 processor but writing an OS for that platform is still cutting-edge. It's arguable if BIOS ever did succeed at its goal of simplifying startup since there are quite a lot of inconsistencies between different implementations of BIOS. However, I think it has served us well despite its many warts.
UEFI is what stands on top of the destroyed BIOS rubble. It was originally created by Intel in 2004 but has now achieved widespread adoption by major tech companies like AMD, Microsoft, Apple, and Google (full list of members here). It modernized the interface for the bootloader by providing file access, and a display interface without having to write assembly. Now some like to believe Microsoft is forcing UEFI on users so they can lock down people's computers and kill any competition from Linux. However, I don't buy into such craziness. Microsoft has NeVeR done anything ethically questionable before.
Which to choose?
Now I'm left with a decision. The simple answer appears to be quite obvious. UEFI is the defacto standard for modern computers and BIOS is left at the wayside. So it seems clear cut right? Well, if I plan on having my OS work on computers in the future UEFI is the way.
There are quite a lot of nice things to like about UEFI. Funny enough it has all the hallmarks of an OS:
Printing
Files
Memory Allocation
Displaying Images
Bluetooth
Wi-Fi
Another nice thing about UEFI is that its executable is the standard Portable Executable (PE) format or better known as .exe. This means that making my operating system self-hosting will be easier since the standard Go toolchain can build a PE file by just setting GOOS=windows. Of course, this is a crude oversimplification of it but it is possible.
So what do I need an OS for if UEFI provides all the hallmarks of an OS? These services are only available at boot time. They are convenient functions to make the process of loading the OS easier and to play that sick macOS startup tone.
Although UEFI is super nice and all, I don't want to completely neglect older hardware. E-waste is a serious issue and if my software can bring new life to a dying machine, that's a win in my book. So it seems like there's no simple solution.
Bootloader
What does one do when there are two conflicting standards of doing things and no easy way to merge them? Abstract. So the goal is to create a bootloader that runs on both BIOS and UEFI systems. It would be even more awesome if it could do it in the same executable! Well, such a thing does exist!
Since I'm lazy and just want something displaying in my emulator as soon as possible, I'm going to choose an already-written bootloader. There are a lot of options to choose from. The de facto standard is GRUB which is a GNU implementation of the multiboot standard. If you've ever installed Linux on your computer you have most likely used GRUB as the bootloader.
However, I don't think I'm going to go with it for two reasons. First, it is quite bloated and has a lot of features that I don't need. This will make it more difficult in the future to port the bootloader to Go. Second, the multiboot standard is hard to implement in Go because it requires a special struct to be located in the first 4KB of the binary to boot but there is no easy way to guarantee where Go places its symbols. On top of that, Go executables are much larger than 4KB. Even a binary that didn't even have any real Go code was still 5KB. There are more valid reasons not to go with GRUB and other reasons to go with it but I'll leave that to people who know more than I do to explain.
If not GRUB which? The two that seem the most actively maintained and supported are BOOTBOOT and Limine. Both only support 64bit higher-half kernels which are completely okay with me. I don't plan on supporting any older CPUs anyways. They also both support ELF binaries which the go build command supports natively by setting GOOS=linux. The creators of both of these bootloaders are also active and helpful.
Go Code
It's time to start writing some Go code. BOOTBOOT already had a Go example but it was not idiomatic by any means. It didn't use types, no go:embed for the font file and required a small assembly. Here's a small sample:
var w int = (int)(*(*uint32)(unsafe.Pointer(uintptr(BOOTBOOT_INFO + 0x34))))
// bootboot.fb_width
var h int = (int)(*(*uint32)(unsafe.Pointer(uintptr(BOOTBOOT_INFO + 0x38))))
// bootboot.fb_height
var s int = (int)(*(*uint32)(unsafe.Pointer(uintptr(BOOTBOOT_INFO + 0x3c))))
// bootboot.fb_scanline
That's some ugly code. This comment from the author describes much of their distaste for the way it had to be written.
/* Go sucks big time:
1. no include (WTF they dare to call this a C killer language without a pre-compiler?)
2. no union support nor pre-compiler conditionals for the arch specific struct part (no comment...)
3. as soon as you declare a type struct (not use it, just declare) Go will generate tons of unresolved runtime references
4. importing font from another object file? Forget it... neither CGO nor .syso work for non-function labels 5. accessing a linker defined label? Forget it... Use constants and keep them synced with the linker script
6. even the "official" bare-metal-gophers example on github miserably fails to compile with the latest Go compiler...
7. if you finally manage to compile it, the resulting file is going to be twice the size of the C version! So we do dirty hacks here, only using pointers to access bootboot struct members, addresses by constants instead of linker provided labels, font embedded in a string, and we need an Assembly runtime too... */
Although the comment appears to be hating on the language itself, these problems manifested because the author used GCCGO instead of the standard toolchain. I find that quite annoying that the GO example used a C tool. Especially since my goal is to create an OS that is self-hosting which means I need to port the compiler that builds my OS to my new OS. So if it uses GCCGO means writing C and I hate C. This means there is no choice but the standard Go toolchain. I plan to fix up the BOOTBOOT example to use the standard toolchain.
Entry
The entry is the first piece of code that runs inside an executable. If you're familiar with writing Go and wanted to know where the program would start executing you'd look for where the main function is. However, this is not the first piece of Go code that runs. Go has a runtime that needs to be initialized. This includes things like creating the first goroutine, calling all init functions, and allocating the first slice of memory. Where then is the first actual CPU instruction executed in a Go binary? It's different for each GOOS and GOARCH. But they all have the same format which goes like this _rt0_GOOS_GOARCH. This is an assembly function defined in a file of the same name. For example, in macOS on arm64, it'd be _rt0_arm64_darwin inside rt0_darwin_arm64.s.
Now writing assembly isn't very fun to write nor is it portable. I want to write as much in Go as possible so the "Hello World" of this OS should not require any assembly to run. This should be easy, right?
func _start(){
// Code goes here
for { /* loop forever */ }
}
The Go linker has a flag to set the entry of the program.
-E entry
Set entry symbol name.
It appears to be exactly what I need. However, there is one issue. Limine and BOOTBOOT both require a special format for the ELF binaries. They accomplish this by using a linker script. What is a linker script?
A linker script is a file that tells the linker which sections to include in the output file, as well as which order to put them in, what type of file is to be produced, and what is to be the address of the first instruction.
The Go linker doesn't support linker scripts. This may seem like a death blow to my intentions to use the standard Go toolchain but it isn't. What if there was a way to call an external linker to link but use the normal Go compiler? There is! It's just three flags to the Go linker:
-linkmode mode
Set link mode (internal, external, auto).
This sets the linking mode as described in cmd/cgo/doc.go.-extld linker
Set the external linker (default "clang" or "gcc").-extldflags flags
Set space-separated flags to pass to the external linker.
You may not invoke these directly very often but they are used by the link tool when you use Cgo.
Linker
Of course, using an external linker will cause issues when I eventually attempt to get the OS to be self-hosting. However, I plan to add linker script support to the Go linker once that day comes. For now, I'll use an external linker to get started on my OS.
MacOS users that have Brew installed will find it super simple to get the linker; it's as easy as one command.
$ brew install x86_64-elf-gcc # or aarch64-elf-gcc for RPI
Note, that even if you are running on an arm64 Mac you'll want to use the command above. This is because I will be making an x86_64 OS since it is the most prominent CPU architecture. However, if you do want to develop an OS for an arm processor (like a Raspberry Pi), you can use the commented-out option. Unfortunately, GCC doesn't natively support cross-compilation like Go. ☹️
For Windows and Linux users, you can follow the instructions on the OSDev's GCC Cross-Compiler page.
Now it is time to figure out which flags I need to pass to the external linker to make everything build properly. I copied the flags that were originally passed to the linker but created issues due to unknown flags. The reason was that the Go linker was trying to link as if it was using Cgo and therefore assumed certain flags were needed. Here's what the final working linker flag looks like:
-ldflags="-linkmode external \
-extld x86_64-elf-ld \
-extldflags '-nostdlib -n -v -static -m elf_x86_64 -T link.ld'"
I won't explain most of these flags because it's possible to read what they do on the LD documentation. However, I did want to point out the two that are necessary to build. They are -static and -m. The first ensures that we are creating a static executable instead of a dynamically linked one. Since I'm making the OS, there is no dynamic linker. The other one tells the linker what type of emulation we want. Originally, Go would set the emulation mode to just 64 which would cause this error:
go1.20.2/pkg/tool/darwin_arm64/link: running x86_64-elf-ld failed: exit status 1
x86_64-elf-ld: unrecognised emulation mode: 64
Supported emulations: elf_x86_64 elf_i386 elf_iamcu elf32_x86_64 i386pep i386pe
make: *** [mykernel.x86_64.elf] Error 1
Magical Pragmas
Now I've successfully built the kernel! 🥳 However, if I attempt to run it inside of QEMU all I'll get is a rebooting program. This is caused by a triple fault which is bad news.
When I take a look at the output window for the build command I see that there are some warnings.
# gitlab.com/bztsrc/bootboot
loadinternal: cannot find runtime/cgo
GNU ld (GNU Binutils) 2.40
x86_64-elf-ld: warning: cannot find entry symbol _start; defaulting to ffffffffffe453c0
x86_64-elf-ld: warning: $WORK/b001/exe/a.out has a LOAD segment with RWX permissions
x86_64-elf-strip -s -K mmio -K fb -K bootboot -K environment -K initstack mykernel.x86_64.elf
x86_64-elf-readelf -hls mykernel.x86_64.elf >mykernel.x86_64.txt
Let's break this down. The first line is just saying that there is no runtime/cgo package. This is expected as I don't want to use any C in my program. However, the warning is there because I am using the external linker which is usually used when Cgo is enabled.
The next warning is the reason for my issues. Why can't LD find my function since it is named correctly? The cause of this error is that Go function names are a combination of the package they reside in and the name itself. So this function is not just _start but instead main._start. I updated the linker script but I'm still getting the same warning. Why?
My issue boils down to creating a Go function that can be visible outside Go. This is different from a normal exported function (capital first letter). It is possible to export a Go function for use by C code. However, the normal, //export comment requires that the import "C" statement be included but this will force the use of Cgo. I need a solution that doesn't require any C.
Cgo is made up of two parts. The package part is found in runtime/cgo. The tool is not a part of the compiler but a separate program that runs as almost a preprocessor to any C code found above import "C". So there must be a solution. I ran go tool cgo on a simple program that contains an exported Foo function for use in C. Here is what gets generated inside _cgo_gotypes.go for Foo.
//go:cgo_export_dynamic Foo
//go:linkname _cgoexp_7b76dae8e671_Foo _cgoexp_7b76dae8e671_Foo
//go:cgo_export_static _cgoexp_7b76dae8e671_Foo
func _cgoexp_7b76dae8e671_Foo(a *struct {
}) {
Foo()
}
What are these magical comments above the function? These are called pragmas. The one we care about is cgo_export_static but what does it do? Well, inside the design doc for Cgo, it reads:
//go:cgo_export_static <local> <remote>
In external linking mode, put the Go symbol named <local> into the program's exported symbol table as <remote>, so that C code can refer to it by that name. This mechanism makes it possible for C code to call back into Go or to share Go's data.
Using this pragma doesn't require Cgo at all. It's implemented entirely in the Go compiler here.
If you are curious as to why this is needed at all, it is because Go by default doesn't export symbols. This can be verified by calling go tool nm on our built kernel.
$ go tool nm mykernel.x86_64.elf
ffffffffffe9e9a0 t _start
ffffffffffe99100 t aeshashbody
ffffffffffec4ce0 t bad_cpu_msg
ffffffffffe9d660 t errors.(*errorString).Error
# more omitted...
What's important to focus on is the lowercase Ts. This means the symbols are unexported and therefore not visible to the external linker. This is even the case for functions that are considered exported in Go (starts with a capital letter). When I add the export symbol comment the T next to _start will become capitalized and that linker warning finally goes away.
Now the kernel still won't work. I'll need one more pragma: //go:nosplit. This pragma is described in detail in Dave Cheney's Go's Hidden #pragmas post. The final version looks like this:
//go:cgo_export_static _start _start
//go:linkname _start _start
//go:nosplit
func _start() {
for { /* never leave */ }
}
Compilation Errors
Now I kind of lied. If you try to build the kernel you'll get this lovely message:
./kernel.go:78:3: //go:cgo_export_static main._start _start only allowed in cgo-generated code
Hmmm. Many people may consider this the end but I was not going to give in to failure. I was so close to being able to build this kernel. So I dug into the compiler to figure out how it knew this code wasn't a cgo-generated file. It all boils down to this tiny snippet of code found in noder.go:
// For security, we disallow //go:cgo_* directives other
// than cgo_import_dynamic outside cgo-generated files.
// Exception: they are allowed in the standard library, for runtime and syscall.
if !isCgoGeneratedFile(pos) && !base.Flag.Std {
p.error(syntax.Error{Pos: pos, Msg: fmt.Sprintf("//%s only allowed in cgo-generated code", text)})
}
I'm going to ignore this comment about security since I am the kernel now and rules don't apply here. However, please don't use this method in normal code.
So how do I get it to work? The function isCgoGeneratedFile checks to see if the name of the file starts with _cgo_. However, I can't just change the filename to start with that because go build ignores files that start with an underscore. And I don't want to list every Go file in its invocation. Ok, so how do I make this base.Flag.Std be true?
It's just a simple undocumented flag that gets passed to the compiler. I just went ahead and added this to the build command.
-gcflags=gitlab.com/bztsrc/bootboot=-std
This command works by telling the compiler which package is the standard library. This won't affect any of the real standard library functions; it just tricks the compiler into thinking we too are the standard library. The package must match what is inside the go.mod file.
Conclusion
This post is already long enough and isn't a tutorial as much as it's a place where I can share the wacky parts of the Go internals. I went ahead and finished the porting process of the bootboot.h file which contains all the structs needed for creating a BOOTBOOT kernel. I also ported the limine header and barebones example to Go. The limine one hasn't been made official yet since there is no way to force Go to use soft floats which is a requirement for limine but not BOOTBOOT. But maybe in the future, it will be official. If you want to test the BOOTBOOT example, my changes were merged upstream and can be found in the official repository. If you build and run it you'll see this:
Thanks for reading! I hope you found this post educational and that you'll continue on this journey with me to build an Operating System in Go.