most are outdoors behave
on the institutional
there will be no drift
not one
Okay, we're going to start now because I don't want to spend any minute speaking about anything else than this topic.
Hello everyone, I am from, well, I was born in France, but now I have the luxury to live in Valencia
Spain and your city is really freezing.
I am just saying I work as a freelance, I have flying phobia, so it cost me a lot to come here.
I'm using that BSD since 1998, yes, I am that old and I am a net BSD commuter since 2009.
And for those who use it, I am the initial author of Packaging the Binary Packaging Manager for that BSD, but not only also for MacOS, Illumos and so on.
And right now, Jonathan Perkin is the current maintainer and he's doing a magnificent job with it.
Today I'm going to talk about a work that started back in 2006, but the real hard work started three years ago.
I always was passionate about making net BSD small.
I don't know why, that was kind of an obsession I wanted to reduce the size of net BSD.
So first I did this with USBK, 64 megabytes, BSD USBK.
Then in 2016, as everyone was really happy about Docker and containers and so on, I created Sailor, which is not a container system.
It is based on sea of fruit, but it allows you to create a self-contained net BSD system running only what is netted.
But at that time, there was noise about this firecracker thing, something that AWS did back then, which is basically their serverless,
spine, you know about it.
And basically what this is, those are virtual machines.
And those virtual machine boots in something around 100 millisecond, which is enormous in my opinion.
And I wanted to do something resembling to it, and I knew that QMU for those who don't know QMU, I can't do anything for you.
Then QMU for ages as this dash kernel flag.
I had no idea what that meant, but what I knew is that Linux always Linux knows how to be booted right straight from QMU with dash kernel flag.
For years, five years ago now, I worked on something resembling to firecracker.
Basically I trimmed up a net BSD kernel, but a net BSD 32 bits, why?
Because at that time, the only version of net BSD that was capable of booting with a dash kernel flag was the 32 bit one, because the kernel in 32 bits add was called the multi-boot feature, which allows the kernel to boot directly from QMU like this.
But I mean 32 bits.
Recently, two years ago, I started this little really badly-named project called MK small NB, which basically consisted in stripping every bit of driver that's not needed from a 32 bit kernel.
So it can be like life-hacked without having to recompile it.
That was this guy calling Percival.
Who was booting a free BSD system using QMU, actually his work was based on firecracker?
And he was able to boot the in more or less 25 to 30 millisecond.
Not bad.
So, two years ago, one year and a half, I said, okay, let's try it.
Let's see what all this is about.
How does the kernel, you know what, I'm going to put like, so I can see it that I'm not drifting.
Five minutes, five minutes.
So yeah, let's see what all this is about.
What does it take to boot a kernel from the QMU dash kernel parameter?
And I mean, I did very little kernel hacking in the past.
It's pretty frightening.
And I mean, it's not easy.
And I thought, you know what, I'm going to watch what Collin did on a story.
And patch, and that's it, and probably there will be nothing more, and it will work just like that.
Nope. Nope.
I ended up patching the, probably the worst part of the kernel, which is the local.s, which is the assembly part that boots the kernel, you know.
And with this, I was able to boot, if we have time, we'll also see the details of the theory after.
With that, I was able to boot the kernel directly using a protocol called PVH.
We'll talk about that.
But it booted in 200 millisecond.
Please.
So I ended up tweaking, hacking, modifying parts of the kernel that some of them are 30 years old, like when I broke the build, tweaking the serial code.
And I ended up booting a kernel in a normal machine, in a recent machine, in about 20 millisecond.
This laptop is a laptop, it's not plugged to the power, so it has less power than a real desktop machine or server.
So don't do me when it takes more than 20 milliseconds.
And as I really wanted to show what it is about, because I did kind of disparate presentation, it's not the same format.
And I did it very terribly.
I explained the code, what I didn't look for and so, but I had very little time to show what was it about.
So today, I prepared demos.
Most of the presentation should be the work, obviously.
So what is the point of all this?
Why do we want to boot a kernel so fast?
Well, let's see that.
So first, like with no other fanciness than just seeing the kernel and seeing a shell.
I created a small project.
I can, yeah, I'll do it.
Because there's something that I want to show.
Yeah, okay, I'm going to zoom a bit.
Where's that?
There.
Yep.
Okay.
So the project I've been working on now for years.
I renamed it from AMK that I don't want to small BSD.
Small BSD is not an OS.
It's an OS builder.
We're going to see that.
So it's mainly composed of a script of two scripts, which are very easy to use.
You create images.
I already created it not to have any surprise while doing the demo.
And the main star is this guy.
There's BSD small, which is a kernel, which has been built with all the stuff I wrote.
And the rescue is basically just an RC file.
We're going to see what it is about.
And a shell.
Shell being from the rescue directory.
If you know the rescue directory, I guess every BSD has it.
I don't know.
PrebSD, I don't know.
Yep.
So basically it's a directory where you have one crunch by it's called crunch binary.
We're pretty much like BSD box.
Ta-da.
Well, 53.
Let me try something.
Because 53 milliseconds.
Poor.
Let's see if I do that.
But that's a very, very simple OS.
Basically, like I said, the kernel and this.
But now, from there, what can we do?
We actually, from there, we can do anything.
We can create our own operating system.
Let's say everything evolves around a make file.
Yeah.
I don't know if any YAMO file, we know with CICD.
No.
It's plain make file.
And in this make file, well, I have targets which are images.
One of the main images is base.
And in order to.
I'm a friend, no.
In order to make it easy to kill the, the, the, the, the, the, the, the
The machine.
You can pass the make file, mount our O equals.
Yes, which will make the, the, the, the fS tab mount slash in
Read only.
allowing me to just kill the machine and don't care about shed down, out, and so on.
Okay?
So for example, I can do base, I'm looking at you.
I know.
So that's the base file system with every tool you should find in base.
Okay, that's cool, not that impressive, but nevertheless cool.
But like I said, you could create your own operating system.
You could, like, modify in it, or change in it.
Change the way the system boots.
And for example, why not?
Why not create a system image that's called system BSD?
And this is exactly what you think it is.
This is exactly what you think it is.
Yeah, I also created a couple of configuration files.
And yeah, this is systembSD.com.
And systembSD.com uses the init, which is basically an init system, which is pretty fast.
And those, the services with the init command, like init, start, stop, and well, you can do fancy thing like this.
Okay, and so welcome system BSD, which is a, like, a net BSD version, which is not exactly net BSD,
it has another init system.
But after that, it's only net BSD.
You have the same command channel.
I'm going to accelerate a bit.
Okay, we're not going to talk about the internals.
I know we won't have time for that.
So, okay, that's cool.
This five-cracker thing and all the container topic was really appealing to me.
And so, what about starting not just a virtual machine, but a container,
with net BSD inside.
So, for example, if I do, I don't know, that's not the one I want to start.
I want to show you something before, like, instead of just having a shell,
we can start the service, okay, like, for example, an HTTP server.
And that's the only service that will start when I boot boot, the virtual machine,
and this works just like a container.
Okay, but obviously, with a bit more security, because a container,
let me remind you that it is only a system called, and share online, I mean.
Here, we have an entire kernel operating system with an isolated process.
Okay, so yeah, what can we do from there?
Well, as it is very light and fast, well, we can just run it as a Docker container.
And here we go, and it behaves like the same, okay.
And, okay, this was a bit slow, a 66, a minute second.
But, and from there, if we can start it as a container, what atrocity can we do?
Exactly.
As we can start it as a container, well, we can have this awful thing.
We can have that BSD as a Kubernetes pod, okay.
So I have a small cluster on this laptop, where I will create a name face.
Yeah.
And I have a pod manifest that uses the container that I've just created, okay.
So there we go.
Hey, it works.
And yeah, the cluster that's running inside, it's called kind, which is Kubernetes in Docker.
And it explodes, you can query the container by querying the host, which is actually the container itself.
So if I do that, oh, much, I understand, okay, I'm getting on with it.
Okay, so the IP of this guy is this.
So I can absolutely curl this, look.
And it's actually the container running the virtual machine that's answering to my curl, okay.
I did all the demos.
So from there, I assume you have a pretty good idea of the vast possibilities that we have with that.
What I showed here was with QMU.
You saw this.
This works also with firecracker, because the basically the technique used to build the channel is exactly the same.
It's using PVH, which is a system included in Xen, like forever.
And, well, first calling, because let's be clear, the main work has been done by calling festival like three years ago, and 22 years ago.
And his work inspired what I did after that.
Okay, the implementations are obviously not the same.
We had some problems, I had others.
But the boot method is the same, it's PVH.
And actually, it's not that complicated.
Instead of booting in the start entry point of the kernel,
there is a special entry point called start Xen, what it was called, start Xen.
And this entry point uses the information that are passed by the virtual machine manager,
QMU, or firecracker, or whatever.
You don't need, when you think about it.
You don't need any bios, any bookloader, anything when you are starting a virtual machine from a VMM.
You already know how many RAM do you have, what's the disks, and so on and so forth.
So, you will gain a lot of time by just grabbing those information that are pushed by the VMM.
And this is the main point of PVH.
It's using a new entry point to just avoid all the bootloader stuff.
Okay.
I will not go through the implementation details, but because, well, you can see that part of the presentation
on the BSDKN, BSDKN presentation, while I go deep into how it is implemented.
I just want to mention that a part from PVH, which is the boot system only.
After that, there was, yeah, there's a lot.
Yeah, I like this tweet because this is the first time the kernel with my modifications.
It, in it, the part of the kernel.
So, at that point, it worked, and I was very proud, as it showed.
So, yeah, another technique to speed up the boot process, and to speed up the kernel in general.
It's, is to use, implement, and use instead of using like the PCI bus, which is, obviously, in a virtual machine, you don't really need PCI.
Okay.
So, you can use what's called MMIO, which is basically memory mapping, instead of a PCI bus, meaning that instead of using all the PCI complicated infrastructure,
you just map an address, which will be used as the bus between the guests, the host and the guest.
Okay.
So, basically, it's, it's copying data through a structure.
There's nothing faster than that.
Okay.
And, Viratayo has an implementation of MMIO.
And so, one of the big works was to create a dummy bus, which is called VT.
Sorry, in the VSD, which I stole from the OpenBSD, basically.
And this bus permits to plug MMI, Viratayo on MMIO and not use any bus, which will be overkill.
Okay.
So, that's the second big chunk.
After that, and I won't go into the details.
So, that's a little joke that Krayo from the NetBSD crowd.
Each time I was booting a faster, it was saying, faster.
So, that's what I did.
And using various techniques, I killed some delays that were, or useless, or avoidable in the context of a virtual machine.
And this is where I achieved those, depending on the machine from 10 milliseconds to 20, 25.
Someone with Intel i9, I don't know what, achieved 8 milliseconds, which is pretty good.
Okay, I'm going to stop there to get some questions, or...
Okay, so...
Okay, cool.
So, that's about the implementation.
Now, calling the tremendous work in the arena of optimization, speed calculation, and so on and so forth.
And for example, it did something called TSLog.
TSLog is very simple, but genius.
TSLog, what it does, you know that on X86 infrastructure, we have a clock, or more like a counter, which is called RSDT.
Basically, it's every CPU cycle.
But with that, you can know what time is it in your boot process.
And using this, along with function names, you could say, okay, this guy is taking me 15 milliseconds.
This guy, 20, this guy, 100.
And you, well, I implemented it, stole it?
Yeah, I basically stole code.
That's more or less what I do.
And using TSLog, I realized that there were their function that probably were subject to optimization.
And that was the boot without optimization, with only pvh and mmio.
And that's where I came down to, okay.
By, again, killing some user's functions, optimizing other things, putting, you know, sometimes 20 millisecond.
You can gain them just by putting a return somewhere instead of if blah, blah, blah, blah.
Sometimes you break the bills, also doing that.
Okay. Questions?
Have you linked the adding support to NVIDIA's hypervisor for microvm's architecture?
Yeah, that's a very good question.
So, NVIDIA's D as an hypervisor called NVMM works with slower than KVM.
Yeah, I didn't say it, but oh, yeah.
I was asked if I tried it and implemented it in NVIDIA's D's hypervisor.
NVIDIA's D hypervisor, hypervisor is called NVMM.
It works on this machine and on my development machines, I use KVM, Linux KVM.
There's a very good reason for that.
I mean, let's face it.
The world is a big Linux.
And people are using Linux and Mac.
So, I wanted this project to work primarily for Linux and Mac, NVIDIA's.
But, I mean, I couldn't let NVIDIA's D be in.
So, I tried all these things.