Hello everyone, I'm Nick, thank you for being here.
So I'm going to talk about a very metal programming on RIS5 and a few cool stuff, so
how they'll start it for us.
I work at 4th and we are involved with lots of European projects and we bring up RIS5
prototypes.
So we get a lot of experimental stuff and we have a sub-seize, we have a bit of frames,
we have a lot of different components and we do integration and we do also bring up
so our hardware team gets all the different parts, puts them together and the software
teams needs to make this thing booked.
So usually when you win the hardware team when we get an IP, we expect it to be verified,
you have past all the compliance suits and we expect a RIS5 core to follow the spec.
This doesn't always happen and even when it does, there are still bugs over there.
There are a few horror stories I can share with you, like okay, compressed instructions,
kind of not-don't work because of the alignment, the MFU is not really working, the branch
predictor, we have seen crazy things and point is that even that you need a lot of testing
and the standard test, I mean the simple test that are out there for just compliance and
not going to cover all of this, so you need to develop your own tests.
Both for testing the IPs, like the hearts, the RIS5 hearts and then you need to slowly expand
your test cover because it's not just the RIS5 hearts, you also have an intercontroller
there, you have like, I may have an IMMU, you may have other other stuff on the system that also
need validation and so for example, when you go out you may, we have a case for example,
that things that are messed up in the bus, like we had to, we used case where the NVMe was
looking to the outside, like another case where our DRAM was messed up and we didn't have memory
because the DRAM traces were wrong. In case testing is really important and as soon as
we cut the bugs, the better, because when you miss something and it goes to the next stage,
it is harder to fix and once you go tape out it's basically game over, like it becomes
super expensive to fix the bugs when you move on, so the earlier you cut them the better.
So what we started was we would be develop our own test for a platform level test, mostly
like a litmus test, for example, for the memory subsystem to verify casco currency with
other devices or other tests like interrupts at the time in the timers and stuff and all
this simple test became a bit of a mess. So we said, maybe we should have, we should create
a simple framework so that we can write bare metal tests for our stuff instead of starting
from scratch. So that's how the bare metal framework started and we wanted to make it a
bit more flexible so that we could grow the test coverage even more so that we could have
write more like being able to write simple test but also expand it to write more complicated
tests. And let's then we figured out that you know what, basically now we have some
we have a framework and kind of an SDK to run bare metal applications and let's use it
for other stuff as well. Let's use it for bugs for profiling or benchmarks, because this
is cool stuff, like when you're running benchmarks on bare metal, you don't have any
noise from the operating system, you can get very accurate measurements and you can
profile your hardware very efficiently. And then when we cleaned it up and it become
readable and presentable, it all works so that this was a good tool for education, because
we are in research center, we work closely with our university and it seemed like a good
way to demonstrate to students like how does like, because when they see the first time
an operating system class, they end up taking out Linux or some very complicated OS and
they get lost in all this mess, like how is the timing subsystem works, how do we get
how the scheduler gets interrupts, how do I have the privilege mode, how do I switch from
one to another, how do I system colors, they don't see that, so we figured out that if
you have a good start-of-arms or something very simple and clean, it's also a good
use, a good tool for education. So we started this, we joined the RIS 5 Foundation
like 2018 and we were already involved with such kind of projects and by 2019, we kind
of this thing started to become a thing and it evolved and it's now in a stage where
it's okay, I guess, to release, so it supports the standard RIS 5 extensions, it supports
AIA, so it has support for the previous generation of Indira controllers, like the clean
the click and then the click was split into, you have the M timer and the device for sending
IPIs, this is the hack-lint and it also has the click an app link and EPSIC, so you have
all, you can support from all the RIS 5 systems that had only clean their click to more
RIS and ones that have EPSIC, an app link and also support them in size, it has SMP support,
we had this because we wanted to play with the memory subsystems also, we had to support
multiple hearts so that we were able to do, you know, a concurrency test between them and
complex memory layout, so I'm going to come to that later and I also wanted to make this
simple for hardware people to play a piece of code it runs and this is the job of this step,
step it's to prepare for the next boot stages, fetch your first days boot loader, your firmware,
jump there and then the next boot stage will fetch the kernel and move forward, so this thing
needs to fetch these images from somewhere, in a normal system you get this from a flash,
in our case you may not have a flash, when you have an FPGA there you usually have an RJ45,
you have connectors, you can plug to a network to a switch, but if you want flash you actually
need to get a flash chip and put it in the FPGA you have it like a separate module, so it's more
frequently to be able to have network there than being able to put a flash also, if you have network
you also have the whole stack, but this is about the net boot thing I'm going to talk later,
the thing is that when you write a boot room you may have the memory layout, you don't, you may
not have DRAM yet okay and even if you have DRAM, your DRAM address space may be very far away from
wrong or your RAM, your restaurant for example or your scratch partner whatever may be very far
in terms of physical addressing may be very far from your code, now when you're in a regular
application the operating system loads you and the memory layout is virtual memory, so everything
is next to each other you don't have this issue, but when you're restricted by the memory layout
if your text if your code is very far away from your data the linker will not be able to resolve
the symbols because we have two ways basically to resolve symbols, one is PC relative, so basically
you have the program counter and you can see you can find any symbol that's before or after
two gigabytes from your program counter, so if your RAM region is like four gigabytes away from
text you won't be able to put any symbols in there, then you have GP relative, so GP relative
basically you have an absolute address somewhere, you place you have a GP registered a global
pointer, you put it there and now you can resolve symbols that are plus or minus two kilobytes
are think from the global pointer, so you have you can have four kilobytes of symbols any
anyway and anywhere in your physical memory, but you're like you're strictly the only four
kilobytes, so there is a large memory model that tries to solve this, but how how do this
you solve this basically, instead of having the linker resolve all those symbols that are very far away
you grab the addresses, you put them as values somewhere, let's say in rodata or in text,
you store them and when you want to find them you load the address from rodata that's close to your
text, so basically instead of resolving those addresses as a symbol, as a from the link table
you grab them as values from the you store them, you store their addresses the banner,
so you resolve them at runtime instead of resolving them on the link time, that if you follow the
larger memory model which is not ratified yet, it creates, it adds addresses for every symbol in there
I mean from last time I checked, so it adds their functions like symbols, so it grows a lot
and a butrum needs to be very small because we are constrained in size and it also last time I checked,
it added those symbols in the text segment and not in rodata which to me is a problem because
I want to be able to have clear separation between code and code and data, so basically what I did was
do it manually and just make sure that I don't exceed those four kilobytes, so I have the data
and BSS and I can put them everywhere anywhere in physical memory, but it has to be
there's on 4K which is doable, so you have to introduce a malloc there to if you want for example
when you do networking or other stuff and you need large buffers instead of having them as global
variables in which case they land up in that and BSS and you want that's flat space, you use malloc
and you have them allocated on runtime which is world wars without a problem, so that's one of the
problems that we had to solve, the other is flexibility, okay, prototypes are still working progress,
so even if you're doing an FPGA design you still need to put this in the bitstream and it will take
area there, so it cannot be too much, you have like to be like a few kilobytes,
the restriction I got from my hardware team was like 30 kilobytes, you have this higher limit,
anything you do, you cannot exceed 32 kilobytes, so that's the pain, how about the joy,
the joy is that you have full control, okay, bear metal, nothing else, like no no is, you do everything
from scratch, you don't need to deal with weird abstraction layers, I mean if you work in
learning of scanner, you have to implement like you need to do right a simple driver there and you
need to attach to all these interfaces and stuff, no, this is dead simple, sorry, yep, so it's it's
cleaner, right, you have full control and it's also cleaner and another cool thing is that because
respite specs define an ecosystem and they are consistent with each other,
it's the respite spec is simple to follow, I mean we don't have like a completely complicated
stuff, it check out the interrupt controllers, check out the the trap handling, okay, it's not that
too, it's not, I mean you can implement all of that, you can support the whole risk five set of
specifications without too much complicated code, so it's the follow and for example, if you go to,
if you have to integrate this to a more generic operating system, because of all these abstractions,
you miss this simplicity, okay, because this needs now to, you have a very simple interrupt controller,
but this now, this driver needs to be part of something that can also handle some really complicated
stuff, so you're missing this simplicity, in this case, because things are simple, you can keep them
simple, it's also a great opportunity to have fun to stress your skills and experiment,
because you have like, access it, full control and full visibility and you don't have any noise
from from the rest of, from like, other applications running there on the application system,
so you want you to profile in, for example, you can get very accurate measurements when you're
doing multiprocessing, you can explore very, very interesting things, and again, in case you work with
students, really, just working in bare metal, just just exploring, this is a great teaching tool,
so, I use case for that, I'm pretty, pretty mature, that in an FPGA, you may have network,
but you don't usually have flash, because you flash is an external device, you need to actually
attach flash there, because, you know, to be persistent, the beach trim is not persistent,
but you can have a network card in there, and with your network card, you can connect to some persistent
storage, and actually, if you connect to the network, you can get everything, you can fetch your
both images, you can fetch your kernel, then you can mount your with a fetch over NFS, you can
put a whole Linux, you can also have internal access, you don't need storage, right, with network
thing, you have pretty much everything, and actually, networking is usually faster than flash,
so, if you just go out, you have like a, you can get a little bit of SFP connector there,
you can fetch your both images like, like this, if you go through SPI to fetch them,
much lower, even with a logic in there, so, it's much more flexible, and I think if you're
developing stuff, if you're playing with embedded boards and SBC, you've probably ended up in
this situation where you have to plug and unplug as the cards all the time, it's been,
or you have to wait for the problem to refresh them, and it becomes a mess in this situation,
you just, or compile your stuff, you get a booted, and it will come and fetch it,
you don't need to reprogram any flash or wait for it, so, I started working on this bootload,
on this bootroom that has network capabilities, so that it when instead of, because we didn't
have any storage and it would be useful, so, I had a support for Immaculate, because it was a
simplest Ethernet interface, we could put in there to play with, it's only up to 100 megabits,
but it was something to start with, I added support for our custom, one gigabit,
Nick, which is based on an accident, from Zylings, and I, later on, I added Virtio,
so that I can run all this thing in chemo and support Virtio net. So, what this thing does,
it's a different block size, when you talk into DHCP to the TFTP, the window size allows you to instead
of acknowledging every packet, you acknowledge a group of packets, and the T-size is an extension
that you ask the TFTP server to tell you the size of the file, this is optional, the block number
under, I have heard that right there, because there is a gap in the TFTP spec, they have this
to bite for a thing for a counter for the block number, and they didn't really define how
this will wrap around anyway. So, this is about the networking stuff, this is what gets you
the image, and now when you get the image, you have to parse it, the reason is that when you want to
put a system, you probably want to put Linux, for example, you need, let's say, open its BI,
but you also need the device tree, you probably need more than one image, right, you don't,
you cannot think, you can compile things in, but then it's not flexible, because you need to
rebuild the whole thing. So, I created the image container format that can have many
partitions, and I have an image parser there, and it has also compression, so it's much
can be compressed, which makes sense, because the whole, if you add open its BI and the kernel
amount, you might just be like 17, 20 megabytes, just tells it for, which is a very simple
decompressor, it's like 90 lines of code or something, you can get down to 5 megabytes,
and I have some integrity checks there, and I'm also preparing this to support secure boot,
and he mentions, I'll go, what, I'm still having this limitation of 32k, right? So, here,
I have a breakdown of how long it's of this part, where I'm right now. So, I am with everything
in there, so this thing connects to the network, it has the DHCP, the FTP, it can figure out the
file name to the DHCP, it will get they much, you know, unpack it, unzip it, verify it, and all of that,
and it's still less than 32k bytes, right? Now, it's 32k bytes with debugging information
and colors and everything. So, if I remove that, and I add a OS and deal with some of the
compiler mess, because when you have OS and LTO, beautiful things happen, it can get like 20 to 20
something kilobytes, and I have space there, I have like a 10, 8 to 10 kilobytes, and the goal is to
execute boot with that remaining space, and of course, I don't have enough space to actually
write the crypto in there, but I have enough space to write a parser, to write a proxy to
Kaliptrek, Kaliptrek is a root of Tras mode to the open source mode, it has a mailbox,
and one of the mailbox commands, it is to verify signature, so I just get the, I can just
hash, they much give the hash and the signature and the public key to Kaliptrek, and Kaliptrek
will do the verification for me, and I believe I can still fit that in there, and I also have
some space left to also support flash, so that I can get things for flash if you don't want to
fetch your images for network, and I will use the same image parser, so I will still be able
to do secure boot with, with that, so goal is to do, to finish all that, so I'll secure boot
in there using Kaliptrek, and still keep it less than 32k, and as a back end, when we do not
have Kaliptrek, this is for my fun, I'll try to add the crypto there to do verification myself,
and keep it under 64k, but I'm not there yet, so that's it, feel free to grab this, have fun,
any bugs please report them, and I hope this is also useful to you and others,
if you're playing with this kind of stuff, I believe it would be useful, I have a demo of you
one, we have a few time left, yeah, right, how do I do that, that's good, better, kind of,
if you have an implement that's here, so that's because I support multiple extensions,
some of the hearts may not have all of them, you get illegal instructions, I have to
skip them, because it's normal, this is the malloc, allocating stuff, and the HP starts,
it gets an IP, I get the server address from the DHCP, I ask for the, I got the boot
too much from DHCP, so DHCP gave me that file name, if I try to open it, it fails, I'll try
another myself, and negotiate the block size, blah, blah, receive it, and jumps, so, so I receive five
megabytes, the whole thing is like 20 something megabytes, it's five because it's compressed,
and this is a kernel, an open BIM, that also contains kernel as a payload, and kernel contains
an inner trauma phase as a payload, so that's why you see these books here, and that's it,
so, any questions?
Why not use FIT as a much format, because if you see the parser, the FIT parser, I again,
30 kilobytes, I have to feed everything in there, my much parser is less than 300 lines of code,
and it has CRC32, on every header, it has separators, I support multiple units, multiple
partitions, I have, I mean, it's better, and it also has, like, it is ready for secure boot,
so I also have, you know, flags for crypto of functions, like sizes and stuff, so, I did, I mean,
much better, and it's actually the header format is like 8 bytes, every header is 8 bytes,
everything is 8 bytes aligned, so that's it, I can simplify the parser, not a problem there,
it just fails that the compressor is configured so that it works per byte, so that I can align things,
anyway, I've done some hacks in there to simplify the code and it's really, it worked,
so yeah, no FIT custom stuff, but I have a Python script that will generate
very much for you, CloudRode it for me, yep, I couldn't find anything that was that small,
and after all, I used to do networking stuff, ages ago, so I enjoy writing network stuff,
it's very simple, I mean, it just type P plus GdP plus R, it's like, again,
430 lines of code, it's, and like you can get it in less, I have some, it's because you can skip
CR6, and you won't, you can like, I, I, all of this, it's because I am, I'm a bit paranoid with
being strict with a spec, like the DHCP, it could be like a hundred, a hundred, a lot of
less if I just ignored validation in the client side, I'm a bit paranoid, so I, you could save space,
not TCP, you don't need TCP, TFTP and DHCP are just UDP, so, anything else?
Okay, well, thank you, this is, this is the projects we are working on, and you are all
welcome to the EU summit, so I, this, I'm working my program committee hard.