All right, and it's time to go.
All right, so next talk is going to be around net but without throwing a fit.
Yeah, so hello everybody, and welcome to my talk.
My name is Ahma Pratung, I work for Pinguotronics, we do embedded Linux consulting and development.
And something I do a lot at work is network-booting colonis.
Yeah, network-booting colonis.
So where I usually do it is I extract the root fs into an NFS exported directory, optimally.
The root fs is already self-describing, it has a kernel, it has a device tree, it has an inner drum
fs, and it has a bootloader spec that ties it all together.
Here is an example, so it has a 618 kernel, it references a device tree.
And then I can point in this case the barbox bootloader added and booted.
Unfortunately, this doesn't always work out of the box.
Because I usually extract with out food, so the user ID and the ID are all wrong.
And all these executables with sweet bits, they won't work,
because they will try to assume the user I have on the development host.
And not root for example.
There are ways around that, you can page, if you are using root for everything,
you can just remove the sweet bit and it mostly works.
You can run it in a fake root environment.
Yeah, so there are work around, another problem is NFS being networked.
That means that you need a lot of special casing.
For example, if you have ethernet switches, you will have to disable some system
ethernet work deal with, but you can also work around that if you have a USB host controller,
user USB ethernet adapter, if you have a gadget device, a gadget controller,
you can since kernel 612 configure USB 9PFS, which I presented on was my colleague Michael last year here at first them.
What's more difficult to handle is when the user space just says no,
because it's very set in its way to expect a boot block device,
or it's an AB system and it expects to have an active partition.
What you need in to do in this case is look at all the inner scripts and other services
and make sure that they would tolerate being run under NFS hood.
Once you add some verified boot scheme into the mix and then you have device map
or configuration for the M-Varity, the M-Priptium Integrity and so on,
you only get more services that depend on a specific block order,
a blocked device landscape to be available.
We also usually lose this nice aspect or with bootloader spec where you can have
a partition that's fully self-describing, because you will have a signed kernel,
bundle with everything in it and it will usually be in a separate partition and so on.
And the end result is that everyone needs to take care to keep the network boot
workflow working or it's just not there and this bothered me a lot in some previous projects
because I just wanted to network with the kernel, they back some kernel issue to a bisect
but always network boot is bitwating because it's not the usual way the system is booted.
So I wanted to rethink the issue, what's the minimum viable payload that I need for kernel boot
and very importantly I don't want to mess with the bootfs at all,
I don't want to mess with any OS-based system like Yachter or something,
I just want to build a kernel, I want to be able to network boot it and leave everything as is.
So I am usually working with arm systems and they usually probe non-discoverable devices with
device tree, so what I needed is the kernel, the device tree and the modules.
As bundle format to put them all in, I used fit, so fit is flattened image tree,
here's a link to the specification, basically you have a number of images and in the same fit file
you have a number of configurations that references this images.
Images can be a kernel, random disk device tree for different boards and what the bootloader is
going to do is that it will look at the compatible property for example inside the configuration
and compare it against its own compatible for the bootloader.
So I am on a ratcha rock 3A board and it will take the first configuration, it will check
out and free scale I'm 8mm, that's not a match, so it will try the next configuration,
then it sees a ratcha rock 3A and the configuration is also a ratcha rock 3A,
so it takes that configuration and in that configuration it will find a kernel, a ram disk,
and a device tree and it's going to load these three things.
The next thing about fit is since 10 it's supported in the kernel, so for arm 64 it can
just generate a fit image, the fit image that's generated will contain the kernel and it will
contain all enabled device trees, so you could just take the normal depth config for V8 and you
will have a very big fit image with, I don't know, thousands of device trees, but it should be
able to boot on all of them, then there is still the issue of modules, modules are usually
in the bootfs but we can choose the modules in the bootfs because they will be mismatched to the
newer kernel that we are going to boot and we don't want to build everything into the kernel
because we might have dependencies on firmware that's in the bootfs for example, so we just want
to overlay the modules and Linux has a nice mechanism that allows us to do this easily
which is its handling of concatenated CPIOs, so you can have a number of CPIOs, CPIOs,
the format for inetram disks or inetram fs and you can compress them individually and just concatenate
them and what the kernel is going to do, it will check the first inetram fs, decompress it
if needed, extracted in the initial ram fs and then takes the next one and so you can on the command
line just to the cat, composite your inetram fs out of individual components and for
619, an upstream target, it's called module CPIO package and it does what it says in the
tin, it's inetram fs with all the modules and it's your usual directory layout with layer
shaped modules and so on and modules order and all these files and there is in parallel a series
by Simon Glass who added the image fit target in the first place which allows adding
run disk support to the fit image and yeah now we got kernel, device tree and also modules in
the inetram fs and what's missing is something to load the modules and we can also make use of
this concatenation mechanisms or if we already have run disk in it we just need to add a bind
mount into it so here for example if I have a shell in my inet adi I can say mount minus oh
bind lip modules which is a lip module is inetram fs and bind mounted over the lip modules in
the new root file system that I'm set the inetram fs is going to switch to if you don't have an
inetram fs available or you can easily add a line my colleague Stefan Kirkman added to our s inet
our s inet is a very minimal run disk inet that we wrote for some of the embedded systems
that we are developing on and the soon it should also have the ability to bind mount
folders from inet adi including the module directory and we can get that in with exactly the same
concatenation mechanism so here is a shell script that ties it all together I build the kernel
with my make all I build module cpi oh package by compress this modules inetram fs I concatenate
it was our s inet are as inet has a make file that builds cross compiles it for different architectures
and then you have a safe contained inetram fs linked against muzzle and then we make image fit
and an extra argument you get a fit image out that contains all these components modules
kind of device tree and any inetram fs extras you want to add last missing piece here is
we miss out on the staff's a bootloader is doing so usually you have quite some bit of complexity
in the bootloader in the way that it adds command line argument it applies device tree
fix ups for example it was bootloader spec you have an options key and you would lose out on that
if you are just using to boot fit image so some bootloader integration would be nice in
barbox I added this in the form of overrides so what's overrides can do now
if you check out my branch I'm still in process of upstreaming it you can tell barbox boot
as you would usually be it bootloader spec be it fits fit fit be it boot script but take from
this fit image this image just as overrides and apply them and then you can replace the kernel
image out of it fetch and you can so append the inetradi on the fly and a very nifty feature
is that you can put these overrides on your TFTP server for example and configure once how you want
to override the boot and then you can just boot it and yeah I think I have like two minutes on the
clock because I started to be glad I have a small demo that I would show you it's very short
I got 11 minutes okay
okay
yeah I am not seeing it so I can tell you something about it so yeah it's a bootloader spec file
that would have been booted by usual it's a 618 kernel it has a bootloader spec file has a
kernel image and a device three image and it and now this is my overridescript it's fetched over TFTP
now it has an image that's a fit image it concatenates some inetradi on the fly notice these
columns that means append it has some command line arguments that's the stuff on double quotes
oh that's a bit with bar and yeah but you can see it when it scrolls up and now by default if I type
death boot it will boot that script over TFTP you just need to set a user so you can share the TFTP server
between different users you see these notice at the top it says it has overridden stuff
it has taken the override for the kernel override for the inetradi or override for the TFTP
history you see now it's a 619 kernel not a 618 that was there before you see modules
are working and they come from the bind mount and yeah now you can just debug something it doesn't
take much time you just write the script once and you can even do some very brute force print card
debugging and I like this a lot because it's like you have booted in a minute and it makes it
very easy if I don't know where to look I just paper over all over the place print case and I don't
need to care about read only rootfs fit image or the emberty I can just get to debugging without having
hard to touch the rootfs and yeah that's it thanks for listening and I think we have enough time
for Christmas yes please
I have a small question about the first bootnet boot image you have what if something going wrong
when you try to load the image how could you debug without any shell or something for example in our
case we had a broken or mist and figure it network and switch and without shell it's impossible
to debug but we shall we debug it easily what what was going wrong how it could be done in your case
okay so you want so you usually network boot your switch but it's not working and you want to
debug that I mean that Samsung wrong is a configuration of the chosen image itself like in the
case you have this kernel device three whatever but no shell there you expect that everything works
always it's not the case okay yeah yeah yeah yeah that's true you can always do a full network
boot with a different rootfs coming via NFS that's the way I usually use it but often the customer
tells me yes I get like audio glitches here with this image and I want to try a new kernel I want
to debug the kernel so this is mostly for the part where you want to reuse rootfs but with the
booter integration you say so at the end you can specify a different tool and you can specify
other root or you can just do an NFS boot so if you have a built system which can give you a root
FS that you can just boot over NFS yeah go for that and use that and don't use what's on the device
but in my case I really wanted to use what's already on the device but yeah if I had the same issue
as you I would not use a root file system on the device but you another root file system
you could also add a root file system via inner tramefs if you have enough ram and boot from that does that answer your question
Hi so I have a question regarding the initarity image
You said about the concatenation right I mean so basically that's does it mean that you
Can can it concatenate many initarity images
Because you want to have access to see how it behaves in the user space type for example
the firmware or any other extra kernel models that needs to be there right
Then as I said the second part but yes you can concatenate as many in a tramefs as you want
Moussa as I am a door and there can even be compressed individually so that allows you to just composite in a tramefs just with a cat
Okay, but okay, let me frame it this way then
Does this so even your concatenating the initarity images does it follow a particular order?
I can be jumbled up. I mean
Uh, there's a later ones overall it's a previous ones so they are getting extracted over each other
And yeah, so you could overwrite
If you have modules for example in you know original in a tramefs which is not too uncommon
You could just replace it with modules that come later as I trust overrides them so they are extracted as far as I am aware one after the other into the ramfs
Okay, but is there like in the main lane kernel
Maybe this mechanism I I read somewhere that this
You don't need to follow the specific order of initarity images that needs to be concatenated it can handle this automatically
I'm not aware that you need a specific order. I mean, I'm not used it. I thought
Yeah, you know, so just concatenates it and the twack that is for me
I also only learned last year about this mechanism because my colleague and told me about it
And yeah, it works very well for this use case
To benefit from
This bind maroon for modules
Without an initarity would it be feasible to add something to the built-in magic that mounts the init
Romets
In the kernel to to benefit from the modules
Okay, so you want to have the modules in the initramfs, but some byte mounting to happen from kernel side
Yeah, right okay
He had a patch series for doing an overlay in the kernel and that way you could just say okay take
I
Yes, so you can absolutely add this logic to the kernel, but I don't know how well you cut
Are you in favor of it because it's that do it was an initramfs
So it's it's doable, but I don't know how what other chances to get this upstream
Because you need to argue in favor of it
And one example is this device to aliases as you might be aware that you want to give the names specific
You want to give the root files the root device is a block device is specific names
This has taken many years until we had aliases for MMCs and the argument all along was yeah, you can do it in an initramfs
Why do we need extra logic in the kernel to do something when you can just script it in your initramfs or use you
That and I think the same would apply here just do it in the initramfs
Why should the logic be in the kernel, but if it was a it would be nice that's for sure
But I think with our s in it it's will be easy enough that you can just
concatenate it at the end
at a kernel argument and
It will leave everything as is but just do the bind mount extra
So that's why I want to go from so you can just
Take it along and even do it offline
You don't need the bootload integration will just lose out on the command line argument fix ups and so on
But if you don't need that you can do that by hand
Should be doable with our just was offline our s in it concatenation and so on
Okay, thank you
Thank you very much for your talk
Thanks for listening