OK, our next talk is by Stefan Greber about Incus and how he managed to enable it to run
OCI containers.
All right, hello everyone.
So, as Christian just said, I'm Stefan Greber, I'm the project leader for Linux containers,
one of the maintainers of Incus.
I'm also the owner of my home company doing consultations stuff and a CTO of future
fusion, which is another company doing large scale and price Incus work.
Today we're going to be talking specifically about OCI, so application container with
an Incus in this case, and kind of what we've done and why and how.
So, kind of kicks in off just very briefly what's Incus, because maybe some of you don't
know that.
So, Incus is a system container in veteran machine manager, and nowadays also application
container manager.
It's all image based based on a REST API, it's got a pretty simple CLI, it's got support
for most of the stuff you would normally expect, so you can use non-flots, backups,
or bunch of different networking storage options.
It also has a small web UI, you can use projects to segment things, so you can actually
with external authentication and authorization, you can turn it into multi-tenant environment,
it can be clustered up to 50, 100-ish servers, so you can run it at reasonably large scale,
it supports distributed storage with a three-step, but we're also adding NINSTOR now, and
we support shared blocks for LVM, and then on the network side we use oven for all of
the software-defined networking bits that are also an option, but physical networking works
just as well.
I mentioned in integrative, if you have external things, it can use oven ID, connect for
authentication, can use oven FG for a fine-grained authorization, there are a number of web interfaces
that's kind of the one we usually go with when someone wants to see it, personally I didn't
always use the CLI, so the demo afterwards is going to be CLI, and yeah, so it's a reasonably
active project these days, we've had 130-ish contributors last year, all written in
go all up and so also, but she too, and get had.
So why support application containers, anyways, from all the way at the beginning of the project
back when we went back home for a still, like the under canonical, we only focused on system
containers, so running full Linux distros on this thing, and yeah, it didn't even do
v-hands, it was really just containers, just full Linux distros, and that worked pretty well,
it was very useful to build a bunch of folks, and our father the time was, well, there's
Libvert people are going to be using that for VMs, and they're going to do both on the same
system, and we're going to be expecting Libvert to become more and more user-friendly, get
an API without kind of stuff all the time, so that everyone would be happy.
We noticed that this never really happened, Libvert's going to remain where they were,
because in a presentation detail used by OpenStark and others, instead of turning into something
that could assume that regular users really enjoy using.
So after a few years of kind of waiting and to see what happened, but like, okay, fine,
we'll just buy the bullet and we'll add VM support to, it was next the other time.
And that worked pretty well for us, but it was actually reasonably easy, I think we've
got it working just a few months, we've painted up all the couple of years to support
a lot of features, but that was pretty easy, and we had to extend the same situation with
application containers.
We obviously, Docker has been a big thing for a while, people have been using it, they've
been using it alongside in-cast, they've been using it inside of in-cast containers, both
kind of work, but Docker alongside in-cast has a bit of a tendency to keep black networking,
because it assumes it owns everything, and so it injects a bunch of firewall rules that
then blocks everything else on the system.
You need to go and mess with that, we've got documentation on how to fix it, but it's kind
of a new online.
You also end up having your online network on storage, both in-cast, and in-docker, and
gets a bit of a new online integrating things together.
If you instead go with Docker inside of in-cast, that works well enough, but now in-cast
generally, running in previous containers with higher security, actually interfered with some
of the Docker images and the wage-run things.
The number of storage options was quite limited in that scenario, and networking was still a
bit of a mess, because now you've got a network inside of a container, if you want to
integrate with something outside of it, and get a bit messy.
So that was kind of the state of things, but we've seen a lot of people have used for
application containers, whether it's for IoT stuff, basically all of the IoT bridges, for
like Zigbee, Z-Wave, whatever, they all shipped us Docker containers these days, like all
of the Home Assistant components, our shipped, our shipped, our shipped as OSI images of
a bit of a Docker hub, and people are consuming that instead.
So there's no, there's no, it's just a bit of a weird fit to try and like manually repackage
those things to run them on top of in-cast, and otherwise you were doing nested Docker, that
was always a bit dodgy, there's like, and more and more applications effectively, officially
shipped as a Docker OSI image, this is.
So there was a bit of a need for that, we noticed that like we don't want to start competing with
Kubernetes or something that's not our intention, but a lot of people just need a few containers
running, they don't start getting them up and down constantly, so it can be a bit
sense to add that for us.
Also the reason to do it, it didn't are being quite easy and fun, so that's always a good
justification for doing something.
Now, how does it work?
Well, what we do is actually reasonably simple, because they're good tools that they're
that simplify a lot of that.
So we need, obviously, to interact with a registry, so we use a scope out for that, then
we need to fetch that from the registry, again, scope out as that for us, then we need
to go and turn that into a viable root-fight system, so we're using much, you're much
free for that, which effectively looks at the layers and squashes everything together.
And once we've got that and we turn it into a normal in-case image, we load the image
into in-case, we create a normal container from it, and at that stage, don't think that's
different from a regular system container, is that we also process all of the OCI config and
metadata, so we look at the environment variables, we look at the extra mass, we look at
all that stuff, and the entry point, and we've quickly put all of that in place within
the follow-config, and then the container starts up.
One common misconception is that we effectively have in-case-riving Docker or something, it's
not the case, we turn the OCI image into effectively in-case image, and we burn it through
a normal container runtime, which is LXE.
We don't use RENCY, only of those at all, in this case, we use the exact same runtime,
whether it's a system container or an application container.
And then, yeah, start the container, and it just works, basically.
So, time for a quick demo, on the first and my file, which is always fun, and so here
I've got an empty in-case project, and the first thing we need to do is actually, so for
all images, it comes pre-configured file image server, we could, in theory, pre-configured
a Docker hub because it's the most common one, but there are many other registries, so
we just don't do it.
So, you need to actually add your registry, so we, in this case, for a Docker hub, you
can do that, and then see the protocol is OCI, and once you've done that, now you can
do Docker, C, NGNX, and my NGNX, and I'll step the image already, don't know that, so we
don't need to enjoy the Wi-Fi too much, and effectively it just launched it.
So at that point, hey, I've got a container running, I can go and check that we do have
NGNX actually running on this thing, which we do, and if we go look at the config, those
were used to normal, in-case containers of VMs, usually the config is really empty at the
beginning, it just has some image and files in some volatile info, that's different for OCI
containers, you can see, like, for just specifically for that.
The environment variables that are defined in the OCI image get automatically added to our
config, and so that there, once that's done, it works a bit differently than what you're
used to with Docker, because with Docker, it's, I don't know, maybe there's some magic stuff
I don't know how to do, but it's not trivial to go and reconfigure things in place, well,
as with Incus it is, like you can add additional months and stuff while the thing is
running, you can change the amount of CPU memory while it's running, you can add GPUs,
what it's running, and if you want to change the environment, you don't need to delete it,
you can just change the environment, restart it, and you're done, so that makes it quite
a bit easier, for my personal use case at home, which is mostly running old bunch of IoT
home automation types stuff, I can run those things basically, if I ever, if I need to reconfigure
where the MQTT endpoint is or something, I can just go change the environment, restart the
thing, I'm done, it also uses normal incase networks, storage, all of those features,
so it's obviously if you're running like a mix of containers and like, it's the
questioners and VMs, now you can do those alongside it and it just all fits nicely, it's
on the same network, you can put the same firewall policies and stuff between them, it goes
on the same storage as Incus, if you're running a production cluster with red and
then storage, then now you've got red and then storage on those two, so it's, it just fits
really nicely and the actual amount of code and effort to do this was pretty minimal.
We did have to do a bit of extra work afterwards, because for example, Incus had zero
need for a notary start policy, because we were running either VMs or system containers
and in those, they usually don't die, like if you, if QMQ crashes, you probably have bigger
problems and if PID-1, like system they in a container crashes, you probably also have
bigger problems, so we'd never needed a restart policy, but obviously with application containers,
it's pretty common that if a service wants to just restart itself to reload, it just
exits the container and dies and starts backup, so we've had to add auto restart, the other
thing we didn't need to do in Incus is because we're running full distros or full operating
systems, they usually have a network management tool of some kind that does the HCP for
network config, that didn't exist here, so we actually need to write the tiny DHCP clients,
which when the container starts that runs, gets a lease, stays in the background and those
renewables, but it also means that you can literally bridge those OCI containers directly
on your physical network, and it will just grab an IP from DHCP nice and easy, you
don't need to mess with static IPs, I don't think so, now we get to the kind of what's
coming up next, I mean for my personal use we're done, it works, but there are always
things we can do better, currently I don't love the fact that we shed out to you much
she and scop here, because both of them are local bases, we are local base, we should be
able to just use the right logic and not need distributions to shift as separate tool,
so that's something that we'd like to do, I know the phone view much easier, it's pretty
easy to do, also a geometry creator and maintainer, it's also an incost maintainer, so if
we need changes there, nice and easy, scop here is a bit worse from what I've seen, it's
not particularly well split, the parts, it's not really designed to be included in other
code bases, so we might need to look at what we do there, there's a bunch of discussions
around handing of private registry which we currently don't do, around how do we handle
the authentication and all of that, like obviously we're sure that's the API with
a kind server type of design, so we're depending on the authentication, but just using
in password, we can pass that through the request very easily, if it's something more
complex where you need to get a contemporary, use token and stuff that gets a bit more
complicated, so we're looking at the best options to handle that in a way that's mostly
natural and easy for those of dealing with that on paratroopers, trees, in Docker and
other tools.
Something is going to be a bigger piece of work, but for us, we can't complete the set
is allowing running goes as VMs as well, so we've for our normal images, if you do incost
launch images Ubuntu 2444, you get a container, if you do dash dash VM, you get the same
thing as a VM, we want the same experience for OCI images, so that if you launch them
as they are, you get a container, if you do dash dash VM, you get a very thin VM layer with
the container, image running inside it, so that's cannot cut a like design, we've probably
a very, very similar design of Q and U, VOTIFS, we've said all of those things are ready
for all VMs, just a matter of putting the right bits in the right places, and the last
thing is potentially handling layers, but the reality so far is that 99% of the images
we've looked at, there's so small ones, there's actually squashed together into a single
every day, so we can manage, we've not really seen a good use case for that, the one big
use case would be people doing the IML type stuff with the massive Nvidia type by base layer,
there it would be a bit annoying to run three containers and have in theory 99% of the image
being shared, but having them duplicates, but then for us to start supporting layers throughout
all of our image management logic, volumes, or trust like, but doesn't different storage
back and across a cluster and all of that, it's not trivial, so it's going to matter
of balancing the need for that, so far, it's like basically if you have that need, might
as well use, keep using Docker, and that's basically it for me, if you want to play with
it online, we've got the online demo that is you play with incurs containers, VMs, and unless
the IP address is changed and I need to be the firewall, normally also OCI images from
the Docker hub, so yeah, that's a good way to effectively get a VM on an incurs cluster that
has nested VM support and that has incurs installed, so you get to play with it for
it, and we've got a few minutes for questions.
I'm going to steal one question, couldn't you, the layering problem, couldn't you
do this similar to what system you're doing with system extensions and contract extensions
that you essentially have images that you compose using overlay, for example?
That's probably how we would do it, yeah, you would want to, don't know the layers and then
do a overlay effect, currently the biggest, and that's not necessarily that difficult
for us.
The part that's more difficult for me is that right now in our image store and all of
our internal tracking, we've got an image like a single object, now with layers, we're
going to have to track potentially 20 different objects for an image and keep track of
like who's using what, and so when we do replication of images we need to cluster, we
can't just have the layer replicated to one hole in the next layer on the other
holes, because then it's on a wrong machine, so it's all of that tracking logic that's
kind of tricky for us, more than the actual assembling the thing at the end, because
yeah, assembling the thing is setting up a value of S is, we've not done it inside
of ink, but we've done it before in LXC, we're pretty familiar with that process, that's
not too difficult.
It's mostly all of the keeping track of usage, when can you expire something, all of
that stuff, which is actually more complex.
Thanks, Stefan, it's very good talk, I was just wondering about something that you mentioned
that you can add, for example, mounts in the container, while the container is running,
and you also mentioned that this is also applicable with GPUs, and I think Christian presented
how the GPUs worked a few years ago, but I was wondering, is it possible to also hot swap
to remove GPUs or external devices?
Yeah, exactly, so we can do hot plug, obviously for mounts, we use some weird tricks to
propagate the mounts into the container, we can do that kind of stuff, for GPUs, for GPUs,
we can do it, because we add the kind of same thing, effectively bind mounts, the character
of the devices that are needed in, and we can remove them as two.
Technically, it's a bit dodgy, because if you remove a GPU that's currently being in
use, you honestly keep track of that, and you don't know what process to start killing
on a thing, so removing a GPU that's currently in use will likely let them still use it
until the next application tries and then it's gone, but that works just fine, and it's
been something we've worked quite a bit on InCurses that's hot plug works for just about
everything, both on containers and on VMs, like on VMs, you can also do GPU hot plug, it
will do PCI hot plug in, PCI hot plug out, now you might get your canal panic if you've
not correctly clear the usage inside the VM when we're yanking out, but we do support
that, and my core VMs will also support CPU hot plug and hot remove, and that works
surprisingly well with the right ACPI events, thanks, sensor.
Is this already integrated into the patched LSD UI?
Yeah, so, well, kind of, we do have detection for those, so they will show up as container
up in there, launching them is a bit trickier because it doesn't know about all of the potential
remote since it's all of the potential hubs, I think it's possible using the Yamal option,
otherwise at least anything that you launch previously, you should be able to select the
cashed image and create more of that, I know we did sort of work on the terraform sites that
the terraform provider now handles OCI just fine, but I think the UI could do for a bit of an improvement
for like saying I want an OCI and it's going to ask you like what registry what's the name
because we, with scope here we can't easily go and list all images on a registry, but as
as we can for distros, so you're just a bit of a gap there, and I think that's it, we're starting
for someone else shortly, thank you.