Perfect, can everybody hear me?
Oh, great.
Thank you all for coming for the first presentation.
It's always a bit nervous to be first out.
But this is a presentation about this big end,
which is a project that I've been working on for the last 83 years.
So quickly about me, my name is Phil Plain.
I'm Swedish and Finnish, living in Berlin.
I'm the creator of Spiegel.
I used to also be a Quarming Caner of a project called Flux
that does like continuous delivery for Kubernetes.
Sadly, I had to this morning edit these slides
because I got laid off from work
so I don't have a employer on my slides anymore.
So if anybody out there is hiring,
just like a shameless plug to reach out.
So yeah, let's get into it.
So just a quick introduction where we are today
and give you a background of what we're going to be talking about.
So in typical container environments,
we've come to the point where running highly available clusters
is become relatively simple at least.
It's not that big of a challenging more to run workloads
that are distributed, running in multiple replicas.
We're probably not managing VMs anymore.
So we've created a new type of bottleneck,
which is distributing the actual application workloads
to the runtime instead of basically having the problem of
like scaling the actual application workload.
Now about distributing the actual image.
There are some challenges in these types of situations
because obviously what we're doing is we're running some sort of
image registry that Kubernetes cluster is pulling from.
So there's a challenge of availability of making sure
that is always available.
There is no point of having this cluster that can scale to
10,000 nodes if you can't pull down the image
and the action will run.
There's also a question of speed.
So we're seeing, especially now, in these
damages of AI models, speed becomes a critical point
when your images are growing constantly because you're putting
large machine learning models in them.
But also cost.
So pulling images from public registries is something
that actually includes a pretty large cost at
the end of the day if you're cloud-built.
So that, I'm the wild.
We see this as like rate limits.
I think most people have experienced the great limits
with like Docker Hub out there probably.
The Kubernetes registry does some sort of like IP
rate limiting, especially if you're running workloads
and perks in a cloud for somewhere reason.
There's like a lot of like rate limiting going on
with their IPs.
Outages is very common and this isn't like
this like to any rate issue provider out there at all.
I think all of them have had outages at one point
another.
Like Docker Hub, get a container registry used to be
like last year, used to be down a couple of times
about a year.
Quay, but even like private cloud providers like
ACR, ACR also have their own outages.
VanWitz are pretty limited, especially
Docker Hub.
I think limits to like 100 megabits a second when
you're downloading images.
But also we're seeing image removals.
I think it was like a year or two ago.
Quay was like cleaning up a lot of their images
and we're actually like removing a lot of tags,
which like caused like a couple of outages
actually where people are explicitly
depending on these images being like available
and then just like over like a night or two
like just like disappeared.
So like the two options that we really have
or that most people have is like their
copying images to a private registry.
So what they're doing is they're running their own
registry and then they're saying, okay, we're just going to
define all the images that we need.
If they're a public or a private and we're going to
move them over here and put them there.
And that's going to be safe.
We're going to have control over this.
Or you're building some sort of like pull through cash.
For example, harbor where you're
pointing all your configuring all your nodes
basically to pull through a mirror cash
as you're getting the image for the first time.
It catches it and then makes it available
locally for you.
The first one kind of depends on the fact that
you're not going to have an outage with your
registry.
So you're kind of just like shifting the
bottleneck and dependencies closer to home
but they're still a risk of like you're
really nice cluster not being a little
scale because you're now managing your own
registry.
The other one is kind of like a best
effort thing where it becomes a
problem because you're not really like
defining what images
you're actually want to cash until you
actually pull them down.
Both of these kind of created like a
management overhead where you're having to
deal with defining what images do we need
where they're coming from and constantly
having to like clear up images that you
no longer need.
So like upgrading versions for example.
And obviously there's like a very
noticeable cost if you're running this
at like a large end of a scale.
So just some quick background I realize
when I present this before that not everybody
is like very into the OCI world.
So it comes a lot helpful to just give
some like some quick background like how
OCI images and artifacts work.
So like the image spec that is public on
get up kind of defines like how a OCI
artifact or like what we maybe know is a
Docker image.
How it is structured and defined.
So basically the top level of a
OCI image is what is called an index for
the most part.
It could be a manifest.
These are all basically JSON files that
reference other components of the image
spec.
What we have and what basically happens is that
in the index you define all the
manifests that are part of the image.
These are reference through
Digests which are hashes of the
manifest file that you're actually
referencing.
And there can be like circular
recursive referencing basically also.
So if a good example of this is
showing a probability to sum it.
So this is like the index file for
spagan itself.
It has a list of manifests and then
it has like digest.
So we can like click at the digest and see
like the manifest for for the
AMD 64 build off spagan and then
it in itself references a bunch of
layers and it has sizes.
So like this is like basically a
mercury that is being created and this is like
how OCI images are composed by these
sets of references.
Let's see here.
This is not going to go full screen.
Maybe loading.
Maybe we shouldn't have done that.
Oh, there we go.
Perfect.
Great.
The second part we should be aware of is
the distribution spec.
So this is what defines like how clients are
pulling images from a registry.
What the distribution spec allows you to do is
you can implement parts of the spec.
So that's kind of what we're going to be focusing on.
We're talking to like pushing in these types of things.
We're only going to look at how a client pulls
an image from a registry.
So there are like basically three end points.
You have to like implement v2,
select v2 and it's basically to like tell the client
that is supports it.
Then there is like an end point to fetch
manifest contents.
The reference there is basically a tag or a digest.
This only works for a manifest but what basically happens
when you pull by tag it actually just resolves
the tag to a digest and then returns the digest content.
And then it is there is an end point to get
the blocks content, which is like the layers
of the actual binary data that creates the operating system
that you're running.
So this is basically like what you need to implement
a read-only registry.
It's like kind of simple actually which surprised me.
So back in, I tried to look this up and I couldn't find it.
But this was probably back in like 2018
some time around there.
It was a while ago considering that
this was back when Kubernetes was still running
like Docker as it's like a container run time
in our community.
I was adding me up listening to a
pretty large company I'm not going to call them out.
Talk about a 1am outage that they had.
We're Docker hub went down.
And at the same time they're seeing
an influx in customers.
I don't know why it 1am but it was probably
in another time zone.
But basically they couldn't scale their clusters
because they had critical dependencies on images
from Docker hub.
So he was trying to send a solution about
how he solved this.
And it was basically him
as a staging into a old node that had the images
running Docker export.
And then SEPing the exported tar
to the new node that was scaling up.
And then as a cluster was scaling up he was just like
continuously like SEPing the images to the new
nodes and then basically importing them.
And it worked.
Like even if it's like a very dirty solution.
It kind of got them out of that kind of scenario.
But it really planned to seed in my head about
while having a physical person wake up at eight
1am doing this work kind of as crazy.
It grew from a technology standpoint that
why aren't we always doing this?
If we could automate this type of process we'd actually
remove a critical dependency towards something external.
And it kind of like start barking
and trying to understand how does this work?
How does Docker export work?
Like where are the images coming from?
At this time I was like basically varying you within
the OCI specification.
So I really didn't really understand everything.
But like I had a couple of other questions.
And so it was like okay so images.
How are they stored in container D?
Like where are they getting pulled from when we're
exporting images?
As I talked before about registries.
Like is it possible to implement my own registry?
Or is like Docker hub like the only alternative out there?
Then it was just like I was thinking about mirroring.
So how are mirror configurations and
continuity configured and like what are the possibilities
around there?
And then obviously like discovery because like my idea is
basically okay if I want to do this at scale
I'd need to like have some sort of component that does
discovery and like to basically find which nodes
has which images.
Eventually the solution I came up with is the project
that I've been working out for a while now called Spiegel.
It basically allows Kubernetes nodes to pull images
from each other.
It also has like a nice fallback solution where if the
image doesn't exist anywhere in the cluster it basically
just falls back to pulling from the original registry.
So this is as I like to call it a
stateless OCI registry.
And that kind of is like a juxtaposition of like
people like this isn't really right.
But like they'll kind of explain a bit more about that.
But through the benchmarks that I've built what we're
basically seeing is an up to 80% improvement in image
full times.
There are obviously scenarios that are sub-optional
for Spiegel but like I've built a it's also open source
a benchmarking tool so you can compare your different
pull performances.
So the major breakthrough that really allow me to build
Spiegel is I realized that container D when it pulls
an image actually stores the uncompressed layers on disk.
So this is disk space that is always used in any type of
container environment.
So the images have always existed in the cluster.
So in all of these outages we're basically going,
oh we can't pull from this registry.
Basically there's somewhere in the cluster most likely
your images have already existed.
These are stored in yeah in valid container D.
Just as like the file names are the hashes or the digest.
Basically there is like some other trickery that you have to do
to like talk to container D to get like the names of the images
and like the tags but that's like pretty easy to solve.
But this basically means that Spiegel can like piggyback
on top of container D. It doesn't have to.
It's not doing any type of storage.
It's not doing any type of like cleanup.
In Kubernetes garbage collection is triggered automatically
on disk pressure.
So you're basically really just acting as a proxy
to serve content that is already there.
And that's kind of like the cool thing of like there is no state at all.
So the architecture is actually very simple.
On each node that you run in the cluster you're running Spiegel.
It basically runs a OCI registry that is available in localhost.
Container D is configured basically as it has a mirror configured.
So when you try to pull an image the first registry that it tries is not the upstream
registry is actually localhost.
Spiegel will then receive this request and try to discover another node
in the cluster that has this layer that's being requested.
And just proxy the request.
That's like all it does.
And in simplicity Spiegel is really just like three different components.
So it's the registry.
It's the routing or discovery mechanism.
And it is the advertising mechanism.
So and it's core sense.
It isn't really more than that.
Content discovery was probably the hardest part in figuring out how to implement these types of things.
I this is not my background in like these types of distributed systems.
But as some might like recognize and like the the diagrams and everything.
This looks very much like the torrent.
And that's basically what it is.
It's a bunch of clients advertising content that they have to other clients that want to discover it.
After some reading you kind of figure out that nearly.
Most of these solutions today use an algorithm called cademlia, which is like a implementation of a distributed hash table.
There is a really nice product called libp to p.
That is used in like IPFS and some other things.
Shout out to these guys.
They have a really good implementation of cademlia and go.
So it comes it was really like this is doing most of the heavy lifting.
I am not smart enough to implement this myself.
So very happy that somebody else did it.
But basically what happens is it's basically looking at all the content locally in this and advertising it on to like the distributed hash table.
And then as a request for coming in basically there's a look up done on the distributed hash table to like find the content.
There's some compatibility issues mostly it's based it's dependent on like how the container de default configuration is.
It works in a case it works in a case you just need like do some minor configuration changes not that bad.
Currently it's embedded into both the k3s and rk2.
So these are like embedded features they don't run as containers actually use import as big as a library into k3s.
And it basically is it's from my perspective honestly a better solution than running it on top as a container.
Sorry.
So use cases.
So when I built this I was working on another company running Kubernetes clusters for like large customers.
So we used it mostly as a best there for cash.
One thing that I've realized and that's like the major user of this project are home libraries kind of makes me happy but like the majority of users out there are using this to avoid a doctor hub's rate limiting.
Which is like as if anybody's run like a small cluster at home is kind of like the major pain point where you just like I don't.
I'm just doing this for fun I maybe don't want to pay for a subscription to pull an image but because you're restarting your nodes or trying something at home you're just like constantly getting rate limited.
Yeah, so there's a lot of people they're using this as like a just like a small hobby project really or like just for their hobby projects.
But also I'm seeing this talk people using this in air gap environments.
I've received information that people are trying to run this on trains which is interesting.
But also like there's there's people using these types of projects or this project to distribute AM models.
So especially in situations where you're running large AM models and you want to like not pull these models from like a external registry you can basically because it's like private local.
Like private local networking you can basically distribute these models a lot quicker internally.
Okay, I will.
Quickly talk about challenges and then maybe we can do questions.
So the challenges that I'm currently working on is mostly about like topology or writing.
There's like a lot of like problems and they're not problems for challenges and can only have because can only is only aware of like the larger network.
It doesn't understand like networking topology and regions and these types of things.
So there is some research work that that's out there that is trying to like.
Nudge networks into like choosing the physically closer nodes compared to more distant nodes.
But it's actually it seems like a very complicated thing and but it's like there's some work that being on there.
Fair load balancing is nothing.
It turns out it's very difficult to do.
Implement load balancing algorithms on distributed systems where you don't even know the peers that you're going to route to before you get the request.
So it becomes like a very interesting problem of calculating bandwidth and not basically overloading a single peer in your cluster.
And then there's like some challenges in how goes HB coffee performance works in proxies basically everything is done in like user space.
You're basically copying memory back and forth between like a response and a request.
So there's like a lot of challenges there and like how.
Like there is some performance degradation when running through a proxy compared to like hitting a registry immediately.
I think there have some more things here but we can skip these.
Thank you very much.
If you have any questions we can ask them here.
You can reach me by email also.
My username at most platforms is Filippaba.
Here's a link to the website and also to get a repo.
There's a QR code there if you want to scan it.
But we can start with questions maybe.
Yes.
I think for the talk.
It's very interesting in sounds very useful.
I was wondering if there is.
I was wondering if there is an authentication mechanism or how does it work in general with the authentication.
So currently it is completely unauthenticated because it's within your own private network.
So the aspect of like.
There are obviously security concerns.
What you're kind of relying on is the fact that you can't modify the existing like mercury.
So like if you're pulling images by digest reference.
What you're getting back because the client will always validate the digest to the content it's getting.
Can't be modified.
Obviously if you're pulling by tag.
You're trusting the upstream registry.
K3S implement TLS.
So it has like some it has it.
We've worked together to add some additional authentication features.
Out of the box bigum doesn't it's something I've looked at.
It's very complicated to like generate certificates and a distributed workflow that is stateless.
It's kind of if somebody wants to help with that.
I'd love to have help.
Yeah.
Hello.
Thank you.
Oh.
I'm.
Speak out of the manager.
The end user.
I could not imagine.
When we discover and the ability.
For example.
Sorry.
How does it manage the end user?
If we need to.
Did it this image is in the plus.
Or even the cabinet's crystal.
Oh, it's bigger.
Yeah. So so basically.
Image layer image advertising is done with a TL.
So there's like a time to live.
What happens is if a so it's bigger doesn't.
Remove any images.
So that would be up to the garbage collector to do that.
When an image is removed.
It's going to be stopped to be advertised eventually.
The time to live is going to invalidate the advertised key.
What happens is if you could end up in a situation where.
The images removed from disk.
But the time to live has an expired.
If what happens.
It's just like it gets a four or four back.
And they just tries to expire.
So that's basically what happens.
Yeah.
I'm happy.
Have you considered running it.
The the other way around.
You said that you have implemented it to circumvent.
Limit rate limiting and stuff like that.
But still you mentioned that there's one like overbearing one single
node that has to serve too many images.
Which is one of the things that I've had a question about when when you started.
And now I was thinking is there possibility to switch around.
I mean, if you don't have a problem with rate limiting.
Could this be something to have more of a resilience increase.
If you have a risk placement for the cluster where you.
Debut default pull the image and only if like this connection does not work.
It's like jumps to to speak a land then gets from another node.
So you run doesn't or run into this issue.
You can you can hear yes do this and container.
Configure like mirror configuration.
You could actually add.
The mirror the original registry as the first mirror.
And then have container default back if that mirror fails.
That is that is 100% of possibility.
I still think that like you probably want to serve it internally.
Like it's it's mostly like a load balancing question of.
Like the the rate limiting internal thing has mostly to do with.
situations where one node has an image and then you spin up.
200 more replicas and they all just go.
Give me the image and there there has to be some sort of like because.
From a from a sort of perspective it's like it would be more beneficial for.
The request to queue because as each image is they are each node is getting pulled.
It's actually helping in in the cluster of distributing that image.
So like but we don't want is like the whole thundering herd problem where 200 clients all come.
We we want some sort of like organization.
I think that it could be solved.
So I like I prefer to solve that rather than just go pull from the external registry.
I just don't.
I don't know the right words for this type of problem.
It's kind of my yeah.
Yeah thank you very much.
Question up there.
I've got a quick question.
You said that you're using a harbour for image as a back end.
Did you reconsider different things like a result project.
Sorry what?
Did you try a result project for that instead of harbour?
Yeah so like all running all these like thoughts, harbour or like the distribution,
registry whatever they're like multiple of these like types of registries that kind of solve these types of problems.
But they they're all stateful.
That you all you like in these types of solutions you're basically having to like run something that has
state that is using disk and that's kind of like the challenge.
Hi.
First of all thanks for the things ready to talk.
I was wondering about what about latest stack.
How are you dealing with that?
What's latest stack?
It's on request.
Yes.
That is a problem.
There is in the FAQ.
I have document how to solve this.
If you're only pulling by latest yes you will basically lock in a single version of that latest.
You can't update it.
There are projects like Kate's digester that like runs the muting web hook and resolves the latest tag for you and then
append the digest.
That's kind of the solution you have to go to.
I will not preach about my opinions about using reusing tags.
Do you have experience with which non-image blocks can be stored locally?
I'm here by the way.
If you have all of us or something where some at the station is attached in the OCI image structure would contain the cache.
Anything of that or would be done to pull that?
Not like non-doctor image.
This is like I skipped one of those slides but it was basically.
Container D is coming with like OCI volumes.
Soon hopefully there's like a there's a PR for it.
That's going to raise like the ability to run like to like mount OCI artifacts as volumes into a container.
When that comes it will work.
Maybe I'm running out of time.
I don't want to mess up somebody else.
Just checking this in one more question maybe.
Okay.
Who was this?
I think he was it.
We can we can talk in the corridor otherwise.
I think he was.
Then the rest you can ask afterwards.
Hey, uh, I was just wondering is it there anyway to use it outside of Kubernetes?
Let's say with part which pulls OCI images for Mac VMs which are quite big for instance.
Wait, where?
There's a protocol taught which runs maximum Mac VMs.
Oh yeah.
I know that there are people running this in Nomad.
Like if as long as you're running container D it should work.
There is like a slight requirement in like how you do the bootstrapping of like the distributed hash table.
If you want to have support for something that isn't no matter Kubernetes,
I'd happily work with you and like try to figure stuff out.
Great.
Cool.
Thank you.
Thank you very much.
Here we go.
All right, so I'm making up.
Thank you.