Hello everybody, appreciate that you all managed to attend our talk which is the latest
one on the second day on that awesome constant conference.
We are talking today about Kubernetes on-premise which is basically quite difficult
and challenging talk and we at metal segment all created a software which helps you to create
Kubernetes at scale in your own data center on-premise and should be or is in our opinion
available alternative to proprietary solutions like Broadcom or VMware from Broadcom
and other solutions to create Kubernetes in your own data center.
My name is Stefan Mayer, I'm a CTO at excellent with a small consulting company from
Munich, my colleague Gerrit will take over in the middle of the talk.
We are located in Munich and about 30 people founded in 1999.
We mainly focused on the financial market so our customers are banks and the insurance is mostly.
And we also became two years ago also hosting provider for Kubernetes as a service which is based on the technology we implemented at metal stack.
We also see in CF silver member.
Yeah, and I try to, or we try to show you how we in Europe can get more confident in doing our own cloud business
and not only rely on American cloud providers.
As you all for sure recognize that the political landscape changed a bit in the last weeks and that will continue.
I'm pretty sure and this enforces us in Europe to act.
It's not only that the cloud providers are dominating the market.
It's also that the on-premise market is dominated by American software companies.
VMware at the most prominent one.
And we at metal stack think or for pretty sure that we should do something against this dominance.
Grown as an open source alternative in the European area.
And we also believe that Kubernetes is here to stay.
It's already 10 years old and proven to be very stable.
And should be the foundation for clouds created and born in Europe for all kinds of applications for all kinds of applications.
And also for as a base for virtualization workloads.
This will help the European industries to gain freedom.
And also leave the proper territory boundaries they actually have.
In Europe the GDPR also requires that we own our data which is not the case if all our data is located or controlled by American or Chinese companies.
On the other hand we are still to shy in Europe to create something new which is big enough to create an alternative.
But we are not to dump we just have to start and that's what we did.
The time to leave the public cloud is now as in the keynote yesterday was said it's the it's the global switch day so in our opinion it's also the global switch day to leave the public cloud now so yesterday.
So what we have done so to bring the cloud into your own data center we created a metal as a service API which.
Basically enables you to turn your own hardware in your own data center into an elastic cloud infrastructure and the shown here in the picture imagine you have.
Here shown to different data centers full fully equipped with service and switches.
They are connected to our API and as a as a user as a consumer as an API consumer you can tell.
This API please give me a machine of specific size and t-shirt size for example with specific amount of CPUs and RAM and hard disks.
Tell this this API and one and a half minutes later you will have a fully installed ready to use bear metal machine with this operating system and specification you gave the API and we use that's that to create later Kubernetes clusters based on that API specification.
And you will ask why bear metal we have virtualization technologies as guys already talked about today.
There are pros and cons to use bear metal you obviously get the best performance if you skip the virtualization layer and the lowest latency and in.
Since we have 100 gene networks or more since we have flash disks you would not give 20% or more away for the virtualization layer.
Instead you will gain the whole performance of the metal you also get the best possible tenants separation because if your workload runs on this server and nothing else then no one else can.
For example interrupt your workloads with the performance he creates.
The stack is also simpler so between your application and CPU is nothing else then the container virtualization which is.
And it's also optimized for GPU workloads or even virtualization workloads which was mentioning beforehand there are of course downsides so if server dies your workload dies of course.
And you can kind of do over provisioning or life migration but these are topics which are covered by the Kubernetes mechanisms so if a pot is not there anymore deployment will guarantee the Kubernetes schedule will guarantee that there will that the pod will get started restarted on a remaining server.
So what does metal stack cover so it manages the whole data center from the from the foundation so it manages racks machines networks firewalls operating systems it's fast them opinionated for this special use case you will have no window lock in it's for the on premise use case you will not have any dependencies to outside cloud providers you gain data sovereignty.
You can use the software in your own data center it's completely open source you will get the best price performance ratio you could possibly get because there's nothing else between and it's completely API driven and implemented in goal.
And we use this in production since about five years in critical industries in Germany to our since 2018 for us the current known production deployments.
Our consistent of about 400,000 containers 250 Kubernetes cluster more than 60 switches which are managed more than 8,000 and a 1,800 servers and five database of assistance.
To get more in detail Gary is taking over and explains how this all works.
So thank you. I think I will do it like this. Okay, so I brought a really complex slide here and I think I don't want to go into all the details it's from our documentation at docs metal stack.io.
And the reason why I show this slide is just to show you that this is not just a single piece of software that runs somewhere it's also part of your data center it runs inside your data center.
And it's also like an opinionated on how the network in your data center should work. So we decided for a BGP in the data center and there's also slide data would want to go into with more details there, but yeah it's just for you to know that we have this bigger concept where you have like control plane with the API and then there are we call it partitions.
So I'll provide us call them zones, I guess. And for us the partition is set or group of servers which participate in the same cluster.
So our solution also includes then how you cable up the wreck and we intended when you go out of machines as a provider that you can easily scale it up.
The installation by adding more x and yeah the clusterology that we propose is failure redundant so all the switches in the infrastructure can be exchanged without interrupting the traffic for the users.
Yeah all our machines need to be dual attached and we even have a constraint that servers cannot be allocated until they are dual attached to the switches.
Yeah, an overview of a supported hardware on docs.metallistic I also we don't claim that we can support all the hardware because we also expose some management and points to managing a machine like power on power off get the power consumption of the server.
The current stage and also obtain a serial console to the machine and stuff like that.
So we implemented this through IPMI and red version all of these friends that you're probably aware of.
So this is quite a tough part and yeah this is quite a tough part and yeah we hopefully defined a good interface to extend this.
So if this is an issue for you that we only support super micro or data servers then you can also just contribute vendor implementation when you implement the interface and it should work.
Yeah on the switches we use Sonic OS so yeah we thought this is a good open source operating system to run on the switches.
And yeah so yeah I think that's it for this slide next one will be a bit about network so we simplified here as well but just to get the idea across so yeah we have like tenants in our API and tenants can create projects and when you.
Yeah create a project you can also allocate networks within this project and yeah in this project you can place machines and you can also put them into your own project networks like node networks and yeah you can allocate machines in this network and then you can the machines can reach each other but.
If you want to establish a connection to other networks then you also need to place a firewall component inside your.
Yeah inside your node network so this is just another bear metal machine which can be a smaller size because it does not have so much to do it just routing traffic.
And this component will do a VRF termination so as I said we have a BGP in the data center this also book that guided us which I can really recommend it's called BGP in the data center so this was our Bible back then for reference how to implement this and.
Yeah with this firewall you can then make connections to other networks like the internet or other internal company networks and yeah also solution consists of an IP address management system so.
Yeah people can acquire IP addresses and bind them to their nodes and yeah the appeared addresses that are bound to the nodes will automatically be routed through the entire switch layer by using FR and yeah the nodes will then.
Okay so the way that we use it is like this.
We actually have like a border for metal stack which is really clear it's just the bear metal machine provisioning and we end there so.
I just think by itself does not know what Kubernetes is but we have an integration with another open source project which is called gardener which I want to show you after this slide actually.
This will then utilize the metal API for creating machines for Kubernetes clusters but yeah as you can see as a whole you can build a complete data center stack with open source projects and it's I think a fast and clean architecture and it works well with Kubernetes and we.
Yeah developed this in Europe in Germany in Munich and yeah so.
Next slide how can we deploy Kubernetes.
Yeah so what is gardener so I already mentioned that it's another open source project that we did not come up with but it was written in the same time when we started all journey with metal stack and we found each other quite quickly and.
I thought yeah will it works well together so it's called the gardener and yeah they started in 2018 and they are actually a Kubernetes installer and it's a multi cloud platform more or less where multiple cloud providers can implement like also interfaces and.
Yeah with these interfaces implemented it can utilize work or not creation.
Yeah automatically and you have like these auto scaling capabilities you have like rolling updates for Kubernetes clusters and yeah stuff like this all the data operations I think they.
Yeah to a really great job in this and also we know these people and they are quite friendly and have an open community so.
If you're interested in like providing Kubernetes in your own company then you should definitely have a look at the gardener as well because I think it's a cool product.
Yeah and it's also good for actually bare metal use cases like in our case because.
Yeah the idea is that gardener spins up more Kubernetes control planes inside a Kubernetes cluster so it's like if they call a cubeception model.
And yeah it's not like when you provision Kubernetes that in your cluster you can have your own API server as an end user you do not see the API server for Kubernetes inside your cluster.
It's running inside another bare metal machine which is operated by by the operator actually and yeah this way you can fully leverage because you have then many tenants in one operator cluster and you can fully leverage power of.
Yeah physical machine there because you can place many tenants inside this machine there.
Yeah so that's it about gardener then there's also something that we want.
To just start with yeah we want actually to make proposals for record configurations so everything we just told you is completely installed in your on premise.
But our idea is to make this a bit easier because the whole deployment is quite complex and you also need to understand the BTP parts and stuff.
As otherwise you will not be able to set this up and also not to you kind of debug this we wanted to propose like reconfigurations.
Which you can use so we have one yeah smaller reconfiguration that we want to propose with just eight servers for starting and then a larger ones.
And the only thing that you would need is an internet connectivity and space and the power.
So yeah as Stefan already said in the beginning it's really hard to establish trust in this domain because for European.
Yeah solutions I think most of the people think it's not ready yet so in 2022 we thought yeah we should even make a hosted version of it so just installed in a data center and we did this in Munich in a data center and offered this for public use to so this we call matters they're cloud and yeah they can try it out and if you don't believe it works and there's now the possibility to do it so you can spin up coordinate is clusters there.
Then this small development update because I'm also the maintainer for many repositories that we have on GitHub and I just want to say thank you at this place for all the contributions that we will see from the outside it's always great.
So what we're currently working at is the basic IPv6 for support so everything right now is working with IPv4.
Yeah and we want to enable also like a service exposure and Kubernetes for IPv6 addresses so there's one pull request for Stefan which is going to be merged I think pretty soon so yeah I hope that.
We will be able to just a provision Kubernetes cluster fuel union IPv6 soon.
Yeah then one thing that we did lately and one of the latest releases is also to enable offline environment installations so it's also now possible to set up metal stack in environment where you don't have like internet.
Connection but it requires you to set up like NTP servers and DNS servers on your own but at least all our components can now be configured in a way that all these external dependencies can be configured in point and three years on data center so yeah you don't need internet connection there.
Then there's also one bigger MEP and MEP for our project means metal enhancement proposal and there we describe how we want to have a B2 API and yeah we in the past treated the metal API just as a low level API so there's no real concept of resource hoping.
And yeah we wanted to change this now so grounded it's not required to have this but actually we want now to treat the API or users.
Yeah with scoping because I think it would make a lot of things much more secure in terms of misusing a token for something different.
Yeah this is one thing that we're going to work on this year and I hope we can finish this.
This will be then GRPC based and yeah we also already have some ideas how to implement this.
And then one other topic that we want to tackle this year is to broaden the hardware support because as I already said we only support limited set of hardware and yeah then it makes sense that we go through this again and maybe propose like a standard API.
Interface or something like this which should work out of the box but if a vendor deviates from the norm then it does really not work but at least you have a like default implementation for how an IPM I mentioned server it should work.
So these are stuff that we want to work on here this year.
And there's also possibility to try out metal stack if you want so we have one project which is called the mini lab and there the control plane will be.
Yeah started in the Kubernetes and Docker cluster and then there's another open source project which is really cool which is called container lab which helps you to emulate switches on your local machine with the help of VMs so it will also spin up to.
And then there will also be like a quamo virtual machines which will be started with specific bios which does a pixie boot with the EFI and then you can provision these machines locally without having to install anything in your data center just on your local machine.
So yeah this is a way how you can try it out easily without investing tons of money into a new hardware or something like this.
So yeah thanks for having us I mean great great pleasure to be here at the first them and I hope that with our project we can give something to the community here and we're keen to hear about the use case that you have and the journey.
And yeah for us here we have the documentation there available for you if you're interested in it the development takes place on GitHub and yeah we have a community selection where you can also join us questions where I think our open minded people so just reach out to us if you have any questions and yeah I guess that's it or do we want to add something.
Okay then that's it and thanks.
Questions.
Okay a question was if the traffic that's going through our switch player is encrypted for every tenant right.
Noter the data plane traffic is not encrypted by default inside one data center there's if you want to encrypted the traffic between data centers you can do this with.
Maxack GPX for example, which are available for available for these type of citrus we use any further questions.
Yeah you said it it manages the racks machines network files so does it replace like net box and I found as well.
Yeah it does it reflect does it replace net box most of the question yeah yeah we started using net box at the beginning.
We thought we can use net box for the ipom at least but it was to slow for our use case and we implemented that natively and go again it's called go ipom quite popular repository at.
For now and yeah we have basically a whole CMDB built in yeah which is not net box.
Yeah so what type of hardware is supported most of the questions so we on the server side we actually support super micro from X 11 and you are.
Dell with iDrag 9 and it's carried said sonic switches so every switch which is capable of running the sonic operating system.
Should be supported and GPUs are actually only the Nvidia RTX 66090 and the H100.
Any more questions.
Thank you.