Welcome everybody, we say in the cloud native world, but with a different project and
same, always fun, so round of a pass.
Hello everyone, I'm Kabatis Pistadinos, and today I'm going to show you a fork done by
me and my colleagues in fourth grade in Greece.
It's called optimizing loan for hyperformans hardware.
Okay, so first of all, I think most of you know about Kubernetes, but have an introduction
slide.
Most of the production now comes from containers, so you make a container in your ship,
you get the flow of your things, and most of the job is on the cloud, so you have
Kubernetes for this, orchestrating your containers, it's kind of like a cloud operating
system.
It does the resource management, the working and manage the same failures done, so you just
describe what you want, there's an abstraction based on primitives and conveysions.
You describe what do you want from the hardware perspective and you get it, and you get
the manage platform for deployment and development.
So you can have Kubernetes on local, on prem, or on cloud, and because of this, there are
a lot of extension and third party tools implemented.
In this production we'll talk about storage, so the basic component for Kubernetes is the
volume object, they have standard API, there are storage classes, you have persistent
volume claims and additional volumes, you do request using persistent volume claim, and
the persistent volume is the actual storage, you can implement the storage class, and
it's how you will get the actual storage.
Because there are a lot of storage interfaces, and there are a lot of implementations done
in the last years, so there are containers stored at the interface, it's an API in
the interface, in order for you to introduce your custom storage offering, and you can do
rather move your storage, you can implement your own lifecycle for the volume, and you can
do synapses and cloning, and there are a lot of implementations, and one of those is
long-hurt.
Long-hurt is an open source, a project part of the CNCF, it's in to baiting project, it
remains a complete software defined stored at CNC, in comparison with other storage
solutions, it's complete, it means that it's not relied on any other third-party tools
and providers, and it also have advanced features like synapses and backups, there are
a few technologies, you can see here in the figure, the overview of the architecture, you
have the CSI plugin with stocks, with communicates, with a Kubernetes, UI for the user, you
can describe from the UI what volume you need, and what node do you want it to, and the
manager gives to you, and the actual data path, the actual data IO path is done in the
engine, where you have the controller, when you need a volume, you issue a request, and the controller
runs along with the node you want the volume in, and there are replicas running in all the
same node or another node, the communicates through TCP protocol, and the replica is where
the actual IO is happening, and the user SSDs are whatever actual storage device you want,
and so anything is the core of the implementation, is where the actual work happens, that's
where the main focus of our work is, and its volume is implemented by only one controller,
so its volume has one controller, and you can attach as many replicas as you want, okay,
so let's see how the performance of the system, nowadays most of the people use cloud,
like AWS or Azure, to run Kubernetes, and in these, the performance there is okay,
because there are limitations in AWS, around 40K IO, you need to pay a lot more in order
to not have that limitation, but when you have on PEM where there is high speed networks,
then VMA and you have a better computer, you are limited to about a 50K to a defyK read right IOPS,
where your actual device supports 400K IOPS, and so in order to find the bottlenecks and what
we can do about them, we are so late at the end in, we split it into three basic parts,
the front and the controller and the replica, and we found the three major bottlenecks, from
the front and the perspective, the use there is quasi-planetation, it is an interface for the
kernel to expose a block device in the user, and the connect the IOPath and redirected to your application,
but it is actually really slow, the controller has to forward all the, all the requests
in the replica, and the user, the wrong communication protocol, that serialized all the operations,
and that's a big bottleneck too, and at the bottom part of the ending, we actually replica,
where they actually read right happens, the user's parts file implementation, where rights are
are very, very limited performance, especially when using multiple snapshots, okay,
so what we can do about it, we explore our alternatives, and we found that in the front and part,
we can use UBLK, it's a new interface, based on IOEuring, which is available in the latest Linux
kernel, but it's in the new Ubuntu 2, in the controller part, we implemented, we found what's
the problem, and we can implement the logic of the communication, in the replica part, we implement
a different storage mechanism, we call it direct blocks, it's a custom direct disk storage layer,
okay, so let's see part of part of what are the improvements, so for UBLK,
UBLK consists of two components, major components, UBLK driver, which is in the kernel,
you can issue a request there in order to get a block device, to get to expose a block device to
in the user space, and there is the UBLK, SRV, which is in the user space, and deployments,
they are IO, it redirects the data wherever you want, you can write a custom driver in order to
get it to done what you want, you can either implement to be in the loop, you can connect to
the loop device, so you can implement it to send the data to the long-horned housing,
which is very modular, you know, just a field, some functions send the UBLK, and it can be used
for IO, and that means task like adding more devices, and why is it that fast, it's based on the
IO during, which is a Linux kernel system called interface, it supports a synchronous operations,
it's based on the logic of the two circular buffers, the one is for the request and one is for replies,
the applications in the kernel, it can communicate through them without actual system calls,
and that means you can do button and the last memory copies, and that's why it's really fast,
you can see you will see the numbers in a minute, okay for the second part of the engine,
the controller replica communication, in the top part of the figure you can see the original
implementation, for every request, a go-long thread, a binary quest, user request channel,
and the problem is that here the loop thread is just a single thread, a single thread,
running and serving every request, so they use a message map to store all the requests,
in order to identify which one is the one coming in the response, and that operation is synchronous,
meaning you have a drop of performance because of the single loop thread, and then you
forward the message to write thread and the right thread issued the request in the replica
and waiting for the reply. In our approach, we saw that the single operations are not good,
so we used an ID channel, and the message array, when a request is built, you get an ID from a
go-long channel, and you use it as an index in this message array, that way go-long channels
assure you that you don't have concurrent, you can have concurrent readrides in the array using the
index, and that way every request issuing to the right thread, immediately not waiting for the loop
thread, it's iteration of the loop thread. Okay, in the last part of the, of the ending, we have
the direct block store, or DPS for short, it can be used in a file or directly in the device,
it supports multiple volumes, now it's very volume, it has an accessing API, you can use it from CLI
or call the API in your code, and the main point of it, it divides the store, the storage in four regions,
it's a super block, the volume is an absolute metadata, the extend metadata, and they use it data.
You can, it's really light-wet in fast, and the extend maps are getting memory,
as you can see, it's around 40 megabytes per one terabyte volume, so it's operations not that
actual in the disk, it's in the memory, it's really fast, an extensive use of meat, but meat,
beat maps, it's done here, so it's actually in the fast, it's written in GoLank, it's open source,
okay, so in the left part is the original implementation, the user has cast a block device,
they get it demon, in the server, here is the old communication implementation, and right here,
sparsified storage and file system goes through the SSD, in our implementation, we used a Ubell
K-block device, communicating with a UBD SRV demon, communicating with a front end,
here the controller has a different approach, and here the device storage is implemented,
so using the whole long-core nanzing, you can see you have 50k read and 25k right,
and okay, in the first layer here, we didn't know what operation for this part of the system,
so in this experiment, only the front end is used, after the front end, long-core can
not long-core now actually, i-scasity G-t can hold up to 60k IObs, but Ubell K can go up to
500k IObs, as you can see the front end, the Ubell K front end has way more capabilities compared
to why scasity G-t, okay, so now because i think it's kind of complicated,
for this metric, we placed the front end part, and we still used the all the communication protocol,
but we also didn't do the actual IO, we stopped the data path IO here, and the
no warp is done, so here we see we levered some of the IObs from the Ubell K, but we're still at 100k IObs,
replacing the old communication method with our new communication, we go a lot to 100k IObs,
and now using the Linux parts file implementation and the very custom implementation,
you drop back to 128k IObs read and 38k IObs write, and DBS,
introducing DBS in this implementation, does the system to 150k IObs, so as you can see we can't
triple the performance of the system using three major changes, okay, so in conclusion,
Ubell K has the biggest impact in accelerating IO, it's a new technology using state-of-the-art
solution for doing IO, so the question is, is NVMFNS-serie, when you are not using over the network
operations, it's actually not that the Ubell K has the capabilities to perform it in some situations,
the second you can do further performance improvements, especially in the controller as you can see
the performance dropped massively inside the controller, we are working on optimization in adding more
features, and in the last part DBS is actually not the novel, but provide a helpful utility for other
projects too, okay, so some pull requests have been submitted, DBS and other projects can be found
at GitHub of Carvice's force, then you can also find the code of the modified code of long
on there, I want to acknowledge the back of our founding this job, thank you, many questions?
Yes?
Sorry, I don't know a lot of very well, but you said a poorly instillified one controller and then a number
of replicas. Yes? To support high availability of it, so failure or something?
What, have you have the ability failure? Yes, yes, yes, all that is done by the manager,
Longhorn is, I don't know how to speak, okay, can you say the question for me the
bit? No, you're like, well, I'll explain what I mean again, so you have the simple controller
multiple replicas of your data that allows for the distributed high availability of volume
is that right? Yes? Can you repeat the question? Yeah, I'm looking forward to you, you have to
repeat for the microphone. I get, but I don't remember the whole question. Do you see the same
performance improvements when you're using distributed multiple replicas?
Multiple replicas. Okay, so the question is if using multiple replicas has the same effect,
our change has the same effect using multiple replicas. Okay, we haven't used multiple replicas,
the only bottleneck using multiple multiple replicas is the network. Using a bigger, better connection
in the network, it's not the bottleneck anymore, so yeah, you're going to have the same effects.
Thank you. But I haven't done any actual numbers for that.
Any other questions? Yes? I'll see the forms improvements of the concept we need to be
to take the understanding of the phone point. What was the question to take the question?
Ah, you asked me about compared to the version two, how it's compared to the
output of the one data and the phone point when you achieve it? Okay, the question is if this is
the version one or the version two of long one, right? Yeah, this is the version one, the version two of
long one will use a speedk and VMOS. In GitHub, I can see the issues and the one two improve
the version one two, so that's why we'll focus on the version one two. version two with a speedk and
VMOS will solve the problem of the front end, but most of the other parts are not updated.
I mean the communication between the controller and the replica is not part of the improvement
in version two and also they will still use the Linux parts file in the end.
So there's also need to improve the version one, but version two is really good.
Yes?
I think that's configurable. Sorry, you asked me if NVMe or F is over TCP?
Maya start, no? Maya start to open EBS. Maya start using something.
Okay, I think when you have on-prem, you have when you have the controller on the same
note, you don't need to use TCP, you can just go and extract it to do that.
I'm not sure, I'm not familiar with Maya start with the implementation.