Okay. Welcome, everyone. I'm back. I'm Stefan Agarzarella. I work in the Redoubt virtualization team.
And in this talk, I would like to go through all the ways we have in Linux, currently,
to do to emulate the Vertejo device. This talk was mainly inspired by Thomas, because last
year, I talked a bit about that. In another talk, here at Foss, I'm about Viewstuser.
And Thomas was happy about the comparison between all the way we have to emulate Vertejo devices.
So I decided to go a bit deeper and show all the pro and cons we have on all the way in Linux.
So before going into the details, let's start about what is Vertejo. Maybe most of you
already know everything about that. But essentially, is an open specification. The current
version is 1.3. There is the 1.4 out for public review. And what they wrote into the
spec is the purpose of Vertejo is a straightforward, efficient standard and extensible mechanism
for virtual device. So essentially, they provide, they define a specification for Vertejo.
What is what the spec covers? So essentially, there is the first part, they define the core
component, which each device used to do, for example, feature negotiation and that kind
of things. Then there is an interest pack for M, a part to define the initialization phase
for the setup. They define all the transport that Vertejo device supports. For now, we have
PCI, M, M, M, M, M, M, M, and Tonellaio. And then there is a big section for each device.
So that's where a block and so on. About the core components, we can say the three main components
are the control path, the data path, and a way to exchange notification. The control path
is mainly used for feature negotiation. So driver and devices essentially agree on which feature
they need to use. Because for example, we can have a new device with a whole driver that
is not able to use the new feature. So essentially, they can also vice versa. So it can be
a new driver and the whole device. So they do this kind of feature negotiation to agree
on what to use. Then there is a configuration space, which is used for example, in an
actor card, is used to set up the Mac address, in a V sub device, the device will tell
to the guest, this is your guest, the CID, and that kind of things. And then the configuration
part is used to set up the data path. Most of the, most of the memory is always allocated
by the driver by the guest. So like the data path. And so the guests need to tell the driver
need to tell to the device, say ER is the word queue. And so the word queue is essentially
the main part of the data path. In the picture, I draw the split with queue. And in the
spec, there is also a new version, which is called part with queue. About the split, essentially
we have three memory areas. One is the descriptor table, which mainly contains pointer to buffers,
with the length of the buffer, of course, flags, that kind of things. And then we have two rings.
One is the available ring, and the other one is the used ring. The available one is used
by the driver to put a century to expose to the device, a new buffer. So for example, new pocket.
And the used one, instead is used by the device to tell to the driver, hey, I use the
buffer. Now you can take it and clean it or reuse or whatever. And in this pocket with queue,
everything is parked into the descriptor table, mainly for hardware, because we will see
now we can, or even hardware are able to expose birthday of devices. And having three different regions
can produce a lot of memory access, a lot of PCI transactions. So in order to reduce that
content, they define a new format for the bit queue, which everything is parked into the descriptor table.
It's a bit more complicated, but can work much, much better with hardware.
And about notification, we have, we call kick the notification from the guest to the host,
so from the driver to the device. And I'll queue, of course, the other way around.
The devices. We have a lot of devices to define it into the spec. It's an open spec. So if you
want to define new device, you can easily open send a serious to define a new one.
The main one, we have a network card, birthday on that, a block device, birthday of block,
V-sock, virtual socket. The first talker about a lot about V-sock, we have a birthday of V-sock,
which support the AF V-sock address memory. We have birthday F-s for shared folders and others.
Okay, let's go into the topic of the talk. This is just an overview, then we will go
in each way. So the first way, common way to do device emulation is to do it inside the V-m-m.
So inside Q-emo, this is kind of the standard way, and also one of the key point of V-m-m is exactly
to do device emulation. And Q-emo, in our case, provide a lot of birthday of devices.
Then we can, in another way, we have is to move the device emulation into an external user's process.
The main purpose is for security and reliability, but as we talk about in Rasmium,
MAM is also for flexibility. We can write in a different language and other reason. In Linux,
for now, we have two way. We have the V-s teaser, which is the one I told before in Rasmium MAM told,
and V-dias, which is V-dPA in user's space. We will talk a bit more about that later.
Then, another way to do, to emulate the birthday of devices, are into the host kernel.
And this is mainly for performance, because we are inside the kernel. We save a lot of
system calls, and currently we have two ways. One is the V host devices, and the other way is
a V-dPA software device. I mean, the V-dPA one is a potential one. For now, we don't have any
V-dPA software device implemented into the kernel, but we did some proof of concept, so it's kind of
doable. And the last one is to move the device completely into the hardware, and this was the main reason
for V-dPA. Again, for performance, scalability, and that kind of things. So, going back to the
first way, the V-dia device emulated by the V-m-m. As I said, as a commotionary, we can say,
Quimo is the factor reference implementation of V-dia devices. You can find a lot of them,
not all of them, but most of them implemented in Quimo. And why to do that? Mainly, simplicity,
as I mentioned, I mean, a role of a V-m-m, one of the key role of V-m-m is device emulation. So,
you have all the API to access the guest memory to inject interrupt and that kind of things.
And that's the portability. If you write a device in Quimo, essentially you can run the device
on all operating system that Quimo supports. Drawbacks are a reliability risk, so your device
is in Quimo, a potential bug in your device can affect Quimo at all, and your V-m can go down.
Another is the performance overhead. So, for example, if you have a natural
card and a driver need to send a pocket. So, essentially the virtual unit driver will copy a
pocket into the V-q, then send a kick. So, as a V-m-m-m-sit, KV-m will write into the I-o-event of
D to wake up Quimo or the I-o-trad. And then there is the device running in Quimo. We'll access
the V-q, pick the pocket, and maybe send it to a top device, so you need to do a write. So,
again, a system call, go back to the card now, send the pocket, then return from the system call.
When the pocket is sent, Quimo, the device implementation will move the descriptor from,
we will put the descriptor into the used ring, and then need to inject an interrupt. So, again,
another system call to write into the RQFD, to tell KV-m, hey inject an interrupt for this device.
So, there is a lot of a lot. A couple of system calls for each pocket that the device needs to do.
So, how to use a virtual device in Quimo? This is a simple example, where essentially we have two
blocks that have option, which are part of the Quimo block layer, and we are telling to the
Quimo block layer, open the Fedora, QQF2 image, and use a QQF2 protocol on top. And then we can link
this QQF2 image to a device. In this case, is a virtual block PCI device, but we can use even
NVMe, emulation, or a virtual escazi, or any other kind of block device that Quimo emulates.
So, we need, we just need to change the device here. V-host. Okay, this was initially introduced
exactly to improve the performance of a virtual unit device. As I mentioned, when you need to send the
pocket, you need to do several C-school. So, what they did essentially is to move the device emulation
into the kernel. The control part is still intercepted by Quimo, and the kernel will expose
a couple of Y-o CTLs, that the Quimo's mold device can use to essentially overflow to the kernel,
the device emulation. So, mainly, where are the virtues in our access them? And at this point,
and the data part will completely be handled by the device into the kernel. Currently, in Linux,
we support three device, V-host-nath, V-host-cazion, V-host-v-sok, why mainly performance, as I mentioned,
having the device into the kernel device, for example, to talk with the top, don't need to do C-schools.
So, we will see those kind of C-schools, or reduce latency, and improve the throughput.
But, for other use cases like V-sok, the other reason is being into the kernel, we are really,
it's more easy to interact with the kernel layers, like the socket layer. So, in the case of V-sok,
we will need to open a socket into the kernel, and having the endpoint of the connection
into the kernel will make us pretty much easier to instantiate a new socket for AFV-sok.
Drawbacks. Yeah, of course, we want to go for us, so we need to pay something,
and there is a security and the ability to risk. Essentially,
a bug into the device, if in Qem, we can crash your VM, ER can potentially crash your entire host,
and security again, there is an attack surface into the host kernel. So, yeah, we need to be careful.
Upgradability, if you want to provide a new feature, you need to update your host kernel,
portability, this is Linux only, it's only implemented on Linux kernel.
How do you use it? It really depends on the devices, for V-sok, it's very, very easy,
all the complexity is eaten by Qem, so you just need to have a V-host V-sok device,
and specify your guest ID, then Qem will set up everything,
we'll create the device for the guest, and then do the ACTLs to upload the emulation to the kernel.
V-host user, I already talked a bit about that in the previous talk,
in Rust VMM, but essentially, the V-host user was inspired by V-host, but instead of having the
device into the kernel, the device now is moving to another process, another user space process,
completely separated from the VMM and from the kernel into the host.
The control part here is a unique domain socket, because we need to move some file descriptor
around, mainly for the shared memory, so Qem needs to allocate the guest memory in a way that
that memory can be had just by a file descriptor, and then pass the file descriptor to the device
using the unique domain socket, and the data part at this point at that point will be handled
completely by the device accessing the guest memory. So why to do that?
Yeah, essentially to mitigate the security and the reliability risk of V-host, because now it runs
completely in another user space process, so if that process can be put in a sandbox, for example,
to avoid any escape, or can any crash, can crash just that process, we can easily restart
the new process again, doing the emulation. Another advantage is upgrade ability, so if you
want to update your device, you just need to run a new process and potentially just upload plug
to your VM, and flexibility. We talked about it in the Rust VMM tool. With the V-host user,
we can essentially write the device in any other language, like we did in Rust VMM, we wrote
them in Rust, and use it in Quimo pretty easily. Draw back a bit more resources, use it,
because of course we have other processes. All devices will run in another process, a bit more
resource consumption, and more coordination. We need to start all the devices first,
then start Quimo. So there is a bit more coordination, but this could be hidden by
management layer like LibVirt, which is used essentially for that. Portability, I talked about
that last year was of course Britain on Linux, but it's not really Linux specific, is any
POSIC system can run a V-host user, because the only thing we need is Unity DomainSugget,
and a way to allocate shared memory address by a file descriptor, and in POSIX, we have
the SHM Open C-School that can be used. How to run it? Okay, it's a bit more complicated,
so in this case, I use it as an example, the Quimo Storage Demon, which is an standalone application
that provides exactly the same block layer of Quimo, but then can expose the device, the block
device in several ways, and one way is a V-host user block. So this two lines are exactly the same,
I use it in the first example, so I'm telling to the Quimo Storage Demon block layer, which is
the same of Quimo. Hey, open the Fedora Cucough2 image, use the Cucough2 format on top, and then export
this image as a V-host user block device on this Unix DomainSugget, on the TMP V-host, this socket.
And then we can start Quimo, open the Unix DomainSugget, attaching it to the device,
to the V-host user block PCI device. One thing we need to use is the memory backend,
a memory backend, which is shareable. So the memory backend essentially is the
way how Quimo allocate the guest memory, so if you don't use it, if you don't specify
the default is ROM, I think, memory backend ROM, or something like that, but essentially Quimo
will do just the mall lock, and which is not okay for the host user, so we need a shareable
memory backend, and the MMFD is for the by default is shareable, because it's the the memory
is essentially adjust by a file descriptor, and we can share the file descriptor to the Unix DomainSugget.
This is Linux only, so if you want something portable, you can use the
memory backend SHM that we recently introduced in Quimo 9.1 last year.
Okay, last one, BDPA, which is Vertio data path acceleration. In this, VDPA was mainly
introduced for hardware, but we will see we can do also software accelerator with BDPA.
In the case of VDPA, the data path must be Vertio compliant, so the device need to understand
the virtues and how to elaborate them. The vendor path, sorry, the control path could be
vendor specific, so in the case of the hardware device, we need a little driver into the kernel
for the control path, and why? Mainly, again, for performance, we are floating the entire device
to the one example is smart nix, so this smart nix now are some of them are able to expose
Vertio devices, so we will save several cycles of the CPU to do the emulation, because the
device will be directly exposed to the to the guest, and the VertQ share directly from the guest
to the hardware. Yeah, physical, they need to, I mean, some of them are implemented with arm
cheap, and they need to be able to understand the VertQ, so the driver essentially will tell
to the smart to the hardware, hey, here is a VertQ, please use it for your device to do the emulation.
It's not very common, you need to buy, I have the driver, the slide later, but yeah, it's not
really common. Another advantage is essentially we support with this, we support both VAM,
workload, but even container, because VDPA, we have a concept of VDPA bus into the kernel,
where we have the Vhost VDPA bus, so if you are touch the device to the Vhost VDPA bus, the device will
be exposed in similar way of the Vhost, the second one we show, so the QEMO needs to use
Iosity else to set up the data, but exactly in a very similar way, but we have also another bus,
which is the Vertio VDPA bus, so if you attach your device to that bus, the driver will be loaded
into the kernel, into the host, so the VertQ will be allocated by the host, so you can attach
your device directly into your host and use for bare metal application or containers.
Another advantage is the unified software stack for VDPA, so as I mentioned, it was introduced
for hardware, but we can write even user space devices, it's called VDOS, which is similar to Vhost
user, but requires some coordination, some models into the kernel, and we can also have the
devices into the kernel itself, like Vhost, as I mentioned, for now we don't have any device
like that, but we have VDOS, I have an example later about VDOS, the advantage of VDPA that
all of them, all of these three ways to implement a VDPA device are exposing exactly the same
way to QEMO, so QEMO hardly need to support Vhost VDPA, and then for QEMO point of view, it doesn't
matter really how the device is implemented, drawbacks, portability, this is Linux only, so you need
things into the Linux kernel, it's not like the Vhost user, which is completely agnostic, so it's
really Linux specific, and about maturity, it's still working progress, so we is supported by few hardware,
not a lot, we support few, I mean for now I saw just net and block supported, has a birthday
you devices, and of course the cost, you need to pay, if you want the other, if you want the
device into the hardware, you need to pay for the hardware of course, and this is example, again
using QEMO's storage demon again, because QEMO's storage demon can also expose VDOS device,
so the first two line again are exactly the same, but we are turning to QEMO's storage demon,
they expose your devices as a VDOS block, and with the name, then we can use the VDPA tool to
instantiate really the device, and at that point we will find a CISFS entry that will tell us,
hey this is a Vhost VDPA 0, and so to QEMO we can just use the Vhost VDPA device PCI with the
tractor device, where QEMO can do the AUSTL to set up the VIRTQs, and yeah that's it, thank you, any question?
The question was about if we can use a bpf essentially to do the device emulation into the
kernel, can can be an idea, I mean currently we don't have any of kernel device VDPA kernel device
implemented, but this could be potential one way to do that.
The question was why not just passing all the PCI device to the gas to do, I mean you can do that,
the main difference is in this case, I mean in that case your entire PCI device will be
assigned to the gas here, so you need to support all the PCI stuff, here the device can be
emulated, can be implemented in any way, that only thing you need, of course you need, I mean
your device can implement several of this virtual function, and the advantage here is that
it expose a virtual device, so in your application you just need, you don't need to change anything
into the gas, because the gas is already able to have, I mean hopefully I have a virtual
driver, can just use it, so you don't need your driver for your PCI device into the gas.
Yeah the question was, several years ago already were VDPA out, so why it's not so popular now,
I have no idea honestly, but I mean maybe because the hardware vendor want to
lock in, maybe I don't know, or maybe there are some kind of bottleneck when using that, I don't
know, I mean I'm not working on VDPA, but sorry we don't have time anymore, but if you have any
question I will be out. Thank you.