OK. Hello, everyone. I'm Stefano Garzarella. Today, we will try to answer this question. So,
if we can run Qemu and Vostuser, or an operating system other than Linuxware, a special Vostuser
was developed. Before going into the date, when we talk about Vostuser as materials, already
mentioned, we are talking about Vortio devices. So, Vortio is a specification, if I think
most of you already know. It is a specification to create standard devices for virtual machines.
I put here some links to the last spec, and also to the GitHub repo where the code of the spec
is. And essentially, the specification defines the core components of every device. The
initialization steps that Gaston host needs to do in order to create the communication channel.
The transports that Vortio supports, which are PCI, MMO, and Trialio, and then there is a big
section for device types. So, hold the device that are defining it into the Vortio spec.
When we talk about Vortio, of course, we are talking about devices and drivers. So, usually the
Vortio driver, sorry, let us start to the device. Usually, the Vortio device runs into the host,
and the guests will have a driver, a Vortio driver, to communicate with the device. The spec,
I mean, defines essentially the specification. And implementation, of course, defines
three paths. One is the control path, the data path, and the notification mechanism to
to wake up both sides. So, essentially, the control path is used for feature negotiation,
so the driver and the device will understand what they can use. There is a configuration space
to expose some information from both sides, like macadras for net device or context ID for V
so, can stuff like that. And then, of course, the configuration is the control path, sorry,
is used to set up the data path. So, the driver allocates all of the components for the
data path, like the Vortio use, and then, to the control path, share to the device where they
are into the memory. The data path itself is mainly, is mainly built on top of the Vortio,
which we can see as a ring buffer. The spec defines multiple format. There is the split format,
which is the initial one, and then they define that. So, another one pocket, the pocket will
queue, which is more optimized to reduce bus transaction. And as we will see, this was done,
in order to move the device also into the hardware. And then, there is a notification system.
Usually, we call kick, the notification coming from the guest to the host, and our queue
or interrupt the other way. Now, the question is, where the device is emulated? Of course,
into the host, but in which component, the common scenario is into the VM. In our case,
we will talk about Qemu, that we can consider the reference implementation of Vortio
specification, most of the device of Vortio devices are supported by Qemu, and Qemu can
easily intercept the control part, can easily also handle the data parts, it has complete
access to the guest memory, and can use, for example, special mechanism from KVM, in order to implement
an notification like the event of D for inject IQ or get notified when some register are
written by the driver. Now, we have also other way to emulated the device into the host.
The other way is Vhost. Essentially, it was introduced initially to improve the performance of Vortio
net. The idea is to move the emulation of the device from the VMM to the host kernel. And
in this way, the control part is still intercepted by Qemu, and then need to be propagated to
the host through an IOCTL's API. And the data part will be completely handled by the kernel,
by the device implementing to the kernel, because the kernel can attach the address space of the VMM.
Linus currently supports three devices, VoSNet, VoSKazi, and VoS VsoC. And as we mentioned,
the main advantage are performance, because we can skip a lot of C school. If we think about
Vortio net device, that usually is attached to a top device. In it's to do a lot of system
calls, because for example, the driver put a pocket into the Vortio needs to notify the VMM. So
do a VMM exit, go into the kernel into the host, then the kernel will notify the VMM. And so
the VMM needs to communicate with the top. So it's a system call, and then it's to, for example,
put the device to put the request into the user dream, and send another C school to inject the
interrupt. So we have a lot of C school. If we move everything into the kernel, of course,
we reduce the number of C school per request or the latency and also the throughput will improve.
Another advantage that we use it for VsoC, for example, is that since we are into the host
kernel, it's easily to interface with kernel stacks, like address family, like the VsoC address family,
so we can easily communicate to the VsoC stack and put the pocket there. Of course, we have
drawbacks. So safety the device is into the kernel. So if it crash, yeah, you can have issue.
And it is really Linux specific, because all the devices are provided by the Linux kernel. So they
are really into the Linux kernel. So you cannot use it without them without the Linux kernel.
And if you want to update the device, you need to update your host kernel. So maybe for fixes,
should be okay. For new feature, it can take time. So another option, which was inspired by Vhost,
was the Vhost user. Mattias also really talked about it. The control part in this case is
done by a unique domains. So essentially, the device in this case is also moved out from the VMM,
but instead of the kernel, it's moved to another user space process. And the control part,
as I mentioned, is a unique socket, but it's really similar to the IOCTL. So the IOCTL of Vhost,
they define very similar messages. And the data path, in this case, is implemented to a shared
memory. So Kimo needs to allocate the guest, RAM, in a special way, in order to be able to
share that memory to a file descriptor. So the memory should be addressable by a file descriptor,
and then pass it to a unique domain socket to the other process. The main advantage is our safety,
is an external process, is even external to the VMM. So if it crash, we can easily reboot it,
and everything should be fine. Device updates, again, is a user space process. We can easily
hot-homplug it, start a new version, have hot-plug it live. And you can write it in a different
languages. We will see, as a lots of Matya mentioned, we have a lot of them implemented in Rust,
because it's a completely an external application. And of course, more isolation. You can
confine, you can put it in a jail or a container or whatever, or a special C group,
just needs to have a unique domain socket to the VMM. Throwbacks? Yeah, the performance
again, the process is moving to user space. So again, we have to do C schools, but there are
other techniques we can use like DPDK, SPDK in order to, or IOU in order to reduce the C
school. And we need a bit more coordination, because that application need to be spawned before the
VMM, but thanks to management layer, like Leapware, this could be completely hidden to the user.
And it's Linux specific. I mean, it was Linux specific. It's not really, it's a spoiler of the
talk, but yeah, it's not really Linux specific. I want just mentioned VDPA, which is another
framework we are developing, where in this case the device is moved into the hardware. Now,
we have a lot of smart nicks, DPUs, so thanks to this framework, Vertejo can also implement it
really into the hardware. If you're interested, we already did a lot of talks about it.
I will not go into the detail. So let's go back to the ViosT user. I put the link to the
specification here. The specification is maintained by the Krimu community, and it's the
finally there's exactly as a control plane to essentially share the word queue with an user space
process on the same most. We use the term of front-hand and back-hand to identify the application
that shares the word queue in our case QIMU, so the VMM, and back-hand the consumer of the word queue.
So usually the application that implements the that emulates the word value of the device.
And the key components, the key components of the specification are the units domain socket,
we already mentioned, the Ancillary data support of Unix domain socket. So essentially,
that support allow QIMU to QIMU allows application to share file descriptors and access to
that resources. That file descriptor could be, in our case, could be a shared memory, which is another
key component. So we need a way to allocate memory that could be shared, passing a file descriptor
to the other process. And so the other process can easily mop that memory. And for the notification.
So vostuser was based on eventfd, which are, we would see Linux specific. But for example, QIMU
is the, I mean, support automatically a fallback to pipe or pipe to to implement notification.
I mean, I interrupt the injection and kicks coming from the, from the driver.
So vostuser input on policy. This was the main question. Can we use vostuser on system other than Linux?
The answer is yes on POSIX is true on Windows. I don't know. But yeah, for for for for other
system that are POSIX compliant, the answer is yes. And here I put all the, I put the links to the main
feature, I mean, the main C is called the finite bypassing. So in the case of, let's see about
the key components. We already talked about Unix domain socket. Yes, they are into the POSIX
specification. And see the read data support. Also, they are defining it into the spec. So they
define the C message either structure that is used to classify the scriptor and SGM rights.
Then share the memory. Yes, the, the, the POSIX specified the SGM open C school, which is exactly
defined as a way to create a connection between a shared memory and the file descriptor. That is
exactly what we want. And then notification. As I mentioned, event of D, which are using on Linux,
are Linux specific. Maybe free BSD also now support it. But I'm not POSIX, but we have PIPEN
PIPQ and Qemu and other VMMs already support the fallback to PIPEN PIPQ for the notification.
So yes, we can. The Qemu already supported most of them. So Unix domain was there and
cillary data notification. The only thing we, we missed was then the way to allocate shared
memory with SGM open C school. Now, let's take a look on how Qemu handle the guest memory,
the allocation of the guest memory. Usually we identify that with the Qemu memory backends.
When you create a VM, you can use the simple dash ham option to specify just the size of the,
of the memory. But you can also use something with the machine option. I mean, you can specify
for your machine, which memory you can use. And this is a more advanced way to define your
memory. Because some of that memory backends support an option, which is shared option,
that is exactly what you want. If that option is turned on, means that Qemu will try to allocate
that memory in a way that can be shared with an external application. Now, let's get, let's
take a look of the Qemu memory backends. So the first one I want to mention is the RAM memory
backends, which is the simple one. Essentially it's the same user when you specify dash with the
memory size. But you can put more option like pre-allocation. So you can ask Qemu to pre-allocate
everything and and other options. The other one I want to mention is the file memory backend,
which is one of the memory backends that support the share option. In this case, the parameters,
the main parameter is a path into the file system. That could be a simple file, but also a specialized
for shared memory or a huge page file system. And as I mentioned, this memory backend can be
used to share the memory with another application. But the most interesting one is the memory,
the memory backend. It's an anonymous, it's allocating an anonymous object. With an anonymous, I mean,
it's not had just by any file into the file system. It's just had just by a file descriptor
that can be shared through the units domain circuit. And this is exactly what we want for
users. So when you use the user on Linux, you can use the easily mammothD, where the share option
is by default true because it's exactly the use case. But it Linux only. So it used the mammoth
decreate, which is not in POSIX. So what we did, we had the new memory backend, which looks similar
to that. It's called memory backend SHM, which do exactly something similar, but using the SHM
Open C-School, which is POSIX, I can use it on previous D, MacOS, or all of them that support POSIX.
That POSIX is called. And it's available from Qemun9.1. And then there are other backends, but
we don't care. So this is the main series where we implemented that kind of things. So most of
them are upstream from 9.1. The red ones are not. We will see in the next slide. Why?
So the first part was just fixes, because when we run the Qemun, the Vostuser application on
other system, we discovered some assumption that we didn't Linux was not true on other operating
system. Then we enabled it. The main thing was the new memory backend, and then we enabled some tests
that essentially highlight some issue that we still didn't fix. So for this reason, we didn't
merge the three patches in red here. I mean, that patches are now I'm maintaining on my fork,
but which I recently rebased. Essentially, if a test is failing on previous D and MacOS,
that the test is the Q-test test, but the main one is the Vostuser Reconnect test. So essentially,
that test was never run on that platform. So we don't know if it is an issue into the test,
or something we didn't fix into the Vostuser implementation. But on 3BSD, X36 hosts,
you can easily reproduce running the power PC 64 using TCG. So using binary translation,
it's every time the issue. On MacOS, on the harm 64 MacOS, running the harm 64 test,
you can hit the issue frequently. So it was hard to debug, and we didn't have time to go into the
date, but yeah, this is something we need to hardware. If you want to try, you can use this this fork.
In this case, I put an example where I essentially shared the root file system
through a Vostuser block device through the VMM. So the root file system is, in this case,
the Vostuser device. In Q-emo, the Q-emo repository contains two application that can
expose file into a Vostuser device. Vostuser block device. The simple one is Vostuser block,
which supports only a row file. So you can expose a row file as Vostuser block device. So you can
use the Q-emo repository team on which is a more advanced application, because it supports the
entire Q-emo block layer. So you can share every file that Q-emo support as a block device.
Like a Q-CoV-2, or also a ZFNBD remote disk. You can share it as a Vostuser block device, or also other
experts like NBD, Fuse, and someone else. If you are interested with, we talked about it with
Michael Lee Kevin Wolf in KVM for some years ago. And then you can start K-emo in this way.
I mean, I put in bold what we need for the Vostuser block part. The other is really depends on the
on the system, because in Linux we can use KVM, Accelerator, or MacOS, we can use the HF Accelerator.
But essentially, in order to attach to the Vostuser device, you can use exactly the same option.
So first of all, the memory back end. In this case, we are using SHM because it's POSIX.
So you can use it on every of them. And then you need the Vostuser block device, which is,
as we mentioned, the control part is still intercepted by Q-emo. So we need this small device in,
in the DMM. And then we have the last one is the Unix part, is the Unix domain socket,
where the device is exposed by the Q-emo storage demo that we start before.
Okay, we talked about Q-emo, now let's talk about the device. As I mentioned, some of them are in Q-emo,
but we have also a lot of them outside of Q-emo, in this case written in Rust. And most of them are
based on the Rust VMM components. Trust VMM is a community which provides the building blocks
for building VMM and hypervisor. Some of hypervisor that use our components are, for example,
cloud vapor visor, firecracker, lead k-run, and all of them use our components. And we are
collaborating upstream in order to create this building blocks. And I put some links to the community
channels. And of course, what we support also Vhosts. Both for the VMM, but also for the devices.
So we have a couple of crates, Vhosts, which is a pearl library that exposed the
message structure and that kind of things for Vhost, Vhost user and VDPA. And then we have a
Vhost user backend crates which essentially allow you to easily create a Vhost user device application.
Other than that, we also support, we also have a lot of devices implemented and maintained by the
Rust VMM community into the Rust VMM Vhost device repo on GitHub. You can find that. And we have a lot
of device supported, like can console, scazi, visor, sound, and some of them working progress,
like GPU and video. We also have other devices outside of Rust VMM, but they are
based on our our crates, like Vhost and Vhost user backend. Can we run Rust VMM crates on
POSIC? We still need to do a lot of work there. Because most of our crates are based on the VMM
CCUTIL crate, which is supported, Linux is supported, of course. We in this partially, but we use a
lot of Linux specific stuff, like Apple, eventfD. So we need to replace them, like for example,
for Apple, we can use the small polling crates. And for eventfD, maybe we can implement some
automatically fallback, like QEmo, does with Python by 2. I put also here some open issue,
because the community is interested in supporting other operating system, like macOS.
And so which are the next steps? As I mentioned, first of all, in QEmo, we need to understand why
that test is failing. And then we have a lot of things to do on Rust VMM, to improve POSIC
support in the VMM CCUTIL. Upgrade, we need to replace the Linux SQL as I mentioned, and for every
device, maybe we need to add up a bit the code, for example, for BERTIFS, the file system, of course,
is different on macOS, for example. And so we need to some specific work on it. So if you are interested,
if you're interested, if you're to reach me, yeah, we have a lot of things to do. It's a side
project for me, so I don't have time, but yeah. Thank you. Any question?
Sorry, the question was, how many people use QEmo on FreeBSC? Something like that, or yeah.
Yeah. On FreeBSC, honestly, I don't know. On macOS, most of them are using, for example,
for Podman, in order to start Linux VM to use containers. And since QEmo does not provide
some devices like BERTIFS, which is a cool one, is not implemented into QEmo, is implemented
only as a BIOS user device. This could be cool to support. In order to use that device, also on
macOS to share directory between the host and the VM that runs the container.
Essentially, QEmo is really widely used in order to do binary translation. So if you want to
emulate ARM on XETC6, and then if you want to use a BIOS user device, it could be nice to have.
Okay. Thank you.