So, I'm Magnus. I'm working in Azure Core on the upstream team. In the upstream team,
we've worked mostly on container-related open source technologies like Kubernetes, OCI, etc.
But during the last year, we also worked on Qemu. I want to talk a bit about the work that
we did in this project. There's an audio issue.
I'll just try to increase the volume.
I can also try to speak louder, but that's fine. One, two, three. One, two, three.
It's better. Okay. So, yeah. First, maybe it makes sense to talk a bit about what Qemu
accelerators are. So, this obviously simplification, but I think it still makes sense to
dump down things a bit to provide context on what this feature is about. So, an accelerator is
essentially the term in Qemu that describes the interface of handing off work to the
virtualization extensions in the CPU. And we've seen this chart like there's, if you look at the VM,
it's not really easy to describe what the VM actually is, but we can maybe use as a kind of working
definition. There's a bunch of CPUs like this indicated here that there's multiple of those entities
and there's memory. And from the host, we map memory into the guest. It's kind of, we projected
it as guest memory, we contributed to the guest. And we also assign certain holes to it. And
the holes are then caught by the high provided by the CPU as a memory access traps. And then the VM
can, to device emulation, for example. So, there's more variants to, to, to, I owe, but this is
maybe the most common one. And you see it's like the actual work in the VMM is actually
this cycle. So, you, you pass off work to the CPU, then it's handed off to the high provided,
this patches it, then to the actual CPU and under certain circumstances, then we'll end up
all things a lot. We end up getting a trap in the guest that the guests itself cannot handle
and we can yield back to the VMM and the cycle continues. Also relevant is maybe a quick discussion
about what the difference are between KVM and Microsoft Hypervisor. So, let me call MSHV. They
are both type one hypervisor, so they run on bare metal. But, why KVM is kind of a part of the kernel.
And embedded in the host kernel, MSHV actually is a very thin layer. So, I don't work on this part,
but people tell me it's really very thin. And the host OS, what we call it, it's really just
a good partition. So, it's a privileged guest that's the main difference here. And when you
look at this topology of how guests and, and hypervisors are organized, then there's also
the option to run the root partition on the Microsoft Hypervisor. And they like in practice
this looks quite like KVM for VMM. So, there's an MSHV device that's controlled with
IOCTLs and file descriptors. And this has been used for quite some time already in an
KS product called Potcent Boxing, where Qatar is used together with Cloud Hypervisor on
Azure Linux kernel. So, it wasn't part of the mainline Linux kernel yet. It was developed
on a separate Azure Linux tree for quite some time. And it's using the Rust VMM integration
that we just learned about. There is the option to run this on bare metal. There's the
option to run it with a Nestetypervisor or direct virtualization, which I described in a
later slide. But essentially, I think the only part that will be really relevant for the
user for the if you want to have a pure open source stack is probably direct virtualization
because for the Nestetypervisor, you would run Microsoft Hypervisor, I was part of the
UV protocol or something. The first part of this have been upstream since
2015 and incrementally more features are being added. The rational behind QEmo Linux and
MSHV is essentially that there's a lot of us from customers to sandbox and trusted code.
I mean, it was obvious for functions as a service. For example, it becomes a lot more
critical when people actually start using those coding agents and they would create
a lot more on your system. So there's just a lot more untrusted code that we are in
those circumstances. And MSHV provides us with the Microsoft term. It's called like
hostile multi-tenancy. Basically means you cannot assume anything about the neighbor or
the tenant being not doing something malicious. So this really a hard isolation layer
that is essentially also used to run Azure virtualization products. So customers
will want to build those platforms, virtualization and Linux is a very popular.
Most popular workload on Azure when you look at the course and QEmo is just the most popular
and comprehensive VMM with a lot of device patterns etc. So customers not only want to run cloud
native workloads, but maybe also more exotic ones. And eventually QEmo is just all the building
block for higher level virtualizations like Libert and QEmo. So if you want to use your
custom hypervisor, you would have to build integrations for those. So this is why we ended up
saying we need to have an integration in QEmo. And what I mentioned earlier is there's
one option that this kind of virtualization technology gives us with Microsoft type advisor
that we can have a direct virtualization. I think the marketing term that was coined
before it was called hierarchical virtualization, which is bit opposite, but it just depends on
where you look from the angle you look at it. Because for the customer it's direct virtualization
from the virtualization perspective on the technical stack it's hierarchical because what we
think is the basic idea is that if you have a root partition, then you don't need to actually
use nested virtualization like this. You can instead just have a logical guest that is actually
a sibling. So you inherit the resources and you're able as a customer to launch
virtual machines in your virtual machine, but they're not nested. You don't need a hypervisor
stack. They actually just use resources from your allocated resources and create a sibling.
And this enables certain use cases. For example, device assignment that is pretty hard
before, so you can build virtualization solutions that have device assignments to your VM
and then you can assign it further to your virtual ed2 guest. And it's also quite useful to
avoid this kind of words, which is that you have to do if you have layers of hypervisors
then you have to like the cycle that I showed in an earlier slide. This gets quite expensive
if you have to do this kind of round trips over and over through multiple layers. So there's
a perfect text that you pay when you do nested virtualization and this one or the goal is
basically to avoid this. So this has been announced for some months now and it should
enter public preview soon. There's like several integration points in Qimo that that we
kind of follow the KBM. Example, or try to model, I think the MSH need drivers
also written in a way that it's quite easy for VMMs to use the same facilities as KBM.
So a lot of aspects are, I wouldn't say, trivial to it to port but it was like easy to
identify where the facilities should be implemented. Like for example, we have the requirement
in MSHV that we have to do MMIO instruction, decoding an instruction emulation. Because
in the earlier slide I showed that if you trap into the VMM and you need to emulate for example
and MMIO access, then you have to have run those instructions in software essentially. And
colleague basically looked at Qimo and saw that HVF, like the Apple hypervisor framework already
has something like this in place. So we were able to generalize this concept and extend it a bit
to what we need to do. And this worked quite nicely and I don't think it was very intrusive. We were
also reusing all the acceleration parts. So this is basically the cycle that we've seen earlier.
This is for real workloads not super practical because you want to avoid essentially
to enter the VMM or to avoid the M exits and to avoid context switches etc. And so there's
a lot of optimization parts that have developed in KBM and Qimo to move the data plane
outside of the VMM. Like we see here, the like one typical chain of events. So we see like the
guest driver for virtual IO driver for example writes to some NIO address. The CPU will
detect as a page table violation so it cannot map this. We have a VM exit and then the
kernel can look is there maybe an event FD registered. And if that's a case, then we can without
the VM exit, we can essentially trigger in a different threat, in a dedicated threat, we can
trigger the processing of this event in a device backend and when the device backend is done,
it can do the opposite, it can write to an iqfd that is registered again in the host kernel and then
it can inject an interrupt that then ends up in the guest interrupt service routine and complete
operation for example. So in this picture, the VMM is not involved anymore and this speeds up
things tremendously. Also for VIFIO, I think we are mostly able to use existing KBM abstractions
and kind of call them Excel accelerated iqfd. So we have to probably do some renaming there,
but essentially it work pretty well. So I have a few minutes still to talk a bit about the
challenges that we're facing. So one of those challenges is VIFIO, so there's existing VIFIO
support in cloud hypervisor and it kind of it follows the same principle you essentially have to
set up a bridge between the VIFIO device and iqfd chip via event of deal and then the interrupt
schedule routed from the past through device to the guest properly. The problem that when we try to
hook this up facing well that iqfd were dropped and it wasn't really working until the devices
reset itself and then eventually after some debugging turned out that the sequence at the MSHV
driver expects is just very subtly different than of different from what KBM does. So just the order
of which component to your create first and how do you wire them up. The end result is the same.
So we have to see how we address this. I think we have options for this kind of problem we could
have just a specific MSHV pass which is maybe not desirable. You could adjust the driver behavior
maybe I think I'm not super sure this works because KBM is maybe a bit less of a distributed
system than then MSHV is because we have still this hypervisor outside the kernel. We could also
maybe ask whether the QE would behave or could be adjusts. So I just experiment around and move the
sequence around and things work for KBM and MSHV. So this is something that we have to address.
Another challenge is migration or even life migration. So this is not initially obvious when you
develop the MSHV because you make imperative commands to an API and the MSHV created and a lot
of magic happens in the background. When you want to take a VM or basically create a safe state
of a VM and kind of reproduce it somewhere else then you start to notice that there's a lot of
implicit state even in X86. Guests and why you initially don't really have to care about a lot of
them like I know ABX 500 12 registers etc. So those kind of things don't play a role for
example and then I owe emulation. But they play a role definitely when you want to migrate
like the kernel with panic. For example, the PU registers are not really in the format
they were before the context which and we need to then figure out how we
capture serialized and round trip this whole state. So this has been quite challenging because
also Kimo has not just the block serialization format we actually want to instruct for example
the PU registers that are stored in X-Safe or MSRs and an additional problem with
parabotularized devices like synthetic interrupt controller that is used on Microsoft Hypervisor
and also by Linux guests. So there's a long-tailed list of subtle hard to reproduce issues and I think
it will be hard to tell when you're done with this so the challenge really is here to define
like a bar that you meet and then you can advance. This is like one of the challenges we have at
the moment and quickly just what's upcoming what we didn't have for a while was like proper
CPU model support so in Kimo you can tweak the CPU features extensively and this wasn't really
properly reflected in the guests so far so this is something that we fixed device pass through
we'll come the life migration and also colleagues started to work on arm and I think
that would be it from my side.
Questions?
You're quickly compared to K-VM in the beginning and yes and another excellent and
you call VHPX I think. How does that one compared to the source of Windows?
Yeah so that's the question about VHPX this is it's using it's more like HVFI would say because
it's using a high level API to create VMs under Windows so this one is MSHV is a Linux
driver so that the root partition is on Linux this is the difference it uses like VHPX I think
can't do live migration and certain other aspects that you that you can do in MSHV it's a bit more
high level.
Not so far because yeah like like the VHPX is essentially like a Windows technology and the
host in quotes is Ninox and the workloads we are expecting is mostly a virtual IO and
like guests Windows guests for example would install virtual IO drivers but yeah maybe in the
future this would be extended so there's one more question.
Yeah so for the hard to reproduce bugs whether I come to the snap shutting machine I mean
what I have this on a slide quickly so what I ended up with essentially writing also my
only bugger workloads which ignore like there's no assets like bare metal x86
program that start very simple and then you start adding features MSR etc and then you see
what breaks and this has been like very useful so I've been using one rust twill library or at that twill
which you can create this stuff like serial console programs on bare metal easily it's
was one of the more fun aspects of this work okay I don't see any other question
then I think we're good thanks a lot for attending
you