So, hello everybody, my name is Marty Nechas, I work at Red Hat, and I work there as a technical
lead team lead on a maintainer on the project called forklift.
It is an open source project for migrating VMs from various places to Kubernetes Cubevert.
And I think most of you here are familiar with the VMR situation, but for those who are
not.
Okay, this is what happened.
We are also acquired by Broadcom, it's not that well visible.
So, I love this image, and immediately after that, we started getting some articles.
This is from Red Hat, from Register, and from Business Insider.
There are many more, and I think some of you who are also from what I heard from statistics
were also influenced by this.
So, it feels a bit like being held for ransom.
So, how to escape this ransom?
It could answer a Ruby 2V, which I think many of you know, and for forklift.
And that's it.
Thank you.
Okay, so, let's go to the details.
So, Ruby 2V, really known tool, been developed by Richard Jones for many, many years,
focusing lightly on the VMR to for migration to Cube.
Why Ruby 2V, why we will one with some choices, main thing is gas conversion.
The VMs running on VMR have the their own drivers, their own devices, and we want
to use the proper Cube materials and virtual drivers.
So, what Ruby 2V is doing, it removes the VMR tools, it insults the virtual drivers,
insults the gas agent, and does a lot of editing inside a VM, and configurations.
It even rebels in a trauma phase, and many more, I'm not a perfect guy for this.
The Richard Jones would be perfect for it.
I'm just kind of user of Ruby 2V.
How does the Ruby 2V work?
Inside it there is a tool called NVDKit.
I highly recommend this possum talk from Richard a few years back, the NVDKit allows you
to attach devices, remote devices to your machine as they would be in your local laptop.
It allows you to do many things to it, the protocol itself and NVD is straightforward.
It allows you to read rights, I have a quick demo, which highly depends on the Wi-Fi,
so we'll see.
For example, I got here the Fedora RhoxE file and using NVDKit, I can, oh, that's not that well visible.
I apply the curl plug-in to the image itself, and the file is exe compressed, so what it does,
it uncompresses all the blocks, so it doesn't download the file and open it.
It reads only the blocks which it needs, and then it applies a co-filter on top of it,
so all rights will be done to the local machine, and I can do whatever I want with it.
I can, let me just copy past it, I can see the partitions, that Wi-Fi, I can mount it,
and now I will not have the TMP, please, I have TMP anyway, yeah, I can mount it and do whatever
with it Wi-Fi is slow, so let's move on, so how does the V3 to be worked?
This was the NVDKit, the V3 to V, we have the VMDK on the VMware side, this format through which
we connect to the NVDKit. VMware has the library called PDDK, which are to manipulate their
disks, and NVDKit has a plug-in for it, which allows the NVDKit to do the same things as I was
trying to show with the curl. On top of the NVDKit, there is started another tool called
Guestafest, which attaches the NVDKit to VM, and inside it it's running a demon,
the LDR allows you to manipulate disks securely and quite stable, and it runs a lot of the conversion
scripts inside it, so we have the VMDK, and then all the changes, what we are trying to do
are written to the cache. So first, we do the conversion, we fail fast, and we see what goes wrong
with the conversion itself. It also allows us to use a tool like a pest stream, so we throw away
the unnecessary files, unnecessary blocks, and they are written to like zeros to the cow cache,
and we don't need to transfer it from the VMDK, the blocks themselves, so we even improve the migration
time. So then we do the NVDK first reading from the cache, and then the remote blocks from the
VMDK, and we got the destination. Advantage is, it fails fast, it's really good at it, if something
goes wrong, it supports many operating systems, various really old Windows versions, the third
depends on the virtual drivers, which are available, it supports Ubuntu, Santos, Fedora, many, many more,
and it manages the disk transfers for you. So it manages the disk transfers, so all you need to do
is run V2V, and you have it. You don't need to do anything fancy with it. disadvantage, it has
high downtime, because the VM needs to be turned off throughout the whole process. Otherwise,
you would get disk eruptions. And disadvantage, it manages the disk transfer. So when for
cliff game, we wanted to do some tricks around it, and we needed to get rid of it, that those
parts. Okay, so that was weird to be. What about for cliff? For cliff is tool around Cuba,
and it focuses on migrating from VMware, open site, over to the Cuba itself. So the whole migration
processes, you have configured VMware infrastructure on on your source, and you need to first
configure the Kubernetes itself. We have some projects and some POCs to do it for you, but nothing
concrete yet. So the administrators need to go and create a storage process, and they need to
create a networking. The system administrators know them the best, so we are relying on them.
Then the user needs to add a provider to the forklift, immediately scans the infrastructure,
there are lists of all the VMs, and sees the which networks are used for the VMs, and with which
storage data sources are used. Then we have internal validations. We tell users which features
are available, which are not if the VM migration will fail. For example, we are not supporting
right now are the VMs. So we let users immediately know. Next, user needs to create the
network and storage. So we have VM with some specific storage, and they need to tell us
from which data store needs to be migrated to which storage class, and the same goes for networks.
We have done two migration types called and warm. The cold one is the V2E flow. It
shut down the VM migrates it, and then boots it up. The high downtime are the same as V2E. It's
actually using V2E like under the hood, and then starting in migrations. But more interestingly
is the warm migration. So we have again the VMDK on VMware side. We have removed the NVDK
from V2V to separate project, called a keyword container data importer, CDI, and which manages
the transfer. This allows us to use additional VM bar features such as change block tracking.
So we create a snapshot on top of the VM, and we migrate underlying this.
So for example, the blue one could be 500 gigs. We transfer it, and then we would be transferring
only the changes, which would be happening. So the orange one can be five gigabytes. And then
we do another one. And the red one can be even less, because the migration time of five gigabytes
it will be much less than migration time of 500 gigs. And we do this periodically, until the
user sets the cut-over time. And that point we will shut down the VM and do the conversion
using V3E to be in place on the already finished disk. So there is still downtime. It's not
life migration, but it's warm, so something in between. Advantage is low downtime. Much
lower than the called migrations. It removes the V3E to be disk transfer. And this allows us
additional features. V3E to be will not continue the migration if the gas conversion fails.
That's how it was created. And if the users want to try using emulation instead of
drivers and proper tools, they can still try it. We have this working progress, but it allows
it to us. Additionally, it allows us to do a short transfer share disk. V2E looks at all disks
attached to the VM and transfers all of them. Here we can select the disks independently. And
we are also starting working on a floating, that we would copy the disks within the storage
arrays. Right now and be the kit everything goes over the network. And it can take some time.
This advantage is it fails the slow and requires a digital tracking on the VM. And it can take
a bit longer because there are more steps we create snapshots, we delete them and do everything
what's needed. And I have a quick demo. Yes, Wi-Fi working. So I do create this. This is in the
open chip. It can be also in the OKD UI. I create a vis-villier provider.
Credentials. And now it's got all the information from the VMR. I see all my VMs. I see
the validations which are on top of it. There is warning that it's running.
I create migration plan with the source provider.
I choose the storage mappings. I need to name it.
I can choose which destination the VM should be migrated to. So it can be isolated for
several administrators. We can choose wherever we go. There are a lot of other configurations and
settings. I enable the word migration for changebook tracking and starting the plan.
We do create this on the cluster itself to which we will start the migration to.
I'll go to the VM. The migration is happening on the background. I write some simple file
to it. As a test, it will be also on the destination.
And then I hit a couple of times. I set it immediately.
I can see the process progress of this transfers.
Then we will run the conversion and at the end we create a VM. If the VM was turned off,
we will not turn it on. If it was running, we turn it on again. We are trying to keep it as
persistent as possible. Now I see a Cuba that the VM is running. I'll look into it again.
I have my demo file. How is it with large scales? We have tested it with many hundreds of
VMs, huge disks, and even a high-eye of databases. Those are a bit tricky. We got it in the end
and now it works without a problem. It works quite well. We are working on the mapping of the
networks and trying to add additional operating systems. The storage of loading that we would
do the copying not over the network, but within the storage areas or some improvement to the speed
transfers. Maybe additional source providers, we've been asked also to add something like AWS
and other hyperscalers, but right now we are not working on them. But feel free to go so and try it
yourself. We have it on GitHub. I really made it to do it a bit better. That's it. Thank you very
everybody.
Yes, please.
Question was if it's already available in open shift, it is available in open shift already.
We have an operator hub that you can install it as operator and which maintains and installs everything for you.
Yes, please.
What are the common problems during the migration?
What are the common problems for the during the migrations?
Most often it's what we've been hitting various operating systems and various configurations.
There is no standardization.
Any administrator can do whatever they want with VM.
We are getting a lot of strange configurations which we are creating custom scripts.
Also, problems are the devices themselves that we need to,
because between VMware and Cumo, there are so many differences.
So we need to inject some new dev rules to keep some, for example, names to be persistent during migrations.
Yes, in the back.
So we know, we know, yes.
Can I install the virtual virtual virtual environment?
Why the machine is on VMware?
Not reboot.
And then move them to VMware by sharing them now.
And then whatever you come, go ahead and do virtual browsers.
So the question was if we can install the virtual drivers first on the VMware side,
and then shut it down and then transfer it.
You can do it.
It would be much better for us.
For example, if the conversion will would fail,
you could do it.
But we are trying to do it this for the users themselves automatically.
But yeah.
Because isn't it good if you can just keep the VMware drivers just in case,
but what is the further problem we're moving it?
So we are doing the,
if we need to remove the VMware tools and VMware drivers,
we are doing coping.
So we are not touching the source VM.
The source VM is still staying there.
If you want to,
if the migration process fails or something happens,
or you decide it's on working for me,
the VM is still there until you delete it.
Yes?
Yes.
I would love to explore it.
Stay on and touch my VM from two of the errors in the letter of the system storage.
Can you tell or,
it's very nice that I'm going to want to come back to the end?
I quite a question was about exporting from Cuba itself.
Yeah.
Could you be asking for data?
So I am not sure if Cuba has something like that right now,
about exporting VMs from Cuba,
but we do a,
the work that allows an important of OVAs.
Yeah.
Because I experimented with the,
the OCI and the work there,
I couldn't find, you know, the opposite of that.
And I was wondering if you're looking for something or anything.
Just,
I thought that you see that sometimes the,
the VM image,
and then talking from there.
It's a pity.
Exactly.
Yeah.
And then you can use it because it's clear.
Yeah.
And then you can use it because it's clear VM.
Yeah.
And then you can use it because it's clear VM.
Yeah.
What kind of problem did you,
with a really high workload system,
like the database system,
you,
you have a,
like,
as a limitation?
The question was,
what problems we encountered with a,
like,
the data basis?
With a very high workload system,
like the data basis?
With high IOPS.
So we had some problems with,
as CDI,
that it wasn't querying correctly,
the changes,
using the VM versus changing bug tracking,
system,
and we fixed it a few months ago.
And,
yeah.
So it returns,
not all the changes at once.
We needed to do some additional queries for it.
But that was just technical problem.
Yes?
Well, it's slightly off talking.
But if,
broken,
or VM,
while say,
18 months earlier,
do you think red hat would still drop,
so both are over?
I have worked on over it.
And I,
I'm not the right person to answer this.
And,
oh,
there were in the world that they wouldn't revive it or not.
Yeah.
At the end of the day,
it seems like Google is the way to go.
They have,
just for the recording,
the question was,
if the broadcast would announce
our acquired VM earlier,
if red hat would kept over it.
Yeah,
you could hear the answers.
Yes?
So,
if we have tried it on other Kubernetes distributions,
like Rancher,
personally,
I have not.
Sorry.
If it would work.
Yeah.
We are not using anything open to specific.
I think it would work.
Well,
if not,
please open an issue on GitHub.
Yes?
If not,
please open an issue on GitHub.
Yes?
Do you have any questions for migration?
Exactly.
Why is that?
If you see the question from Google,
Google,
can I say,
Pakistan trust with Google,
only after Google based?
The question was,
if we can order or group the VMs throughout the migration,
that is feature,
which will actually cost us the lot.
We've been thinking about it,
and not right now,
but what we can do is create a separate plans.
You can do a group of VMs together,
within the plan itself,
and then you can create another plan for additional VMs.
Cool.
Anybody else?
Cool.
In that case,
thank you everybody.
Thank you.