All right, so our next talk is going to be breaking that architecture barriers, running
8x86 apps and games on ARM by Tommy Vaseca, I hope they're not saying it right.
Thank you.
Bonjour, put the tools and welcome to my talk.
We don't have a lot of time, so I was just going to bore you really quickly about
who I am.
I come from a couple of different corners.
One of the first, oh, quick start again.
There we go.
One of the first things in the community that I've been doing is working on video game
console and my data.
So if you've ever been trying to work on play, game queue, we, or previous games,
or PSB games on a computer, probably some of my work was involved there.
I really enjoyed just the feeling of being able to play video games on systems that
there were never designed for.
Whenever I first saw this kind of thing happened in practice,
it was always mind blowing to me and I always wanted to understand how these things work,
and so I just still went to the screen.
But professionally, for the longest time, it looked like I was doing other things.
It was actually studying physics, up to my master's degree.
Good stuff to be had there.
Turns out more pictures.
So some cute pictures come out of there.
Some cool masks, like I really liked the theoretical stuff, really good challenges,
but at the end of the day, it didn't really work out for me.
So I ended up working on graphics drivers,
so making actual hardware work for rendering video games,
specifically in terms of Vulcan drivers.
So if you have used an imagination, a GPU or a MD GPU on Linux,
chances are you were using some of my work as well.
But today we're going to talk about a different fun challenge and that is facts.
And I heard there's a couple of people who like trains here, so I brought this little picture.
There we go.
The airport express from Berlin to the airport,
but sadly it's not what we're going to talk about today.
Instead it's going to be about boring emulator stuff.
Sorry.
So actually a little bit about facts itself.
So the project was started about seven years ago by Ryan Hoadeck.
I think what you pronounce this name.
Seven years ago, I personally joined the project four years ago,
so it's been a while as well.
We've had our ups and downs.
It's been a fun ride, but we're really getting somewhere nowadays.
So yeah, what's actually involved in making x86 emulation working?
So the task that we have is we have a whole software library,
like from office software to other proprietary programs,
but in particular games, like people want to have games on their hardware.
Always sounds like a fun little thing, but at the end of the day,
this is what keeps people away from switching from Windows or Mac OS to Linux.
It's that they can't play their games on their machines that they could use for other software.
So we want to have a solution for that, and that's why we want to have a translation layer
that that's existing software automatically for armed devices.
And yeah, at the lowest level, what you have here is the CPU.
And you can sort of build a layer diagram out of this.
If my slides are cooperating, yes, thank you.
On top of the CPU hardware, that's it's the operating system kernel,
which takes care of all sorts of things from memory management,
threading, process management, making sure, for example,
that your video game that is running cannot read your browser passwords at the same time.
That would be kind of undesirable.
On top of the Linux kernel, lives a set of libraries from your standard language long,
run times like the C run time, the Java, the JBM,
the top net runtime, for example.
Next to that we have graphics APIs to make things actually come up on screen.
And of course, we have the actual game engines like Unity, Unreal, you name it.
And at the top of all of these, you have the actual game,
like the Kit and Adventure, for example.
And we're looking at the stack, and we're tasked with the problem,
every single element of this needs some sort of equivalent
in our translation layer.
But let's start, let's start at the base.
We're just replacing our physical x86 CPU with an ARM CPU.
Well, the standard approach to what we can do is binary real compiler.
So we take all of the instructions of our program and map them from x86 to ARM.
So it's not simple, but let's actually look at what that looks like in detail.
So if we look at some example x86 code, this is just the...
Where's my mouse?
This is just a simple title example.
I was some basic instructions that I came up with.
The details don't really matter all that much, but conceptually what we're doing here
is we're iterating through all of these instructions.
One by one, and we're meeting corresponding...
corresponding.
I'm going on side.
There we go.
So we're using an intermediate representation.
Because we don't, like ultimately we want to have ARM code,
but for reasons that are kind of technical, it's more convenient for us to actually generate
a lot of garbage in terms of an intermediate language that represents this semantics that we want.
It blows up a little, but this allows us to do sort of optimization paths on top of the whole thing.
So the garbage that we generate from every single one of these instructions
gets compressed down to something more legible, legible.
I'm trying to find my mouse point, but that was actually...
My mouse point, but I...
Cool.
The next thing.
So what happens after we kind of blow up the original X86 code into our intermediate representation,
and we then optimize the down.
That's the point where we can turn the remaining instructions back into ARM instructions.
And that is in principle code that is runable on any machine.
Now, small problem here, it's not always that easy.
So if you look at a slightly more involved example, there we go.
So just a simple loop example in this case.
You can see the ARM code on the right side is actually blowing up quite significantly,
so it doesn't even fit on the slide anymore.
So there's a couple of reasons for that.
Some of it is just in here and translation overhead.
So every time we enter our exit one of these blocks,
it's just the thing that happens.
We need to emit some boilerplate code just to deal with the fact that we're not the original code.
We're just set a studio sandbox and we need to enter and exit that sandbox every now and then.
And that's in particular where all of the garbage at the very bottom comes from.
But also if you look at something like the this decrement instruction,
which just takes the value that is given on its right,
and it subtracts one from it.
Where we come in and loops, for example.
This is mapped to, I think, this.
Yeah, these three instructions here.
So you can see that the actual decrement operation is mapped to your subtraction,
which seems intuitive.
But then you get a couple of garbage around that,
in form of the C set and RMI f instructions.
And this is due to a thing called flag handling.
So every time an x86 arithmetic instruction, like an additional multiplication runs,
it could be that the result doesn't fit into a 32 bit or 64 bit integer.
So there's overflow flags or negative flags,
underflow flags, all of these kind of things.
They are computed unconditionally by your x86 CPU to inform the program
that's something ran wrong with the computation.
And an x86 happens all the time, whereas on arm, it's optional.
The program or the compiler explicitly has to opt in.
And this difference is something we need to compensate for.
So we get these extra instructions on the right side.
And our optimization passes really help with reducing this to the bare minimum necessary.
Normally it would be like 10 or 20 instructions
of overhead just to compute all of these flags.
But thanks to our optimization passes, we can get rid of this.
That's mostly.
Yeah, one thing I should probably also add is,
there's a ton of these x86 instructions depending on how you count.
It could be like up to 6,000.
It's definitely more of an upper bone.
But imagine you have to go through the process of identifying 6,000 functions,
decoding them, re-implementing them in arm,
and then somehow making it all fast as well.
That's a great challenge in task.
One other challenge in task is the x86 memory model.
And they're not much time to go over this.
But imagine you have a very simple program with two threads running in parallel.
And what the first thread is doing is really simple.
It's just resetting some state.
And then it's waiting for a result to be computed on the second thread.
The second thread now might just say, okay, let's compute 42.
It takes a while.
But once it's done, it assigns the result to our result variable.
And then it sets it done flag to one to wake up the other thread on the left.
The other thread we're done for the result,
and now evenly you might expect the result to be 42.
Since that's what we've computed on the other thread.
On x86, you would be perfectly correctal about that.
But on arm, maybe not.
So the problem here is that the x86 memory model has an implicit guarantee here.
That stores also variable rights on that we trigger on thread 2.
Okay, thread 2.
You know what I mean?
The most because it doesn't work properly.
So actually it's computing, it's writing the variables in the order result.
And then done.
But on arm, we actually don't have the guarantee that the first thread is observing the rights in the same order.
thread 1 might actually on an arm platform see the done flag being set first.
And then the result flag.
And that is problematic because now we might have a race condition.
We might exit out of the slew immediately before the result has been written and we'll try to print it.
So in that case we would still print the original initializer up here.
And we would see zero.
So this mismatch in terms of the semantics of the memory model is really difficult to resolve.
Because we don't know for which memory access is important.
And so basically we have to patch it up somehow for every single memory access that we have in the program.
And there's a couple of strategies we can employ to do this.
The root force solution is to just make every single memory access atomic.
It's very expensive.
Instead we try to use euristics wherever possible.
So actually one other thing we can do in terms of atomic is we can use something called half barriers.
This is a trick that I think we took from the mono runtime.
That makes the overhead a little bit better, but still if you enable us for everything you still get a performance it of something like 20% or least.
So this is really something we can't afford.
So we try to employ additional euristics on top of that with a really fun hack for a unity game specifically where unity you.
I think it's mono as well.
It uses some sort of ring buffer internally to store something.
So it relies really strongly on these memory model semantics.
And we can very reliably pattern match on the exact instruction set used by the cyclic buffer though.
So we have something like a relative note from a register with an offset of 61 bytes.
It's something like that. So very specific instruction pattern.
We know more or less reliably that this will match the cyclic buffer.
So we enable these strong memory semantics only for this code.
And that makes unity games work for everything else.
All the other memory accesses for most games we can just disable this accurate memory model simulation and we're just working.
So it's a really smart combination between a reasonably fast default implementation.
So smart euristics and some other fun tricks that you can quite at the time to talk about though.
And yeah, so this is just the last thing is just the audio we might observe on the arm program.
So that's taking care of the CPU simulation.
The next thing that we wanted to show talk about is the kernel layer.
So the kernel has a couple of subsystems. As I was saying, the threads and like pose signals for example is one element for file system.
We have system calls and we have the GPU driver.
And what effects needs to do for these is to build a sort of compatibility layer.
So every time a user space program tries to, for example, access files or talk to other threads.
Effects sort of acts as a pseudo runtime between the game and our arm 64 Linux kernel.
Just to make sure that what the game sees underneath itself actually looks like an X86 kernel.
So you just need to have this building block in between that wraps all of the functionalities like this.
What does this look like? Well, the game at some point will have a code block that uses a system call instruction, just like this.
Yeah, like the system points actually here.
This just was pre-ghost kernel functionality. We run this thing through our X86 reconpiler to get a equivalent arm code.
But in place of the system call instruction, we place a call to an external C++ function.
Which then looks fairly normal. It's just a switch on the system call ID and then triggers accordingly other code actions.
So the whole point here being is that we get from a very low leather perspective of this raw binary machine instructions.
We try to build abstractions a lot of the top of this so that we can actually focus on implementing the system call logic using plain C++ logic.
See, plain C++ code instead of having to dive in these low level assembly details.
Yes, if you can want to pause the slides online, you can look at the details here, but the gist of it is this is often quite straightforward to do.
But again, there's hundreds of these system calls, so it's a lot of work.
And most of them are straightforward, but some of them tend to be quite tricky as well.
Particularly when it comes to memory management, for example, where effects also needs to allocate its own memory and then needs to make sure there's no conflicts there.
So that's the kind of compatibility layer, and now the big question becomes what's about the last layer we had in our diagram, the actual libraries.
The C runtime, the unity engine, or graphics API.
Well, it turns out we don't actually need to do anything special for these anymore, because if we have a reasonably accurate binary reconpiler and a reasonably accurate kernel,
then all of the libraries above will just magically work as a part of complex interacting system with each other.
But as far as effects are concerned, the libraries together with the game just form one huge blob that interacts with the kernel, and that's all we need to worry about.
So we get this nice separation layer here in the middle.
Where we are running unmodified X86 on software on our new ARM hardware, that is made to look so to speak like X86 hardware using effects as a compatibility layer.
There is however one more trick we can apply to this all thing, or rather two tricks even, because if we look at the graphics, the graphics API like Vulkan or OpenGL, this is actually a huge bottleneck of emulation sometimes,
because games are accessing these APIs so often.
And now you have a well-defined API here.
We know the game is talking to an X86 library, but can't we just take the existing ARM Vulkan implementation instead?
And indeed, that's what we can do with a couple of tricks.
Maybe. There we go.
So whenever the game tries to access this book in X86 library, we just inject our own smaller replacement wrapper library on top of that.
And what this replacement wrapper library we do, it doesn't do a lot, but it just forwards all of the different individual API calls to the corresponding ARM library.
And this has two advantages. First of all, we don't need to emulate the original X86 library, so we save some emulation overhead.
But then also we completely interesting.
More interesting. Oh yeah, okay, great.
So we saved translation overhead, and oh yeah.
So I just leave it like this.
We saved translation overhead, and what also happens is that we can cite that the wiggling kernel compatibility layer.
Because it's clearly just making troubles anywhere.
So we get these two APIs. Another thing that we actually can do in addition, though, is to save further compilation overhead, we can employ a code cache.
So on the second and third, so in subsequent runs of the game, we can just reuse the compilation result from prior runs.
So that, for example, when I'm entering a new scene in a game that I played before, that I don't get a micro study every time.
I enter that area, but rather we can load the code that is already compiled from disk.
And so we can save a couple of a bunch of compilation overhead like this.
So with that being said, this was more or less the entirety of effects.
I mean, obviously the details always look different at the end of the day.
And I think my slides crashed. Interesting.
So at this point, I would just reload the slides and I prepared a little demo, which is very convincing now.
So do you want a demo, actually?
Yeah, I'd like to thank you.
So the demo is actually something, and I really wish the slides are working now.
Oh well, so the demo is already breaking, because you're looking at the demo.
And we're seeing the demo effect in action.
I think my machine is just running out of RAM, that's a problem probably.
So these slides actually weren't using a react-based slide presentation framework.
And it's running in an X86 build a Firefox.
We're just quite impressive when it works.
Thank you.
And now it's gone.
It should not even kill it anymore.
Excellent.
Well, I've prepared a little other thing for you though.
Hello.
So we can actually run steam in this thing as well.
Okay, sure.
So to say, sometimes I already launch steam, but this is just a regular steam installation started using effects.
And you can use it just like any other steam installation and launch games using it, where exactly where we get there.
Ah, there's audio, not on stream I think, but maybe people in the room can hear it.
So it should have come to Brussels if you can't hear the audio.
And it's really quickly wanted to show that this is actually usable, not just for joy to do games, but actually for real 3D games as well.
Some people even run on more powerful machines on this than this.
Like things like cyberpunk, for example, so even triple A games can work to some extent.
Also, I think the game might be crashing as well.
So now it's working, yay!
Yeah, working just as you were expecting.
So we have five minutes left, so I'm just going to try to fix my browser.
Are you going to take questions?
Yeah, the problem is I can't actually kill the existing Firefox instance here.
So it can't restart.
There we go.
That was annoying.
Okay.
We've got slides again.
So you can also see the funny picture that I went into show before.
So do you want a demo?
Hello, Firefox.
You already looking at it.
Great.
Okay.
So making this really quick now because we have less than five minutes left.
This effects actually used in the wild.
Yes, yes, I will in the next project.
It's using us to run or integrate PC games on Apple hardware running Linux.
Parallels desktop is using us for something we don't actually know why, but they created it in some blog posts.
So yay!
Crossover is integrated effects into code we've integrated effects into their crossover product.
Also really cool to see that one working.
And finally, Firebase is going to release the steam frame.
Sometime this year, which effects is an elementary part of at this point.
Thanks to Valve for actually funding a lot of this work.
So I'm personally paid by Valve to work on facts and a couple of other people as well.
None of this would be possible without them.
Thanks to them.
Really quick note about Windows support, which I didn't mention above, but probably some of you might be interested in playing Windows games.
So what you can do in fact is you can just run wine itself as an x86 Linux program.
So it's a Windows game and you run this game using x86 wine with the corresponding set of Windows libraries on top of the x86 Linux libraries on top of the x86.
On top of the x86 Linux kernel and the x86 hardware, but now you swap out the CPU.
So you can introduce recompile and the kernel compatibility layer by effects, and this whole stack all works.
But one other small thing we can do is we can do two things actually.
We can swap out the x86 build of wine with an arm build, more or less skipping over details here.
And we can recompile instead of to arm to a custom binary interface called arm64EC.
Think of it like a calling convention.
And what this allows us to do is shift the recompile up the stack a little bit and then recompile the Windows libraries themselves to this arm64EC adi.
That allows us to throw away the x86 Linux library Linux libraries entirely and the kernel compatibility layer entirely.
So we end up with a much simpler stack where the essentially only the game and some few remaining libraries remain to be emulated.
So much more convenient for emulation.
If you compare to the other stack where we just run x86 wine and effects, just a few less instances here.
We end up with a much simpler architecture where there's few components to emulate and much less time left over, much less time.
With your work to be done over all.
If you want to read more about this, just check out the blog post down here.
And what's next, we will end this talk.
But what we will make text better is the short version of it.
We don't use AI for our work and that's all I have.
Thank you so much.