So our next speaker is Claudio Blay and he's going to talk about remote execution with
Nix and back to a round applause please.
Yeah, hi everybody.
I'm Claudio.
I work at Twig.
I'm going to talk about how we could integrate Nix and back to in the realm of remote execution.
A bit of a background at first.
We were working on large mono repo, which is a really large Haskell code base.
A bit of a 10,000 Haskell modules for Mercury.
So they're using Flex already and we are currently migrating this code base to back to.
So what is back to back to is new Polygraph build system from meta.
I think they released it roughly two years ago now.
It supports distributed builds and also distributed caching, which is quite nice.
It's still being worked on by the way, so they do release twice a month and they did so today.
So if you want to try it out, just head over to there, get up back to repository and download it.
It's just a single Raspberry, a static binary.
Very easy to use.
Okay, why would you even think about remote execution in this case?
No, you want to have faster builds, obviously.
So with the large code base, you're limited to just your local computer.
You're going to feel a bit of a pain if you have to recompile lots of your modules.
And remote execution is your faster build and also test execution through scaling of nodes available somewhere in your cloud or on print whatever.
And also it provides you a consistent development environment.
Think about reproducibility and you can reuse all the artifacts that are built by your remote executor.
In this case, the remote executor or the remote cache is planned by our CI, so you can reuse those artifacts for all the developers.
So they can really benefit from the build results that are done by your CI.
And you don't have to rebuild stuff that has been already built.
In either way, or even in your purchase or on your masterboards, it doesn't matter.
And yeah, there's just one catch sort of because
Back to and then nicks together do not lend themselves naturally to remote execution.
This is because, yeah, buck likes to control their, let's control of have to control of the outputs.
And all nicks is giving to buck to is similar to nicks though, right?
So if you send that slim link, you can send it to to the remote executor that gets executed.
But of course, it's just dangling if you don't have some preparations in place that makes the next door path appear on your remote remote runner.
So in this case, the easy way out for this is to use a custom Docker image.
That's what we do for the link in Linux use case.
Since we are building this with buck, so we're building this pre-cand image upfront before we are even starting in the remote execution.
The remote executor has access to the system and it uses it, so we make sure that every tool that we are using for the remote actions is actually,
already there, right?
But we also have to support MacOS since a lot of developers of Mercury, the company we are working for, are using Macs like ARM64 Macs, M2M3 machines and for MacOS, this whole thing is more challenging.
For instance, you would need to provision the system upfront before we're doing the remote executors or running the most actions on your runners.
And as far as I know, this would be a bit cumbersome deal with, I don't know, there's probably no MacOS image that you can just build upfront.
And also it's quite expensive, having to run those machines, keeping them running or, yeah.
So when you talk about this with a remote executor service that we are using, they recommended us to first look into how you can build your actions on MacOS, but targeting Linux, right?
So, to such as symbols, to just use across remote builds in order to leverage the existing infrastructure that you are already in place for Linux.
And that's what this talk is basically about.
So in a moment I will show you how we set this up and how everything fits together, just doing a quick demo.
Yeah, just wanted to mention there is some prior work from my colleague and the blog post.
I've linked here, Konstantinos and Eugen, who managed to get this implemented in better ways that they're using a shared file system for the runners.
As soon as the mixed packages build by some remote mixed machine, the other runners have immediately access to transparency access to the packages inside of the store.
So this is very cool stuff, but it's only working for Linux, of course.
Still, we are looking into doing that also for our project.
When it comes to remote execution, the history for Linux is actually quite neat.
All you need to do is configure next to be able to forward its build jobs to another machine of the other architectures, so when you're on MacOS,
I'm 64, and you've got some Linux x8664 server.
You just need to configure this machine, all you need to do is you need to be able to access H entered,
and when you try to run some or when you try to build some package, some derivation for a different architecture,
the next will just carry this machine and see if it fits the required features and the requested architecture,
and then it will send off the build job and it will get back the build results, and they will appear in your store locally, and we're going to go.
So this just notice is basically the reason we only build next packages locally.
So we will see this in the second part.
In this setup, we want to have remote cross birds, but we do only builds next packages locally,
so to have them transparently send off to some other machine and come back to us, so we can actually participate from that machinery.
For the back-to-use case, remote execution is a bit different.
For one, there are two different variants of back-to, and there's the internal matter variant, that's these different protocols, different systems,
and there's the OSS variant which uses the basic remote APIs, so it's able to use any,
based on any service that also implements these APIs.
There are some commercial ones, like build by build by your own flow, and there are also some open source,
and build your own servers, services like build by, and just to mention it, there are also a few clients that speak these protocols.
There are some more, so it's quite a cross-pring ecosystem in that sense.
So how does back-to-run remote execution work? There's basically a service, and the service is able to spawn up some workers, depending on your level of cloud infrastructure you have, basically.
It works on the action level, so this is in contrast to Nick's, which has a much larger bracket, right?
So you just walk on the, on the derivation level, basically.
And it usually supports the caching, so the route server will cache your results, and if it happens that you want to build or somebody in your team builds the same,
once it builds same action, it doesn't need to be, it's already cached, and you can download the cached results.
Also, many of the remote execution services provide interesting telemetry, nice UIs, or graphs, where you can see how much time each split took, how much of the cache was used,
you can even inspect the logs and so on, and some of these also have some suggestions that you can apply to your configuration, say,
toggle this lag, and it's going to be faster, such things.
Okay, now it's time for the demo, I had, yeah, I had a faithful Raspberry Pi with me for this purpose, but unfortunately it didn't survive the, it didn't survive the second, the train ride somehow.
Just have to fight, yeah, that's better.
Okay, okay, that's the, okay.
So I basically created a virtual machine last night, and this is the machine, right?
I don't, I'm not sure.
Yes, okay, that's a good machine.
So you quickly could just, like me.
Okay, just, it's working, it's very slow, the Raspberry Pi was much faster, but you can see, yeah.
And it's filling the stuff, you know, the thing, the next way action is run locally, as I said, we only want to run this locally at any point of time.
And then it's going to build, okay, it's going to build the executable, and since I,
I said, it should run the, the resulting, resulting binary, it failed because, of course, we were building for a little six 86 64 on an arm 64 machine, right?
So just wanted to show you how this is basically set up.
What this does is we have two platforms, actually, define execution platforms, one is the local platform that's always related to your local system.
And the other is the Linux 86 64 platform, which is only locally enabled, or which can only run local jobs, if the host system is the same architecture, of course.
We noticed, always enabled, and we are setting some configuration, some constraints, basically.
And also, though, the remote execution properties that we set in order to inform the remote executable, what kind of, what kind of machine we are expecting here.
So I have, I'm running, I'm running native link here, so that's the remote execution service that is running on my laptop on the host machine.
And that this one is executing the remote, remote jobs, if you change like something from the hello, sorry.
Yeah, I don't even have a decent editor here on this machine.
Okay. Ah, no, that doesn't look, yeah, cannot edit the command line.
All right, it just build it, so I hope, no, but of course, couldn't run it.
Let's remove the target, let's remove the target platform, and now it should build locally, right, so, and now it should run.
There, okay, yes, that worked.
We could even have a look, it's, back to the nice system, you can inspect it, and you can even have a look at the execution platform resolution, so that's a back to audit command, and let's see what it says.
You can always debug your configurations that you did in, when you register your execution platforms.
In this case, you can see the original platform.
This, this was configured with is the Linux X8664 platform, and also selected as an execution platform, the same one, and also for the two chain depths that are hand with nicks.
It's even, or it also selected the same platform, so just one important thing to note is that you have to set.
In order to manage this correctly, you have to set constraints on your, on your, on the two chains, that's the last thing I wanted to show.
That's the two chains.
Okay.
That's the main important part is, okay, you don't see my cursor, okay, yes.
This thing which looks a bit funny here, the select statement is basically say, it's basically looking at the current configuration of the target, and then it is given a mapping, right, and we are basically giving it an identity map.
So what this is saying is, okay, when the target system that you built for is X86 Linux, the executable that is, when you use this as an executable, this is also just the same platform, right, so it's a one to one mapping, it's an identity map as I said.
So this makes sure that the right platform is selected from the exit platforms that are registered.
This is kind of important because back to tends to select just the first one, that is available, when you do not add these constraints.
Go back to.
Okay, yeah, that's my time. Thank you for, thank you so much.
Thanks for the talk, great talk. Are there any questions?
The question is, why would you not choose one or the other?
Yeah, the main thing is the developer experience, basically, you want to have really fine grad actions, and we do have put great efforts into doing that, introducing that, you can really build like single objects of your, of your build graph, you don't need to mix, you cannot do that, right?
You can even pick some HTML file that you want to build from your, from the API docs, right, so you just want to build one of these.
Also, it has, it has its caching for, for normal build artifacts, but also for tests, right, so that is, that is, that is, at least what we are after. So it's not done, we are not done with the project, but we'll get to it.
Thank you very much.