All right.
All right, folks.
I don't have a lot of time today.
Initially, I thought this is going to be a 45 minutes, but turns out it's only 25.
And so I tried to cram a lot of the technical details.
And I might be missing a lot of things.
So I'm rushing here.
If there's any question, feel free to ask me in a whole way after.
In case there's no time.
First of all, about me, my name is Sun.
You just call me Sun, that's fine.
I'm from Vietnam originally.
I'm currently based in the Netherlands.
I have been living there for the last five years.
I contribute to GitLab and Gitaly.
I'm quite a version controlled not in general and built tools as well as developer toolings in general.
If you have anything that you like about version control, built tools, developer productivity, come talk to me.
I'm currently working as a solution engineer, built by that logo is right there as paying for my trip.
So is there.
And in the past, I worked for booking.com and Lazada, Ali Baba in general.
Some excuses.
I recently have to travel six, seven city for the last one week.
I'm extremely jet lag.
This slide was finished at six a.m. this morning.
So a lot of mistakes.
Please contact me.
If you've spot any things or if you have any questions.
I can be rich on various different channel.
And yeah, what are we talking about today?
We're talking about first all CI challenges, a scale.
I've been working for big companies.
Recently, I've been talking to a lot of big customers.
Customer who are in the top 10 biggest company in the world, even.
And yeah, the CI is definitely not a solved problem at scale.
There are a lot of pain points.
And that's what we are talking about today.
One of the solutions coming out of all those challenges is artifact first in hermetic builds.
Using those modern and newer built tools.
And I would be touching on that.
And using that is a background.
We can talk about how distributed built system often work.
And the set of remote built execution API, which is the open source project I'm here to talk about.
And yeah, that's open source.
Even though I'm working for a startup based on all these API that API is open source,
you are free to use.
Okay, let's define scale.
What is scale?
First of all, a lot of employees, right?
Whether you're a 90 person small company or
you're dealing with 20,000 engineers, right?
The slower your built, the more time your employee is going to spend waiting.
And those are working hours.
You're paying salary on top of that.
That's costing you money.
That's a problem, right?
The slow built is definitely a problem when your company has a lot of employees.
Having big code issue, that's not just a synonym for monetary posts.
Even though with monorepos, you see that problem happened a lot more often,
because tooling is bright when you use monorepo, a lot faster.
But that's also a problem for multi-repo as well.
For example, one of my previous employer was Ali Baba with tens of thousands of engineers,
contribute to multiple, sometimes hundreds of repuls at the time of certain projects.
And it's slow, right?
Your built is going to be slow if you have a lot of code.
That's just very obvious.
Number three might be not so obvious, but depending on the business that your company is in,
you might have different requirements for risk tolerance.
For example, if you're making robotic application for hospital, right?
You might have slightly lower risk tolerance compared to an e-commerce company,
because one mistake would cost you human life compared to just, you know,
a couple hundred thousand dollars.
So, yeah, depending on the risk tolerance as your risk tolerance increase, right?
For example, you're making more money over time, right?
You get more revenue.
One month could cost you a lot more money, or if you're in the business of, for example,
self-driving hospital equipments, or even aerospace, right?
Those can be quite costly as well, but in terms of human life.
So to improve on that, you have to do a lot more testing, security testing,
shifting left.
That means you have to do a lot more testing, and all that increasing the amount of test,
meaning that you have a compute problem, right?
More tests equal, more compute.
You can see in this scale, you can start off with a very simple pipeline that has a single job
that would do you just fine, initially.
But over time, you're going to see yourself evolve into a multi-station pipeline,
because, you know, you have more compute, you have to distribute your compute into multiple steps
that are very traditional CI system kind of deal.
But even after you distributed it, you still have to pay the bill for those compute, right?
And how do you save money when you have a distributed compute problem?
You cash, right?
Selective testing, cashing, and how do you cash things accurately?
You have to, and how do you select which node, which step to run, which step not to run?
Right?
Eventually, that's usually come down to defining some condition, right?
When to run, when not to run.
For example, if I changed this external dependency, then run my bill,
but if I didn't change it, then don't run it, right?
And all of those conditional will often be encoded into your repository, your code, your CI pipeline,
at the final state, which is a graph, right?
This is usually a deck, but I have seen more than a deck, but a deck is usually reasonable enough.
And overall, with this, you know, your compute keep growing and growing over time.
So, yeah, finally, cashing, right?
But when you have cash, you have to cash invalidation problem, right?
Cash invalidation is really hard.
It just is one of the toughest kind of problem that you have in computer science in general,
and below naming, of course, but naming is, we don't talk about naming here.
You have a gift coming in, right?
You would preferably, you would know that, you would want to know that,
hey, how does that this would affect my bill graph over here, right?
Some of the notes that are not affected should not be run, right?
You can either eliminate them by cashing and reusing the previous result,
and you have some sort of smart quarrying engine that you can filter out the execution,
and only execute the note that a relevant to your bill.
But even with cash at scale, you will still have a compute problem, right?
You, no company might have a lot of users.
Actually, your user could be running tests on a different waste commit, right?
For example, I could be testing on top of the waste commit that is on top of master just five minutes ago,
but my coworker could create their branch from a waste commit that is a window go, right?
With different base, you have different graph, different bill graph underneath, right?
And with that, it means that the Tory amount of graph that your bill system need to compute increase over time, right?
All these notes cannot be dW kit, and it would be nice to be able to dW kit them somehow, right?
Otherwise, you are just spending your money away, solving this compute problem.
And yeah, we are a CI provider, and even with customer with 90% cash hit rates,
we still see a huge amount of compute required to serve CI system.
So yeah, to recap, this is the definition of scale.
This is what we would try to address by the solution of bill tools today.
First of all, you have big code problem.
Then, when you try to solve it with the compute needs that big code introduced,
you're going to run into cash invalidation problem, right?
And with cash invalidation problem, you want to solve it with dependency tracking,
by creating a DAC in general.
And even at the end of the day with cash and dependency tracking in place,
you will still run into a fundamental compute problem that is really hard to solve.
The solution I'm presenting today, don't claim to solve all these, right?
But it does a really good job at solving a lot of these.
Even in the biggest of scale that we see today with the top company in the world.
So this is a solution I'm talking about, which is modern bill tools, right?
It was what's better than constructing a graph than bill tools.
Even get up action today is just a glorified distribution of bill tools, if you think about it.
Because at the end of the day, all that does is it take up your source code and construct a graph and then execute the graph, right?
Here I'm talking about specifically Basel, which originated from Google,
but originated from Meta, Facebook, and Pans originated from Twitter, right?
There are several other bill tools that are really nice, they are getting there, but not quite.
The reason why we like Basel, Buck, Pans, the most is that, first of all, they are one of the first who was able to get a lot of the catching story, right?
A lot of the distributed bill correctly, and finally with Basel and Buck, they are very mature in terms of a bill telemetry that is built into the tool, right?
And you kind of need that as scale, but I'm not going to talk about bill telemetry today because I need to remove all that to fit in the 25 minutes.
Anyway, so what is artifact first bill tool?
So when you think of bill tool, you often think of Meta Phi, I would assume, right?
And Meta Phi can, if you spin hard enough, you can think of it as an artifact first bill tool, but it's not quite.
And here I want to try to highlight what's the main differences.
And I think the main difference is that artifact first bill tool, when you think about it, it would try to, the number one goal of an artifact first bill tool is that it's there to produce the artifacts, right?
To run the task that produced the artifact.
So if the artifact that is very on disk that you have is very exist, it either exists, then the task should not need to be rerun.
And that definition provides you with a fundamental way of passing things in your, in your bill system, right?
And I would have a narrow slide to talk about none artifact first bill tool later, but the fundamental thing is that if the artifact does exist, then don't run the task, right?
That save you a lot of compute.
But yeah, in terms of side effects, for example, a lot of people use Meta Phi to do release, right?
Release doesn't really produce any artifacts, right?
It's just called an API and get it over with or deployment.
Those are often we call sci-effect.
And with an artifact first bill tool, because artifact first bill tool do not handle this artifact.
This artifact will be handled by a separate system.
So with a setup that use artifact first bill tool, you have a segregation of concern, right?
artifact first bill tool is there to help you produce the artifacts.
And then you use something else to ship that artifact to production.
And both of them can be quite mature in terms of solution.
But here today, we are mostly focusing on the bill tool part.
Inside an artifact first bill tool, this tool for all the bill tool that I least earlier, which is based on buck and pants, you have an actions, right?
Then action definition here is that it's a transformation from one set of inputs to a set of outputs.
What can the input be?
The input can be your source file.
It can also be your tool, right?
Your compiler, your linker.
It can be the command that you use to invoke that tool.
For example, you can say, GCC, go compile me, this source file.
That's a command, that's also an input.
The environment variables going into that process to run this action also can be an input.
The platform information, the target CPU, et cetera, can also be part of the inputs.
And all of these can create you some outputs, for example.
Example, output can be standard, I'll standard, or of the action, and the bill artifacts, like directory and files.
All of these will get hatch, right?
They will get hatch down to digest.
Digest is a combination of, of a hash, a strong hash.
It can be shot to 56 or like three.
If you have really big block, you would prefer using like three instead, and the size of the artifact.
Why do you want to have the size here?
And get there's no size.
Well, the size help you predict the size of the action overall, right?
So, in later on, if you want to run this action remotely in a system that is not on your own computer,
these size will help your scheduler think about where to put the task in.
Right?
So, all these gap hash, all the source file, it will be put into different directories.
Those directories also going to get hash, right?
And all of that will become a macro tree.
And what does that do?
First of all, it led you cache stuff in the remote cache really well, right?
Everything get hash, and it gets star, oops, sorry.
It gets star in a content addressable star.
So, everything get dd duplicate really well.
But on top of that, you can apply a schematic onto your action.
The schematic essentially say that in a present of a sandbox, your action is reproducible, right?
What's goes in is completely in control, and you can reproduce it reliably, right?
And with the source identity that I just mentioned, change can be accurately identified,
because everything will compute into a mercury, and if you invalidate one source file,
if you change one source file, the hash of that file will change the hash of the entire action will change,
and the action can be invalidated, and the cache can be invalidated correctly.
I'm rushing here a bit, because I just get a sign that I only have a few minutes left.
But yeah, this is how you cache the action.
You put all the input file, the command, the platform into one file.
You hash that file, that's the final 10 on your mercury.
You use that as a cache key, and then the cache value is your outputs, right?
And that would go into the action cache entry.
Now, the nice thing about the action cache entry is that multiple notes on your CI system can use that action cache
to avoid having to run that action over and over again, right?
Humanity CD can guarantee that if an action has run on one computer, it can be reproduced,
so it doesn't matter where it runs.
It can run on a second computer or it can run on a computer in a data center somewhere, right?
And it doesn't matter.
But in fact, you know, implementation, why Humanity CD is not, you know,
binary property is not a yes or no, but it's a combination of different things that can be applied depending on your requirements.
Yeah, and because, you know, with hermatic actions, it doesn't matter where you're running it,
you can run it on the same CI notes, or you can run it in a build farm, right?
And in case of build farm, your CI notes going to upload the action to the scheduler,
the scheduler will assign it to a worker, the worker build your outputs, and then upload that to a remote cache.
And then multiple different notes in your CI system can benefit from that same action.
This, the nice thing about distributed computing, which is that if you have different CI systems that issuing,
that requiring the same action to be compute, the scheduler can be replicated, right?
And this can help with the distributed computing requirement a little bit at scale, right?
And this has been proven to be working for the life of Google or Apple in general.
Finally, I just want to touch real quick on the remote execution API.
Powering all of this is the set of API for different components to talk to HR.
On the left-hand side there, you see basal talking to remote cache.
It's talking to action scheduler.
Over here, you see our worker also talking to the remote cache.
Behind all of that is a set of open source API written in Protobuff and GIPC, right?
It has several different components into it, but essentially on your left-hand side there are different ways that different client and worker can upload the block to the remote cache.
Right? All with fine-missing block and get tree RPC, they are essentially for you to save in terms of the total amount of download upload or the amount of round trip that you need to interact with the cache.
On the right-hand side here, as I said, action cache is just essentially a key value store, right?
You can upload some key and value to it or you can get a value to it using some key.
Finally, you have execution which is executed remotely on server farm.
All of these open source and there are a lot of different implementation of these APIs, right?
These are all the built tools that has implemented some part of the remote built APIs today, right?
And these are all the server that implement the server side of it, right?
So the nice thing of a remote built API here is that you have interrupt, right?
There's an open specification that essentially help you avoid vendor lock-in when you start selecting these kind of technology.
Obviously, one of them in my employers, I'm plugging them, but yeah, it's supposed to work with any of these setup right here.
Now remote API is not perfect, right? There are a lot of things that we don't do well.
I'm just highlighting here just a few points that come up in recent meeting.
I mean, the working group of remote API, but first of all, we have not captured windows that well.
We had Microsoft came up to a basalcon recently and gave a talk about the internal build system that actually was inspired by remote APIs,
but they use a slight deviation of it.
So that's going to show that this solution, the scale well, but there are some slight Microsoft specific logic that they want to be in there.
We know that Unreal Engine recently introduced their own remote build system,
but they don't use remote API and we really want them to do it.
So hopefully we can invite them to come to the table and help build remote API in the future for window use case better.
To be fair, right now, everything is still working on window, there's a few edge cases that might pop up here and there.
We have a working group from support massive block, right?
Recently, we started seeing a lot of LOM being used, a lot of containers being built with your build tools.
So a massive block is going to require some more work to fit it into the protocol.
And finally, we don't have a standardization for build telemetry.
Even though both basal and buck, well, buck 2 in general, come with their own build telemetry.
They are now standardized, right?
So each of them using their own API, their own set of events.
So right now there's no standardization for it.
So different server implementation are using different API over there and you cannot interrupt it in between system and that's a little bit of downside.
We are trying to get that sort of hopefully within the next year, but yeah, come talk to us if you're interested in this area.
And yeah, that's my talk. I think that's about time.
I'm sorry, I'm brushing through a lot of details, but yeah, that's the gist of it.
The slide is available on a first-time website.
And if you have any questions, fire away or reach me via email.
Yes?
So if I'm driving you on these two, it's come from, you know, the security system world or web.
Are they using what can get that system world at all or...
Doesn't really good question. Are these two being using embedded system?
And yes, the question is yes.
There are recent several talks from Google, where they are using basal to create different embedded system.
Obviously, they have like their all Google home line up as well as Google watch, Google phone.
All of that is currently being built with basal or blaze internally.
Now, you have meta who are working on their own glasses or everything in meta are being built with about 200%.
So the story for...