Hello everyone, I would like to introduce our next speaker, a Patrick Steinhardt.
Thank you, so good morning everyone and welcome to my talk, evolving it for the next
decade.
Let me first introduce myself, so my name is Patrick Steinhardt.
My interest in open-source software started around 22, when it was 11 years old.
My elementary school teacher back then was very big into computers, so we spent almost
every week in a well equipped computer room.
One of the things that my teacher introduced me to back then was this magical thing called
Linux.
Linux really got me hooked.
You could play around with it, make it to weird stuff, and break the whole computer while
trying to make your Windows burn down when you close them, or just to get a fancy 3G cube
if you for example want to have virtual work spaces.
My father on the other side wasn't that happy, because we had frequent arguments around
why I wiped the computer once again, or why the engineer doesn't work.
That being said, that eventually kickstarted my interest in software engineering.
I bought my first book about programming when I was 12 years old, and eventually started
to do some small contributions to open-source software in 2011.
In 2015, my involvement with open-source software development changed significantly when I found
a job posting that was about contributing to an open-source version control system.
The deal was rather simple.
I had to do something related to version control systems, and that knowledge that I gained
was then sold to customers by doing trainings and consulting.
I had reached a choice of which software projects I wanted to contribute to, and there's
exactly what I wanted to do there, which was awesome.
For me, the choice was basically between subversion and git, and I had to release that
by both ecosystems.
To be honest, I just came from a job where I had to use subversion, so you might understand
why I'm not wanted to have anything to do with subversion in the first place.
I instead chose git, and that's how I eventually became one of the core contributions
just to both git and lip-git two.
In 2020, I then switched to git lab as a back-end engineer, where we're working on giddily.
Giddily is the RPC service that sits between git lab and your git repositories.
Eventually, my responsibility shifted the gattle a bit, to us contributing to git upstream
only.
We faced multiple performance bottlenecks that we wanted fixed, and fixing those with
required significant long-term investments to git.
One of these efforts was, for example, the Reftable Back-end, which I've been talking about
way too much over the last couple of years, and this somehow also gave me the nickname
shiny the Reftable Guy and some contexts.
Giddily's done.
What is there to change?
This is something that I heard quite often over the last year, and I get it.
Just last year, git has turned 20 years old.
Anyone uses it.
It works, so why mess with the success?
The success of git is indeed quite staggering.
94% of all the developers out there are using a day-to-day, and there are hundreds of
millions of git repositories out there, and many more strips depending on it.
It is safe to say that git is everywhere nowadays in the world of self-redevelopment.
So that appears the question, is git really done?
Is it the perfect version control system that doesn't require any changes anymore?
Well, it might not surprise you, but for me the answer is a definite no.
The world has changed quite significantly since 2005.
Git was designed for a different era.
In 2005, Shawan was considered to be a secure hash function, but that has changed with
the shatter-to-tack and other attacks on Shawan.
Back then, the linear strunnel was considered to rather large repositories.
Nowadays, it's dwarfed by repose like the cromely repository, which are almost a hundred
gigabyte in size.
CI systems were kind of the exception, and it could count yourself lucky if you've
had, for example, a Jenkins instance available to you.
Nowadays, frequently see projects where you have crazy huge CI pipelines, where every single
commit kicks of thousands of jobs.
Monor repose is a term that nobody has heard about back then, but nowadays, everyone
is using them.
And also, it was very hard to use back them, but to be quite honest, it's still hard to
use nowadays.
So the world has changed, and that makes us clear that Git needs to change as well.
But the unique position of Git means that we cannot have a revolution.
There are millions of developers out there, and many projects that rely on Git.
So we must make sure to not break the world, which simply can't break the world.
Too many people depend on Git, but we can make it better, one step at a time.
This evolution is what I want to talk about today.
I want to highlight a couple of important transitions that Git is currently going through
to ensure that the product stays relevant as the world keeps on changing.
So let's dive right into our first topic.
This is one of the most visible, user-visible changes that has currently happening in the
Git world, the Shart of 56 Transition.
Shawan is a central part of Git's design.
Every single object has an identity, and that identity is computed by hashing the
contents of the object.
So for blobs, we hash the file contents, for trees, we hash the directory structure,
and for commits we hash authorship information, commit message, and the root tree.
Git objects are set to be content addressable.
Given the contents, you know the name of the object.
The result is that you have implicit integrity verification for your object.
You can easily deduplicate them, and the history becomes immutable.
This object name is computed by using Shawan.
If you have, for example, a blob that contains the storing hello world, you only need
to prefix the object header, and then compute the Shawan to get the object name.
This one problem though.
Shawan is not a secure hash function anymore.
In 2017, Google and the CWI Amsterdam Research Institute proved that known theoretical attacks
on Shawan are viable in practice with the Shattered Attack.
The result of this partnership are two syntactically valid PDF files that both result
on the Shawan hash.
The attack requires around 9kF1 computations, which requires 110 years of single GPU computations.
This may seem like a lot, but if you for example want to brute the force Shawan instead,
a hash collision would require 12 million GPU years to compute.
So quite broken.
Also, you can imagine that with all the recent type that we have around artificial intelligence,
data centers have become increasingly expanded their GPU capacity.
So nowadays, it is very much in reach for a large player to compute hash collisions at
well, if I want to.
Of course, as Git heavily relies on Shawan, the Shattered Attack has kicked off a huge
and intense discussion on the Git mailing list.
It has been asserted since the beginning that the use of Shawan is not primarily for security
though.
There is a couple of arguments that are made in this context.
First, the object hash is mostly used as an integrity check to detect disruption, bit
flips, and transmission errors.
Also, source code is transparent.
If you see a merchant best for example, where somebody enters random collision data into
your source code, then you might probably ask some questions.
Also, the object format that Git adds has at some protection, because we prevent the object
length to the object.
This means that you can adjust like, prevent collision data to that object.
And last but not least, there is also other security measures, like GPU signatures, and
correct the transports at a web of trust between developers.
But the reality is that things are a little bit more complicated.
Also, you can use GPU signatures to sign your commit, but unfortunately, that signature
is on the Shawan commit hash.
Subsequently, if you can create a collision, you cannot trust GPU signatures at all anymore.
Also, not everything is source code.
Whether you like it or not, some repositories out there contain binary blocks like firmware
or compiled assets, for example.
It's almost impossible to verify whether those might have contained trusted collisions or
whether because they are not human readable.
Also, a lot of our modern tooling builds its trust on top of Git commit-tashes.
You see this in CI-CD, in scripts that interact with Git programming languages that perform
dependency-pinning and so on.
And many of those use cases, we implicitly trust the Git commit hash.
And finally, government and enterprise policies also mandate their removal of Shawan by 2030
in favor of more secure hash functions.
So overall, it's safe to say that even if Git itself does not rely on Shawan for security,
the ecosystem very much does.
The fix for this is of course quite obvious.
Let's just swap out Shawan and replace it with a different hash function.
And that's exactly what happened.
In Git-229, which has been published in October 2020, we have added support for Shawan
and Shathive-56 instead.
You can simply create the new repository with saying Git in it, dash-upperic format equal
Shathive-56, and then you'll get to use a different hash function.
The code is there, it works.
You can use it today for your new repositories.
But somehow, nobody out there is using Shathive-56.
So what is taking us so long?
The problem is that Git has very strong networking effects.
You cannot just implement Shathive-56 in the Git command line and then call it the day,
because we also need to consider the ecosystem.
But unfortunately, this situation looks somewhat grim here.
Next to Git, there's only a single forge and a single library that foolishly support Shathive-56.
GitLab and if you other libraries have also experimental Shathive-56 support.
But some, well, rather insignificant players like for example GitHub, don't support Shathive-56
at all.
This creates a chickened act problem.
Nobody's moving to Shathive-56 because it is not supported by large forges, but large forges
don't implement support because there's no demand.
The problem is that we cannot wait forever.
It will become more and more feasible over time to break Shawan.
And the next cryptographic weakness may be just around the corner.
We need to consider that even if we had full support for Shathive-56, projects still need
time to migrate.
So that's why we need to break the cycle.
The Git project has decided to make Shathive-56 the default hash for newly created repositories
in Git3.0.
Our hope is that by making Shathive-56 the default hash function, we are forcing both forges
and third-party implementations to adapt.
The message is clear, Shathive-56 is the future, get ready for it.
This transition will likely not really be an easy one, and it may result in a couple
of hiccups along the road.
But if you're interested, you can also help.
We can start playing around the Shathive-56 backshared and repost with it.
You can show your favorite code forges that you care about Shathive-56 so that they
bump the priority, and you can even try and help third-party tools that depend on Git
by adding Shathive-56 support.
Together, we can hopefully get the ecosystem to move before the next vulnerability in
Shathive-56.
I need to live up to my nickname Shathive-56, and talk a little bit about my favorite
topic, which is RefTables.
RefTables are another significant shift that is currently happening in Git repositories
as a new back and to store your references.
Before we talk about RefTables, though, I first want to give you a little bit of context
about how references are starting Git right now, and why that is a problem.
When trading or updating references, they are by default stored in the loose format.
Every reference is stored as a separate file.
The reference format is really easy to understand.
To demonstrate, let's create a simple repository with a commit and a branch.
The first file will examine its hat.
This file indicates what you are currently checked out branches.
As you can see, it has a rough column prefix, which means that this is a zombolic reference,
where the target is roughshath's main.
So we know we have the main branch checked out in that repo.
All the other references are stored in the RefToric to rehire key.
As we can see, we got two files in there.
RefTables feature and refshath's main.
These are all branches, and they contain the object that either pointing to as contents.
Now storing every single reference as a separate file works well when your repository only
has a handful of them.
But when you have hundreds or even thousands of refs, then it becomes
really inefficient.
Every reference needs a separate I node, which typically also means that it needs a full
disk sector.
Listing all your references also becomes increasingly expensive, as you have to read many
many files.
So git regularly packs your references to contract this.
Instead of storing each reference in a separate file, they get packed into a packed refs
file.
You can manually pack references by saying git pack refs-shall.
You don't typically have to do that automatically that yourself because git does it automatically
for you.
As you can see after executing this command, our loose references have gone.
Instead, we now have a simple packed refs file that is a simple sorted list of all the
references that have just been packed.
Now you know how the files back and works for storing references.
Why does it need to change?
The first problem is that file systems are simply weird.
Once special system here in my heart is Windows, which resolves all kinds of file names.
And as we encode reference names, via the file system path, it means that you cannot
create a branch that is named con, pure n, ox, null, con1 to 9, LPT1 to 9 and more.
There's also many file systems out there like nTFS, fat or hfs plus that are a case insensitive
by default.
And again, the consequence is that you cannot create two branches that only differ in
casing.
Mac OS also does somewhat weird stuff, where they may change the way that your file name
is represented and encoded in case they can say in unicode characters.
So what did one store on disk and what Mac OS actually decides to store on disk might
be different.
In the best case, you know that these restrictions apply and won't ever try to create
such branches.
In the worst case, you're stuck on Windows.
If you want to write 20 references, you have to create 20 separate files.
This does not only take long when you consider performance, but for typical file systems
it also means that each of these references may require four kilobytes of storage.
And that is up rather quickly.
Packer references though is expensive, but mandatory when a repository has many branches
to retain good performance.
You have to rewrite the complete pack rest file on every repack though, which is typically
fine for repost that only have a handful of them.
But, get users are not always reasonable.
One of the worst repositories that we for example host at GitLab is containing around
20 million references, which sums up to a pack rest file that is two gigabytes in size.
Which brings me to the next point.
Deleting a reference also requires us to rewrite the complete pack rest file.
So every time someone did release a reference and that repository, we have to rewrite
two gigabytes of data.
And to add insult to injury, this repository typically deletes references every couple
seconds, not exactly efficient.
Concurrency is an afterthought.
There are multiple issues with representing references as single files when you have multiple
readers and writers in your repository at the same point of time.
One of the problems is that it is impossible to get a consistent view of all your references.
To get that you would have to open multiple files and each of them could change concurrently.
So when somebody writes to the repository while you read references, you cannot tell whether
the result you got is consistent or whether it is a mixture of the old and the new state.
Similarly, it is impossible to write more than one reference.
Each reference you want, you want to write is a separate file and this you cannot commit
them all at once.
There is also no way to log the reference database.
Well there could be a central log file including that retroactively would likely break
all kinds of use cases out there.
These problems have all been known for a very long time already and that is where the
Refttable Backend comes into play.
You can create a newer repository with the Refttable Backend by passing dash as Reftformit
equals Refttable.
To get in it, that is basically all you need to know.
After what the repository is expected to behave the exact same as with the files from
it.
But still let's have a deeper look at what the repository is structured like.
Here isly enough we still see that we have the Reft directory and the Had file, which
were also present in the files backend.
But these files are merricompatibility stops.
They don't contain any actual data, but they have to exist because it does not require
a repository to be a repository unless those files exist.
What's new is that we now also have a Refttable directory.
The first data structure that is specific to Reftables is the tables.list file.
This file tracks the currently active list of tables in the repo.
So whenever your update reference, give it a right and new table and depend it to this list.
This mechanism is really important because it allows for atomic updates.
You can get a consistent snapshot by reading the table slot list file and then loading
all of the reference files.
Then you can perform an atomic right of multiple tables by writing a table, writing a
temporary table slot list file, and then atomic killer renaming that into place.
The tables themselves are stored in a binary format.
While the binary format is more complex than a text-based format, it allows us to store
data more efficiently.
Also as reference names are not in code if you have the file system path anymore, you
are not subject to file system limitations here.
Reftables use a block-based structure.
Every block is exactly 4 kilobytes of data that so that it fits exactly into a disk
sector.
This allows us to efficiently read a single block.
Each block also has a specific type.
Reft blocks, for example, store our references, but there are also other types that we
will not go into detail today.
Furthermore, every section of blocks may have an optional index.
Each index entry stores the last record name that a given block contains, which allows us
to quickly find a specific record.
So let's say before we want to look at the branch called D, we would then first with the
index block, it tells us that the first block contains references up to rest hat C, and
we know that this block will not contain the reference that we are searching for.
Second, we see that the second block contains also references up to rest hat G.
So if the reference we are searching for exists, it must be in that block.
So read the target block and search our reference there.
Now if we zoom into one of those ref blocks, we can see that the blocks contain a lexicographically
sorted list of refs with the irrespective object ID.
One important bit here is the great out part of the ref names.
These are prefixes that are common with the preceding reference.
The ref table format uses prefix compression to save a little bit of space.
Instead of storing the full ref name, we only state how many bytes to reuse from the
preceding reference, and then only have the deferring bits.
A major difference is also how we pack references.
With the files back end, we write lots of ref loose references all the time and eventually
do an all into one repack.
With the ref table back end, thanks for a little bit differently.
Every time we write references, we append a new table to the stack of tables.
Afterwards, we verify whether the tables form a geometric sequence.
The next table must be at most half the size of the current one.
This check happens every single time we update the stack.
So we basically ensure that the stack is well optimized as we go.
To demonstrate, we start out with two tables, one that contains eight references and one that
only contains a single ref.
So let's write a new table with a single reference.
We see the geometric sequence, property is not maintained anymore, as one is not smaller
or equal than half of one, so you're merged them together.
There is no need to come back further though, as two is less than half of eight and
so the remaining tables form a geometric sequence.
We create another table with a single ref.
We see the geometric sequence obtained, no need to merge.
If you now create another table, then we will have to merge one time, but the geometric
sequence is still not maintained, so we have to merge a second time.
The result is that we always have at most all lock ref's many tables, which ensures
that reads continue to be fast.
We now have a very rough understanding of how ref tables work, but why do we do this whole
exercise to swap out the storage layer in the first place?
With the files back end, we're subject to a lot of specifics of a file system, as reference
names are derived from path names.
Reft ref tables all of these issues go away, as the ref names are encoded in the individual
tables.
Also while very much hope that you don't have to rock and repose that contain million
of references, such repose exist out there, and if you work in them, then you might
very well appreciate the improved performance.
All of the files back end does not allow for atomic updates, as references are written
one by one, as separate files.
This issue goes away with the ref table back end, where our reads are consistent and rides
our atomic.
You probably don't care too much about this on the client side, but on the server side,
this is a huge improvement.
In last but not least, ref tables will also become the default in get 3.0.
So if you for example use githin scripts, or if you use it on the server side, then you
should verify that you don't play weird games by accessing reference directly
if you're at the file system.
You should always access references via githin command, and if you do so, then you shouldn't
observe any differences.
So, we talked a little bit about how we improving support of our references in get.
But this addresses some theoretical issues, I would claim that most of you in this room
probably don't encounter those problems in practice.
When it comes to scalability bottlenecks, the more important problem tends to be large
files.
The storing large binary files in get is unfortunately not a use case that is well-supported
nowadays.
As a workaround, develop a set to reserve to third-party solutions like, for example, get
LFS.
This is something that we want to change.
But first, why are large files a hard problem for a get in the first place?
There's two important issues here.
The first problem is how get compresses objects.
Get works extremely well for repositories with text files, like for example source code.
First, it uses z-lib compression to reduce the general size of objects.
And second, get no store incremental changes to objects as deltas.
Together, this achieves great compression ratios for text files.
After all, this is what get was designed for.
But unfortunately, z-lib compression tends to not work well for binary files.
And computing the deltas becomes increasingly expensive for larger your files get.
The consequence is that even small edits, to such files, end up trading entirely new objects
without using any deltas at all.
The second problem occurs on the networking layer.
Whenever you clone a get repository, you get a full copy of all of the history by default.
This is what you want for normal repose.
But once we're talking about large monorepos, with binary files in them, then you probably
don't want download hundreds of gigabytes of data.
This is further stressed by the fact that there is no support for resimble the clones.
So if you have download 400 gigabytes out of a 500 gigabyte repository and your network
disconnects, then you will have to read download everything.
And because deltification does not work for large binary files, you have to read download
the full block contents every single time a large binary file changes instead of only
fetching the incremental changes.
The result is that many teams work with large files simply avoid using get all together,
which is unfortunate.
Of course, large monorepos don't only cause issue on the client side.
Code forges are also struggling with them.
First and foremost, forges don't have the luxury of partial clones, for example.
It needs to have all objects available, as it would otherwise not be able to serve those
to the client.
The consequence is a significant storage cost.
Our analysis on GitLab.com has shown that 75% of our storage space for get repositories
can be attributed to binary files larger than one megabyte.
The huge repose sizes also cause repository maintenance to become very expensive.
We have to rewrite objects every once in a while, for example, to delete some of them.
If your repository contains large binaries, then this data becomes computationally very
expensive.
Also, it is not possible to offload any of those objects to a content delivery network.
All data needs to be served by the git server, and that makes it become a significant
bottleneck.
So in summary, large objects are a significant cost factor for any large git provider
out there.
Git users have adapted to work around those shortcomings with band aids.
GitLab as is one such solution.
Instead of storing actual file contents in Git, you end up storing only a pointer to the
object contents.
The actual content is then stored on a separate server that is better suited for storing
binary data.
This solution keeps the repository small, and is well supported by hosting providers.
But it's not part of Git.
It doesn't know to transfer deltas, and once you have accidentally committed any large
blob into your history, then it's stuck there forever.
Azure clones allow you to clone a repository while filtering out certain objects, like
for example, blobs.
This mechanism is a native part of Git, and transparent to the user, as Git will automatically
fetch those missing objects on demand.
But users have to know how to use it.
There is no automatic pruning of large old files that have not been accessed for a long
time, and servers still cannot offload the traffic.
Azure clones already have existed for quite a while, but I bet many of you have never
used them before.
Overall, it's quite simple.
It's simply specifying object filter, and from there on Git will fetch those filter
blobs whenever required.
If we, for example, clone with dashed to filter blob none, then Git would first fetch all
the non-blob objects, which basically leaves us with the repository shape only, but we
don't have any contents at all yet.
When the check-out begins, and Git realizes that it actually needs to fetch some file
contents.
So, it does a back-to-fetch for only the blobs needed to satisfy checking out the hack commit.
If we eventually need other blobs, that we don't have yet, then Git knows to fetch those
on demand.
For this specific repository, the result is that we have to only download one of the two
gigabytes of data instead of two.8.
Let us mention the problem is that Git users need to know how to use them, and on the
server side, the traffic's still cannot be offloaded to secondary servers.
That's the problem that large object promises aim to solve.
The idea is quite simple.
When a client clones, the server announces a set of promising modes to the client.
Where each promising mode has an object filter attached to it.
The client then automatically selects a subset of those promises and uses the attached
filters to perform the initial clone.
The use promises get stored in the configuration, and are from there on, used whenever the
client needs to backfill some data.
This tries to solve multiple problems.
First, we can now offload traffic to a secondary git server, and this reduced load on the
primary.
Also, the functionality is built right into Git.
There is no need for external tools anymore.
This solution is also fully transparent to the client.
The server can announce an optimal filter, and the client can automatically use it.
But there is one key feature here though.
Git doesn't only support HDPS or SSH clones, but in theory it can also support fetching
via arbitrary transports.
This support is extensible by having so-called remote helpers.
A remote helper is a binary that is simply called Git remote something.
So when Git for example sees the S3 protocol, it loads most to look for a binary called
Git remote S3, and if it exists, it reduces that one to talk to the remote.
The key realization now is that announced problems are remote.
Main of example uses a protocol that stores large objects in an S3 compatible store.
This allows us to offload objects to a content delivery network, and it allows us to store
large blocks in a format that is much better suited for them.
With large object promises we will have the infrastructure in place to let service offload
binary files, and clients will know to automatically use them if desired.
But we still have another issue.
Even with promises, Git's object format still doesn't handle binary files efficiently on
the client side.
This is where a plug will object databases come into play, which will allow us to introduce
a new storage format for a large binary file specifically.
As mentioned before, Git's pack file format uses data compression to store incremental changes
to objects efficiently.
This works amazingly for text files, but for large binaries, computing Delta's this way
is way too expensive, so Git doesn't even try.
Instead, even small edits create entirely new objects once the objects reach a certain
size.
We need a format design for binaries, where incremental changes to a binary file only leads
to a small storage increase.
This new storage format also needs to be efficient for any file size.
The computational complexity should at most grow linearly with a file size.
In last but not least, the format also needs to be compatible with the existing format
somehow, so that you can mix and match the old storage format for text files and the new
storage format for large binaries.
The storage format is deeply baked into Git, but alternative implementations like
Git2, Gogit and JGIT already have plugable back ends.
So there is no fundamental reason why Git can't do this too.
It requires a lot of plumbing and effecturing, but it's certainly a feasible thing.
Assuming that we had plugable databases and that we could swap out the back end, the
idea would be to introduce chunking into Git.
With our current electrification logic, we have to do expensive calculations to find
ideal data, which is simply too costly for binaries.
With chunking, though, we can deduplicate common parts by cutting a large binary file into
smaller chunks, and each of these chunks can then be deduplicated individually.
There's two significantly different ways of doing chunking.
The first and obvious way is to simply split a file into fixed size chunks.
To on this example, use for example, cut the file after every fourth character.
The problem, though, is that if you insert new data at any point in the file, then all
the chunks that follow afterwards will now change.
The result is that we cannot deduplicate those chunks.
The alternative is contentifying chunking.
The key insight of contentifying chunking is that boundaries are determined not by length,
but by the content itself.
Every time a specific property is triggered, we cut a new chunk.
The result is that the file will cut the cut into chunks of variable length.
So if you insert data at the beginning or anywhere in the file now, then the first chunk
will of course change.
But because the boundary is defined by the content, we know that we are still going to cut
subsequent chunks at the exact same boundary, and this remaining chunks will remain identical.
The mechanism used for this is to pick a rolling hash function over a sliding window.
When the hash matches a condition like for example, being divisible by n, then we cut.
This is how tools like for example, the rest is or bark.
It also our clone can handle large file backups efficiently.
We don't really need to replace its entire storage format.
It works quite well for text files, and contentifying chunking would likely make compression
ratios worse for them.
We get already supports multiple object sources attached to it, so that you can use the
alternatives mechanism to have two different storage types.
The idea is to connect two object sources, and based on whether or not a binary file is
a binary file, you would either store it in the chunked format, or you would store it using
pack files.
The two efforts to introduce large object promises a pluggable object that they are based
on progress in parallel.
The initial protocol implementation for large object promises has landed in get-to-50,
and has been extended in get-to-52.
The next steps are to automatically use filters and promises on the client side.
Overall, this is quite close to being usable in production.
I would assume that over the next couple of releases, we will have all the required parts.
What is of course the missing is also support in getforges.
The effort around pluggable object databases is not that far yet.
Over the last couple of get releases, we have spent some significant time refactoring the
code base and how get access is objects.
Stouching with get-to-53, which will be released tomorrow, no, in two days actually,
we will have a unifiesed object database in the face that makes it easy for us to change
the format going forward.
In get-to-54, I then expect that we will have an initial proof of concept, but implementing
the chunk forward will probably take a little bit longer.
Once those parts have landed though, get will become a lot more viable for large binary
files without work around.
The last couple of sections have been about technical details.
One core area though that get does get a lot of complaints about is its UI.
Many commands are extremely confusing, and some workflows are significantly harder than
they have any right to be.
And recently there's been a competitor that makes us have a hard look at ourselves and
what we're doing.
Judges is a modern version control system that's fully compatible with get-to-repositories.
It will start to the couple years ago by MatiFunzvike back at Google, but by now it has
a growing open source community.
You can use it and exist in existing repositories, get-to-repositories, push to large
forges like GitLab and GitHub, and your collaborators won't even have an idea that you're
using JJ.
Everyone knows that gets user experience is not exactly the most loved one, and indeed,
many people seem to prefer the JJ's experience way more.
It's of course not much of a surprise.
The get-to-user interface has grown somewhat organically over the last two decades, which
leads to inconsistencies and commands that simply don't feel modern.
JJs is started from scratch, and it took all of the lessons that I get learned the heart
right way directly to heart.
As a get-to-repstopper, I was naturally quite curious, so I had a look at JJ quite early.
I looked at it, found a confusing, and it's called it stupid.
It just didn't make any sense to me at all, so I simply discarded it.
But there was a steady influx of people who have seen the lights at a say.
So I decided to eventually have another look, and that's been a finally clicked.
That moment when you realize that a tool simply fixes all the UI issues that you had,
and that you have been developing for the last 20 years, was not exactly great.
But I had two options, either I can despair or I can learn from the competition, and
I chose to learn from it.
There's a couple of significant departures of from what get-tas.
First, history is malleable by default, and you can basically shape your commits as you go.
It's almost as if you were permanently in an interactive rebase mode, but without all of the
confusing parts.
Also when you re-write history, dependence update automatically, so if you added a commit,
all children are rebased automatically.
There is no special detached hat mode.
In fact, you often often don't even have local name branches, so you're constantly working
with detached hats, so to say.
And also, conflicts are data, not emergencies.
You can commit them, and resolve them at any later point in time.
These are just nice to have.
They fundamentally change how you think about your commits.
You stop treating them as precious artifacts, and rather start treating them at drafts
that it can freely edit.
As said in the intro, get us old, so we cannot just completely revamp our UI, and those
break all the workflows up there.
But there are some things that we can definitely steal from JJ.
The primary way to re-write history is by using git rebase, and specifically interactive
rebases.
But interactive rebases are making some tasks a lot harder than they have any right
to be.
One example is splitting up a commit.
First, you need to figure out which commits you want to split up, and let's pretend
we want to, for example, split up the commit, introduce A and B.
You would now start an interactive rebase.
This already causes the first confusing moment.
In order to edit that commit, you have to rebase on top of its parent, not the commit itself.
You're now presented with an instruction sheet in your editor, where you have to manually
search for your commit, edit the instruction from pick to edit, and then save.
You get used to it, but it's somewhat weird.
You're now put on top of the commit that you want to edit.
You have to undo it, because you want to split it up and trade to new commits.
Be careful, you stitch the first file that we want to put into the first commit, and commit
it.
The original commit message is kind of gone at this point of time, except if you know that
you can pass dash dash re-edit message equals hat at curly braces 1.
Not exactly ergonomic either.
We can edit another file now, and then we create the second commit.
And finally, you conclude the action by saying get rebase dash dash continue.
All of this requires 7 commands, editing an arcane instruction sheet, and some scary operations
like discarding commits.
And if you had other branches depending on this commit, well, they're no pointing at the
old object at you.
As you can see, the old introduced A plus B commit still exists in this history.
And is referenced by both the other branches feature A and feature B.
In Jiu Jitsu, all you have to say is JJ split, and then it asks you which changes should
be part of what commit, and what the commit messages, and what the commit messages should
be.
I kind of get why people actually prefer this workflow.
As mentioned, dependent branches would have to be rebased manually after an interactive
free base.
This is becoming a problem though, if you want to work with stacked branches.
The style of working that is becoming increasingly more popular.
Let's assume you want to build a new feature that consists of a couple of logical steps.
The traditional workflow typically creates a single branch that contains each of these steps
as individual commits.
The end result is one big merge request.
This is easy for the developer, but painful for the review, because they now have to
read through the hundreds of lines of change.
The alternative that gains more and more traction though is to have stacked branches.
Instead of putting every commit into the same branch, you create a set of dependent branches.
Each of these branches builds one small part of the bigger feature, and each of them uses
a separate merge request.
This overall of course requires more steps, but the review will now go a lot faster, because
the changes that a other person needs to review are much smaller overall.
The problem is that it makes maintaining these stacked branches quite painful.
Let's say you for example work on top of feet off to address review feedback.
You simply fix a typo in the commit message, and then suddenly all the dependent branches
will be orphaned now as they point to the old version of feet off.
You have to manually rebase feet API and feet UI.
Do this a few times a day, and you will quickly abandon the stacked branch workflow entirely.
We aim to solve these issues, and make stacked branch workflows easier with a new get command,
get history.
The goal is to have a couple of opinionated subcommands.
These subcommands do one thing, and they do it well.
Get history, we were, for example, we were the commit message of one of your commits.
It's just like get commit dashed as a man, except for an arbitrary commit in your history.
Get history split.
We'll work just like JJ split.
You give it a commit, get ass, which part should be part of what commit, and then you type
into commit messages, nice and easy.
Get history absorbed, takes all of your stage changes, and figures out automatically which
commits to apply them to, and squashes them into those commits.
All of these commands are heavily inspired by what JJ provides.
In fact, JJ itself was also inspired by other version control systems.
Get history absorbed, for example, and JJ absorbed have originally been implemented
in material a long time ago already.
This is, of course, only a start, and we plan to add more subcommands to it that make editing
your commit history way easier going forward.
Some examples include squashing a range of commits, dropping a specific commit, reordering
them, and so on.
The more important part, though, is that we don't only aim to make recurring tasks easier.
This command also knows to automatically revise dependent branches.
So let's revisit our example from earlier on by using get history instead.
We again have the same state as before with get rebase.
To split up the commit, we simply execute get history split with the commit ID.
It now goes through all the changes one by one, and asks, should that be part of the
first commit or not?
So we answered the first question with yes, and the second question with no, which means
that file a will be part of commit one, and file b will be part of commit two.
If you confirm, get will now ask you for two commit messages.
In both cases, it also knows to retain the original commit message so that you have it
as context.
But if we do another lock, you can see that the commit has been split up into two commits,
but more importantly, you can also see that all of the dependent branches, main feature
A and feature B, have been updated automatically to point to the new revert in commits.
Get history has undergone a very long discussion on the get mailing list.
The first version was posted in get 250 tier realized hack on August last year.
Since then, it has been significantly worse with a lot of bike shedding and swan to
handle stack branches, workloads way better.
The initial architecture for get history has emerged upstream, and will likely be part of
the next get release in about three months probably.
For now, it only supports re-wording, but splitting up a commit will move into review next.
More subcommands will follow, and in that context we will have a look at first class conflicts
as well.
Which are a central part of how jj works with stack branches.
I'm also very certain that the get project will have a deeper look at what else jj has to
offer.
Not all of these changes will land and get, because the design is simply different,
and as a consequence not everything that makes sense in the jj world also makes sense in
the get world.
But I'm very certain that there will be more to come.
So this has been a little bit of a whirlwind through two or through what's happening and
what's cooking and get right now.
I hope you have learned a bunch of new things, and have a little bit of a clear picture
of where the get project has had it to.
We have been looking at the chart of the sixth transition, the refibled back and to story
references, upcoming changes to improve large object support, and some of the upcoming
history re-writing features.
If you have any feedback or questions, I'm very happy to discuss them, so please just
approach me on the hallway and have a chat with me.
Thanks a lot for your attention.