I'm going to talk about butter fest today, so two questions for you.
Who here has a laptop with them, just right now.
Show of hands.
Okay, leave your hands up.
If you know what the file system is on your laptop right now.
I don't think anyone put their hands down, so my friends.
I'm going, like the very brief outline for the talk.
I'm going to talk how my company cuts our storage cost by 74% over the last 18 months of onboarding butter fest.
And how we productionize it in the company and in the Google Cloud platform.
Just a little bit of business context.
Our coronavirus is a company selling observability.
Softer is a service.
It's a software observability, software is a service, which is selling among other things telemetry.
So metrics.
We do store a lot of time series data.
And we have some big customers that we run thousands of database service servers per customer.
And each disk of the database server is at least one terabyte in size.
So there are usually more than a terabyte, but that runs to many or some petabytes stored in Google Cloud platform.
In mid 2025, all of it was on EXD4.
At ten at some point, we started thinking, why, well, what could better of has to do with it?
One, it's compression.
It can save a lot of disk space.
And compression is fully transparent as I'll show you in the demo soon.
And check sums in butter fest can save on CPU cycles as well and application complexity.
Just to ground this into reality a bit, I'll show you the demo of in line of butter fest compression on my laptop.
So I have sacrificed, I have sacrificed 256 of my disk of a partition for butter fest.
So there is a file system there, I'll just mount it.
Right, so here, I just know, I use the wrong arguments.
You'll just see why.
So I am compress, I am mounting the partition for butter fest on a directory and I'm telling butter fest to compress it with this standard compression algorithm.
And I have a dump of English Wikipedia.
First billion bytes of English Wikipedia snapshot from about 20 years ago to be pedantic.
It's one gigabyte in SI units and in slang that would be 954 megs.
So what does this English Wikipedia look like?
It is an XML document, so a lot of text.
There is useful stuff in this XML document.
Let's try to search for a string.
We can see the search works and who knew until today that was them is free open source software developers European meeting.
Kind of new, but now I do.
No, it's from Wikipedia from 20 years ago.
So important things here, we can see that the file is almost gigabytes, like it's a gig on off data.
We can read it with cat, grab, etc., we can open it with a text editor.
You know, just a regular file, but what's interesting about this file, that it is stored compressed.
So we can see that it uses 340 megs on disk.
So about a third.
This is like a 1 terabyte 2 terabyte disk.
If I only start English Wikipedia or just Wikipedia, I would be able to use like to start 3x of them and would not have to change anything else.
So it's transparent compression.
The applications that we did do not see that they need to uncompress it or compress it.
For regular desktop, since all of my laptop is on butterflies, we will see that it is compressing not only text data,
but things like program data.
So we will get back to this in a minute or so.
So we just show you the demo where we show transparent compression on Wikipedia.
It was around 60%.
Yeah, around 60% compression.
And that was 65%.
35%.
So that's 65% compressed.
So we used 3rd of the disk.
So this is exactly when we looked at butterflies over a year ago.
We initial tests with the production data in Corona server, with the telemetry production data.
Show that just if we move the data on from EXT4 to a butterflies with transparent compression,
we cut the disk off by a third.
Sorry, by 2 thirds.
So we are down to minus 65%.
This was huge.
But once it's a lot of petabytes multiplied by, it could be huge savings.
But immediately what jumped into my mind is the bad reputation from the past.
If you have, if you saw my speaker bio, I tried to use butterflies 15, 10, 15 years ago and it ate the data.
And it was, I still remember that.
But I have used butterflies over also recently and it didn't.
So the first question is, okay, the savings are significant.
They're interesting, but the reputation problem is, is there.
And if you look on the internet even today, there are people reporting data loss,
or just unable to configure it, or other problems.
It's all over the place.
But there was one specific talk in OSCon 2020, which was documented by Linux Weekly News,
by Joseph Batzik from Facebook, a kernel developer from Facebook working on ButterFest.
And he was explaining how Facebook maintains and uses ButterFest on their millions of servers.
And this talk was really a confidence booster that this is something that could actually work outside of my laptop,
you know, broader for the company.
So with the projected possible savings and this anecdote and talk from Facebook,
we decided to have a better look to evaluate it actually.
See if it could work and reduce our bill.
So once we started looking at that, there were a few known unknowns that we knew that we'll have to go over.
First confidence and reputation, I just talked about it.
The talk mostly mitigated that while talking to my peers and the company.
ButterFest is infamous.
The other thing is ButterFest is infamous about reporting this space.
But how much there's this space remaining?
Because it manages the way it lays out files differently than other file systems.
But this can be mitigated and it's pretty well documented.
If you're interested, you can look up those terms, background reclaimed threshold and dynamic reclaimed.
It's fine.
You have to know what you're doing.
You have to do it differently.
But you can know how much space you actually have remaining.
Also, there was no mention of ButterFest at Google since coronavirus on Google.
This is something that needed to be looked at.
It could be that it actually did not work.
Somebody tried and said, no, this is not working on our offering.
Or nobody just implemented, nobody bothered.
Another thing we've been using EXT4 for the decade on this database, we knew how it behaved.
We knew it performed well enough.
Since ButterFest is completely different, we didn't know how it will react on a database workload.
And there were issues we'll get into that.
There were, we knew that there are unknown unknowns.
Something that we didn't know that will not work before we start the project.
That happened to you.
So here, so I told you about the known unknowns.
We figured that we figured them out.
And then I'll give it to you that I think are the most interesting ones.
The title of the one surprise is Reclaim.
The ButterFest term for defragmentation.
It's causing a massive IO spike on large delete.
My colleague Fastest gave permission to share this graph from a production outage a few months ago.
And this is a commit log Q size.
You can see it grew a lot.
And this was a user customer visible outage.
And why did that show that database clusters?
One line is one database server that it was overloaded.
It was not able to flush rights on time.
So it was not accepting.
It was delaying rights.
And the problem here, the root cost for this, is Rida Head.
Right.
Yeah.
No, for the last one that was not Rida Head.
That was balancing a Reclaim, which I talked before, which we have toggles now.
The other one is Rida Head.
So this is a screenshot from another outage, which I don't have chat sessions for, but we have database graphs for.
This graph on the right is the database, the P99 database latency, which is usually a few milliseconds.
And at some point, they spiked to multiple seconds.
So this was user visible query degradation.
They were not able to see some of the queries.
And if we looked at the disk graphs, we could see that the disk throughput,
but was flattened out at the top.
So this was overloaded.
And why is that?
There are two Rida Head settings on Linux, conveniently, named the same way.
Who, anyone here can tell the difference?
No hands?
Oh, they're going hands?
Yeah, shout out, I'll repeat it.
Yeah, they said they're different places.
Yes.
So the difference is 32x.
So the first one is block level.
So if a kernel sees that you're reading a block,
it will pre-fetch 1208 kilobytes,
kibibytes from the same, you know, from the disk.
But butter effect is smarter.
If you're reading a large file, if you start reading the large file,
it will pre-fetch up to four mibibytes, megs,
from the point that you're reading.
So this alone could cause 32x read amplification.
So you're reading, even though you may not be using it,
but it will still fetch it from the disk.
And this was exactly what we saw in the graph.
It was pre-fetching a lot of stuff for the files that it didn't actually use.
So once we, this is a configurable toggle,
we moved the four megs to 1208 and all as well.
I'll skip the timeline.
So we started the project in June,
and we finished it at the end of last year.
And takeaways.
We, our disk savings,
instead of 65% predicted,
where 74% butter effect is now available
in upstream Google Cloud Platform by the infrastructure
that runs Google managed Kubernetes.
We're planning, we as Kronosphere are planning to go full upstream
in the next one or two months,
and you know the sites Facebook,
who are very open about their butter effect use.
Now you have another company to refer to
that are using butter effect in production with quite a lot of data.
Special thanks to Boris Borkoff and Yots of Bicycle
for reclaim, for managing the reclaim,
making our lives easier to manage the cluster,
and for spreading the word.
The talk was a game chain.
The talk that I mentioned before
was a game changer to adopt it.
Thanks to all of butter effect maintainers,
and Google Cloud Platform,
internal champions,
who champion butter effect and made it work on their end.
That's all from me.
Thank you.
Yeah, we still have a few minutes for questions.
Last time I checked butter effect,
as usually got a lot worse performance numbers
in terms of throughput,
compared to X4,
what's your experience with that?
Yes.
In our workload,
that was really favorable,
and there was no visible difference at all,
or latency and throughput.
And what's the size of your Nickstore,
so you ran the command like...
Oh yeah, yes.
60%.
So 40% savings.
Hello.
Yeah.
Thank you for your talk.
So I'm a big ZFS fan,
and there's this thing called
opportunistic compression,
where it's not going to bother compressing the rest,
while writing.
If it doesn't detect any savings,
I don't know if butter does that as well.
Yes.
So it looks...
The question...
Yeah.
It will look at the beginning of the file,
and it will determine
whether it's work compressing.
And it will either do it or not.
There's another one.
Behind you.
Have you ventured into
butter effect rate support with rate 5 and 6?
We did not find a need for this,
so I did not test it.
Hi.
How are you managing FLOCated files
that are uncompressible,
usually created by database,
record, and other?
So you asked about FLOC?
Yeah.
It's okay, no butter effect.
So in our database,
since I'm in the database team,
we control the code that writes to the disk,
so we don't FLOC here.
That's it.
I think you can also turn it off,
but that could be...
Yeah, that's it.
What is main reason why it
chooses butter effects for servers,
for server uses,
instead of Z-Vets,
which is usually more popular in...
It's much easier.
Yes.
The question is why butter effects
not Z-Vets?
Basically upstream support.
You get it in the kernel right there.
There's a lot of tooling already in place for butter effects,
for those user space parts,
like the container storage interface.
It's a much tougher sell for a big,
for a larger company to use that effect
because of the licensing and non upstream status.
With butter effects,
it's pretty straightforward if you can prove
that it will work technically.
Are you using the snapshot features of butter effects at all?
The snapshot and the revert features
are any of the more interesting bits
from a normal system usage of it,
or is it just file system that is?
It's right now.
No, right.
Yes.
We have just finished moving everything.
So we're not using the advanced features.
Just check something.
We have disabled some of the application checksums,
which we knew we could disable,
and everything else is just a dumb database.
So I'm wondering on butter effects,
the copy on the right behavior,
is it has the reputation to not work very well with databases,
and I was wondering what database engine do you use on butter effects?
So we use M3DB, which is,
which was open,
which is our fork of open source M3DB.
So we wrote it,
and then because of the nature of the database,
we don't use a lot of splicing in the middle of the file.
That's why it's a really good tip.
But I acknowledge it doesn't work for every database,
but it just works really well for us.
You said that the checksumming
will simplify your applications.
Yes, so we don't do it.
That's it.
In some cases, we don't have to do the checksumming,
so it saves on the CPU.
Well, the checksum is still happening just in the kernel,
but it's not happening just frequently,
because in our case it's easier to control.
Like, we don't have to do as much checksums as we need it to,
because we know that what we wrote, we get out.
As long as you know before it goes out of the network.
You are planning to do the, the, the, the application.
The application, what, can you elaborate?
You know the feature of the application.
So for the header, for metadata.
Do you see a advantage for,
Yeah, in, in this case for Google's persistent disk,
there is no need to do duplicate anything,
because they're reliable enough.
If we ran off on our disks,
then that would be a different thing.
Hi, thank you for your talk.
Um, do you use the send and receive features of betterFS?
Right.
I can't hear or see here.
Sorry.
That, thanks for the talk.
Do you use the send and receive features of betterFS?
We tried them, but for our case,
the CPU overhead was prohibitively.
It was, it was faster to do something else.
Like, it didn't help.
Through put increased, but it still took a lot of CPU.
I've been prepared from the next speaker.
From the glass.
Thank you very much.
Actually, I'd be.