In a previous life, I used to write enterprise firmware for HDDs and SSDs, and now it's my
life mission to remove those from all systems I touch.
This is my book, the Tori DocTax. Stick around, there'll be more later. This is my dog
turn. So today, I'm going to talk about why our company is building a database. What
wasn't working with the first database we built, how we decided to re-architect it,
and then the open source projects that we built it upon.
So first of all, why are we building a database? There's a lot of really good databases
out there, why don't we use just one of those? Well, there's a lot of reasons. One of
them being we didn't want to be tied to certain licenses that might change out from under
us, or billing that might change, it became a business decision. But we also thought,
because what we're doing is a little unique, not all of the off-the-shelf databases fit our
needs. We're building a database for continuous profiling. So continuous profiling is my
too close, sorry. Is the process of capturing stack traces off the CPU, storing those in
a database, and then allowing users to query back that data over time? This allows them
to answer questions like, why is my application slow? What is the most expensive function
across my entire infrastructure, or why does this user have a worse experience on our system
than any of the other users? So the goals of our database were we needed to be easy to
operate, we're really small company, we don't have a huge team, so it needed to be simple.
We needed to be cost-effective at scale, we are trying to build a business, so we needed
it to scale, cost-wise with customers. And then we really wanted our storage and compute
to be decoupled, that way reads, don't affect our right path, then vice versa. So this
is our first attempt at a profiling database. It was a database for stacks. So this means
is we check the stack trace on the CPU, we record all the functions we see on that
stack trace, and we record the timestamp in the values that we see for that. So this gives
us a really specific data shape. The data is repeated, we see the same functions called
over and over, they're deeply nested, so function A calls function B, function B calls function
C and D, and so on. The data is also immutable, once we've recorded a stack trace, it doesn't
really make sense for user to ask to update it, because it's just a snapshot of what happened
on your CPU. It's temporal, so we're just capturing the data at a specific time. It's
also non-relational, one stack trace isn't necessarily for it to another. And importantly,
it also has dynamic columns. So you might be more familiar with this as known as labels
in the metrics or log world, where you can label your data sets, where it came from, what
runtime it was using, what user ID is attached to the stack trace, things like that.
And the last thing is not necessarily a data shape, but how the data comes in. This type
of data is really right dominated. You might have a cluster of 1,000 nodes shipping profiling
data to you, but you probably only query that data once a day twice a day. It's user
driven, so it's just going to be naturally more right dominated. So this lens itself really
well to call in there, and thank you for your Matt Sopel for pointing out Apache Aero. That's
typically half the story. Apache Aero is really good for that transfer format, the in-memory
format. But then there's its companion piece, the file storage for it, is the de facto standard
being Apache Parquet. So this stores all the data in calendar format, and we get really good
compression out of it, which is really important for our repeated nested immutable data.
So this is where we came up with Frostyby. It's written and go. It's backed by Aero and Parquet.
It was built on top of an LSM-ish tree. I say LSM-ish because, again, our database is immutable,
so we don't actually need to deal with updates or deletes. So it basically just looks like we
have a bunch of Aero records in memory as we receive them. Once we accumulate enough of those,
we compact them into Parquet files. Once we accumulate it into those, we compact those Parquet files into
one just big chunkier Parquet file and ship it off to object storage. So great. We had a database.
We built one for the enterprise scale. We basically sharded out a bunch of Frostyby instances.
Rights came in. They get split across a bunch of instances. Reads would just fan in from those
instances, and then we used preemptible instances to keep our costs really low. So this is great.
We had a database that was easy to manage. It scaled, and it was working. And then we had customers,
and they ruined it. So our customers ended up growing bigger. They started using more and more of
our database, which was good. But we got to a point where we were unable to write out our memory
in time for preemption. So we had preemptible instances. We'd get notified it was going to be shut down.
We tried to write it out, but we had so much data in memory that we couldn't get it there in time.
We could have solved this by scaling out the number of instances further, but now suddenly our cloud
bill just skyrocketed and we're not accomplishing our goals. So we reached into every database
to open a tool kit, and we got it right ahead of log for durability. So now rights are written to the
right-head log on a persistent disk, and then they can be acknowledged to the clients. It's great.
We no longer need to write out our data when we're preempted. We can just shut down, come back
up, replay the wall, carry on from where we left off. Everything's working again. Now we've coupled
compute and storage a little bit, but it's not too bad. And those damn customers.
So because we're replaying a wall, our customers noticed that when that was happening,
their data was no longer available. We couldn't answer a query because we hadn't replayed it from
the right-head log. All right. Well, fair enough, we have to go back and start replicating the rights.
So now each of our frosty beinsons sees a copy of the right, technically we're
triplicating the rights, but now when we're replaying the wall, customers can read their data because
they'll just get it from a different node that isn't currently replaying. Great.
Happy customers again. We have availability. We're not losing data. Okay, but we did just increase
our cost a little bit because now we have three times the data footprint. All right. Well, it's working.
Except for those damn customers. Now we have long replays caused because our customers,
footprint grew so big on the nodes that the nodes just become really out of date after the replay.
So if a replay took 10 minutes, that node wasn't receiving rights for 10 minutes. And so now it's
10 minutes behind the other nodes and can't answer queries again. So yet again, we have to make changes
to our system. And we had replay coordination. So now when a node comes up, it replays it all.
It talks to the other nodes. It says, hey, how far behind I am, they say, these are the rights
you're missing. You're this far behind. They send them over. Great. The node replays those rights
from the other nodes. And now it's caught back up and is a fully functioning node in the system.
All right. We're back to a working system. But if you haven't gotten the just to my talk,
it's those damn customers again. So some of those customers get so big that it created really
complex failure scenarios. So one of the things we saw was we'd have a node that would
go down, come back up, replay its wall, request a bunch of data. It took so long to replay its wall.
It got really far behind. The data requested was too much for the node to handle. So it
umkilled. Kubernetes reschedules it. It comes back. Now it's further behind. And you can see where
this goes because it requests more data, crashes, cycles. It's a vicious cycle. Often what would
happen is that node would be doing that so long. Another node would go down and then it would just
create this cascading failure or something there whole database was down. So one of my favorite
systems quotes is every system is perfectly designed to get the results that it does. So we had
perfectly designed our system to be challenging to operate, expensive at any scale, and we had
hard dependencies between storage and compute. We're awesome. So clearly none of our architecture
goals were being met. It was time for a new architecture. So this is where Apache data fusion comes in.
This is the first thing we reached for. We learned when we were making frosty be that creating
your own career engines really hard to get it right. It's really hard to get it fast. And we
really didn't want to invest all of our time again in building that. So we reached for something
open source. Data fusion is really fast. It's written in rust. It's calling it by default. It's
very extensible and it works with mini table formats out of the box. So it didn't really lock us
into what the rest of our database could look like, but it was going to give us really free
performance. So we were really excited about that. So if we decided to move forward with data
fusion and writing our new database in rust. The second thing we reached for on the right path
was Delta Lake. So Delta Lake was donated by a Databricks, a open source community. It's an
open table format, much like Apache iceberg. It provides acid transactions, really fast writes,
quick reads through file pruning and schema evolution, which is again really important for our
database because of the labels we see we don't know the scheme of beforehand. So great. So our
database looks like this right now. We have clients writing arrow. The arrow gets written to parquet
and is object storage and then committed to the Delta Log. The only problem with this is this is a
rather expensive architecture. Remember, you might have thousands of nodes each of those represented
a client all writing to the tables. This creates, you know, one part k file for write and then a
commit for write. So huge write amp become really expensive. Most cloud providers bill you for every
right you do. So kind of an expensive database operate. So we inject a buffer writer. So because
it's an observability database, usually you don't need to see your data within sub-second intervals.
So basically what we did is we no longer acknowledge or write to the client until it's buffered
in this buffer writer and then is flush the object storage. Once it's fully committed to log,
then we'll acknowledge the client. So usually it's however long it takes up to fill a pre-configured
buffer size or one second however long it takes. So there's a one second delay in being able to read
the data you write, but clients don't care that they can't see the last sub-second of data.
So great. This is a lot less expensive to operate because now we have larger files and fewer
writes to the system. The problem with this is if you notice we have lots of clients from different
customers all being shoved into a single table. So now if we have a really big clients, it can
affect the reads for other smaller clients because you have to first sift through that big
client's data to get at the data you want. So now we have this weird paradigm where one customer
can kind of ruin the query experience for other customers. So this is where we add a compactor
and tenant tables. So it's just Delta Lake again, but now once a customer hits a pre-configured
size in the primary table, we basically slurp all that data up right a new bigger part K file in
a different Delta table that's just for that tenant and we keep really good performance. So now
when a read happens, they just have to get their reads from the primary table which is kept
rather small and then their reads from their tenant table. So great. This is working really well.
It's really simple to operate. You'll notice we have very few moving pieces. We have no disks.
Thank goodness. Everything's working great, but one thing we noticed and one of the cool things
about building a profiling database is you get to profile your own database. So as we were doing this,
we're learning, hey, we're spending an inordinate amount of time converting par k to arrow every
time we need to move data or read data. There's a huge serialization cost. We also noticed it
with Delta Lake that the normal commit log is all written in JSON and sure JSON serialization can
be fast, but when you do it a ton, that ends up being a huge footprint for your CPU. So shout out
to our friends at SpiralDB for their Open File Format called Vortex. So Vortex offers zero copy
reads for Apache Arrow. So basically we no longer have to serialize the data when we write all our
arrow records to files and we don't have to desialize it when we read it back into memory. So we actually
replaced all the JSON files as well as all of our PAK files with Vortex. This gave us faster queries,
your copy reads, which gave us lower memory overhead. And then one thing we're really excited about
is it's an extensible file format so we can bring our own encoding schemes. We haven't done this yet,
but there's a lot of room to grow there in terms of career performance for profiling data.
So now we have an easy to operate database. It's really cost effective at scale. The storage
and compute are completely decoupled. Overall from our FrostDB database to this one, this has got
us a 50% reduction cost. We've increased, the decrees are clearly in say by over 200% and we no longer
have data unavailability. So big shout out to all the open source projects that made this possible.
We literally just built on top of all these things which is pretty cool to go from zero to a database
in just a year's time with these awesome projects. Last dog text. All right. Any questions?
Yeah, exactly. Yeah. Yeah. Yeah. So the question was, did we consider other table formats?
So yeah, we looked at basically the big three, which is, you know, iceberg, Delta Lake,
and that third weird one I can't remember the name of Kudi. Thank you. What a weird name.
Okay. Yeah. So we looked at that and one of the things that really stood out for Delta Lake
was the way it handled rights because our workload is so right dominated and we need to get really
fast rights in there. And because we pay our cloud provider for every right we do to object storage,
we wanted something that has the smallest amount of right amplification and that can do rights
the fastest. So it's typical Delta Lake right only requires you to write a single commit file.
So it's a single right potentially with the checkpoint. But it's a single right for most rights.
So that was really advantageous for our type of database. Whereas Apache iceberg is usually like
two or three different rights because you have to update manifests and then the commit version and stuff.
So that was why we picked that, um, say that again. Yeah, I think it, I think it does make a difference
because that's, sorry, yes. So the question was, even with the buffered rights in front of
the database does it make sense between which open table format and we think it still makes a
big difference because it's still one right versus three every time we flush that.
Any other questions? All right, in the back, you got to be loud.
Yeah, yeah. So there's a lot of cost factors in our old database. So one of them is we had to
triplicate the data. So now we have all of these extra nodes that we have holding on to this data.
We had to add disks to all of those nodes because we had a right-to-head log just cost money.
So we had to do with that.
Those are kind of our big primary factors was our compute and our disk storage. I think I'm probably
missing some other things. I think our networking was also a cost because we were triplicating the
data across regions. So our network bandwidth actually ended up being quite expensive for that as well.
Yeah, so faster is kind of two really big components. One of them being that
data fusion was better than our goal implementation of query engine.
Maybe if we had continued to spend more effort, we could have gotten that but it was just faster.
And then the other decision, this isn't necessarily strictly related to the database change,
but it's the replacing of parquet with vortex, gave us huge career performance gains because we're
no longer spending any time serializing to arrow. So those are kind of our biggest scans.
All right, thanks a lot.
Thank you.