Okay, welcome everybody.
Thank you so much for coming.
We're going to talk about observability databases.
All right, so if you call Alexander's talk,
you've covered some stuff that's very relevant.
What I'm going to be talking about is it was a great talk
on comparing B3 indexes and LSM indexes.
If you didn't catch that talk,
I highly recommend that you go check out the recording
after mine because it'll help fill in some of the
more technical details of what I'm talking about.
We're going to kind of step back, look at these things
from a little bit of a high level.
Talk about some of the different trade-offs in the architectures
and when you might want to use some of these things.
Okay, so first, I often give this talk
at more developer-focused conferences,
and as Patrick McAdden says,
a developer will never ask you,
hey, what file system is that?
But you're in this room.
So you're special.
You do care about these nitty-gritty details,
and that's awesome.
Quick note about myself, my name is Josh.
I am an open source advocate at Altinity.
We do hosting and support for Clickhouse,
but our lawyers wanted me to tell you that we're not affiliated
with Clickhouse Incorporated.
We're just humble open source contributors.
I know that in this room,
sometimes it's kind of hard to see the slides.
So if you want to scan this QR code,
you can just download the slides right now
and have them on your own device,
or you can have them for later.
That will be on every slide,
so it'll be available.
Okay, and with that out of the way,
let's dive in.
So first I'm going to talk about what observability is.
Really, really quickly, right?
It's about visibility.
Can you see everything that's happening in your systems?
And understanding,
do you have enough context to know what it is that you're looking at?
And that's a simple definition,
but it requires a lot of effort and a lot of data.
In my experience,
you're going to have probably about 50,
maybe even more times data for observability,
than you will have for the actual system that you're observing.
This is going to vary depending on the shape of your system.
It's just a rough rule of thumb.
But the point is,
we're dealing with a ton of data, right?
Alexander made that point when you're talking about
the trillions of rows and petabytes of data
that we have in time series databases.
Wrong direction.
Okay, so what is actually in all of that data?
Well, the bulk of it is the telemetry itself, right?
So if we're talking about open telemetry,
that means metrics traces, logs, profiles,
and, generally, just all of these things can kind of be thought of as events.
We need metadata that allows us to identify
where those events came from, right?
So that can take the form of labels or tags in
Permitius or an open telemetry.
It's called resource metadata, right?
But basically it describes the things that we're talking about
whereas the signals themselves are the actual samples
of actual values, right?
With that resource metadata,
then we can start to do things like
graphs and topologies of how those resources relate to each other.
We might want to have snapshots or point in time,
you know, views of how things looked at that point in time.
We might want to store deltas because that's a nice cheap way
to store the changes from those snapshots.
And then finally, we're going to have some really simple stuff
right that you can throw in like a sequel-like database
if you wanted to, that's your configuration.
Your alert rules, your users, and your dashboard configurations.
So what do we actually need our database to do in order to do this?
Well, most importantly, we need really, really fast ingestion.
In the worst case scenario, if your ingestion pauses or falls over,
it can actually cause the application sending the telemetry to fall over.
If you're in that scenario, you have done something wrong,
but I have seen it happen.
So that's bad, right?
We don't want that.
We need to be able to ingest the data really, really fast.
We also don't want to lose observability data even if the system doesn't go down.
We have so much data that having efficient compression and storage
is very important for a cost, right?
Typically observability data is oriented around time,
so we want an easy way to manage time oriented data.
And we need real-time right means different things to different people,
but we need something to proximity real-time analytics.
Peter Marshall, I'd like to say anything you can do with a group by,
that's what analytics is.
So what you're saying here, right, is that it's about aggregating
over billions and billions of rows.
It's about being able to do operations like some
or finding specific percentiles or means over tons of data
really, really fast.
That's really what we mean when we say real-time analytics.
So we need fast multi-row analytics.
If we don't know exactly what we're looking for in our logs,
we might need full-tech search.
If we have a really good organized metadata,
like because we're using open telemetry correctly,
then we need good tag label search, right,
which is easier to index than full-tex,
but still requires an index.
If we're doing a learning, we definitely need to be able to read
the last sample across a huge range of time series really, really fast.
And we probably don't really need to be able to update the data, right?
We can kind of treat all of this as an appendomly situation.
There are some use cases where you might need to update your observability
to, we're not really going to talk about that today.
Okay, so databases come in a bunch of flavors, right?
We have OLTP, that's Postgres, that we're all familiar with.
We have OLAP, these are analytics databases that specialize
in kind of what we're talking about here with the real-time analytics.
We have time series databases.
Alexander gave us a great overview of how those work.
And then we have like combination search analytics databases.
This is not like a scientific textonomy, which is sort of how I break things down.
And in reality, oh, right, so we have a couple of examples of each of these,
right, so Postgres is our OLTP.
Cassandra is another OLTP database that we'll talk about a little bit,
although in this room I'm probably not that qualified to talk about Cassandra.
And then we'll talk about elastic and open search,
and Prometheus, and finally, Clickhouse.
But tech armies are hard.
A lot of these databases, kind of borrow tricks from each other,
and can kind of do all of the things.
So if you need to shoehorn a use case into something that was meant for a different use case,
you probably can.
But if we're talking about like archetypes in a two-year space,
this is kind of where I put things.
And I'm a little less sure about solar and Cassandra, so those are kind of great.
But I'm pretty sure on these other things.
This is where they fit in.
All right, the storage on disk is like the most important part of the database architecture.
We heard a lot about this from Alexander, but this is always going to be the bottleneck.
This is always going to be the slowest part of your database.
So really all of the architectural tricks that we're talking about is like,
how do we avoid reading more data than we need from the disk?
How do we lay out the data so that it can be read really, really quickly and sequentially?
We think about Postgres, it does not optimize for this, right?
It uses heat pages and a commit log.
We have time series blocks, that's like Alexander's talk,
goes into great detail kind of on how that works.
And then we have a mutable parts or segments.
These are just like the different ways that we can sort of store our data on the disk in files.
So if we wanted to make a postgres, we'd use a right-of-head log, right, a wall.
We have heat pages and a multi-view concurrency control.
That's how Postgres achieves the multi-threading, which is awesome.
And we have B3 indexes, which Alexander talked about how slow those are for this,
right?
We need, we need, we need really, really fast ingest.
That's going to knock a B3 index over because it's way too many right operations.
This is what Postgres and my school do though, right?
So they're optimized for this update and upset use case with strong asset guarantees.
And I'm sure you've run into this, but like scaling horizontally,
is really, really hard.
And so then you end up with these mission critical, massively, vertically scaled,
transactional databases, which is fine, but it's not great for analytics use cases.
So let's talk about how we actually do analytics.
And we also got a little bit of a preview of this from Alexander, if you're here.
Log structured merge tree, right?
So this kind of emerged in like the early 2010s.
It is the underlying storage for a bunch of databases.
It's also one of the ways that Kafka can persist data, right?
Basically we have key value pairs that are buffered in memory,
and then sorted before they're flush to the disk,
which allows us to have more efficient file structures on the disk
that are already optimized for the reads and the queries that we're going to do.
It's pretty nice.
This diagram comes from a blog from Ben Software.
It's a great blog post if you want to read more about like the details,
the nitty-gritty of how log structured merge tree works.
So one implementation of log structured merge tree is a patchy-lucing,
which is an engine that is used to power a number of databases.
You probably used it even if you don't know that you've used it,
because it's like built into a bunch of like commercial observability platforms,
as well as these open source databases,
Cassandra, Alaska, open search, and solar.
Among others.
It's based on that third storage style that we've talked about a couple of slides.
It go right in mutable parts and we've seen they're called segments,
not parts, but it's essentially the same concept.
And then we have this background compaction process
so that once that stuff is getting written to disk,
it's further optimized into these larger and larger files
that are easier to query.
So it's like this, right?
You have a batch insert that creates a new part or a new segment,
and those parts get merged and background into larger parts.
Sandra, kind of a great example of this architecture.
It's really, really good at storing wide events, right?
It doesn't, it's schemaless.
It's a source key value pairs, like a document database.
So we can kind of just throw whatever wide events we want at it,
and we have some semblance of OLTP, like transactional guarantees,
which is cool, but it's still not fast enough for most of us
or really use cases.
Vector engines and search kind of take that
losing engine, right?
And that, those are immutable segments.
And we add some tricks so that we can find the data really, really quickly.
We already have the way to ingest it really quickly.
Now we need to find it really, really quickly.
So those tricks will dive into each one of these in order.
Inverted indexes.
I think there's a talk after this one that's going to go in
to like more depth on how Inverted indexes work.
Or I might have been before this one, but check that one out.
I will talk about Bloom filters, which is a way to eliminate data that you might not need to read,
and approximate nearest neighbor, which is for vectors.
So Inverted indexes really quickly, right?
You just have basically you flip the script.
So instead of your keys pointing to your data,
your values point to your keys, so you can say,
if I'm looking for cat, which documents contain the word cat,
pretty straightforward.
Bloom filters are a little more complicated.
Basically what we do with the Bloom filter is rather than having all of the different values in our text
that we might want to index and pointing to the keys that contain those things,
which can write like that's a pretty dense index.
That's going to be a huge thing if you've used the last sticker open search,
to build these indexes.
It takes a lot of time, and it's a very big thing that you have to store.
A Bloom filter is much more compact.
And basically what we do is we take a string,
and we run some arithmetic on it to condense it down into many fewer bytes or bits,
and it's glossy, right?
So we don't know necessarily if we use a Bloom filter on a piece of string,
we don't know necessarily that that string does appear, right?
Like if we're running a query for Otalk Lecture prod-01,
then we can look for any row that contains 31,
and know that might contain Otalk Lecture prod-01,
but it might also contain Otalk Lecture prod-10.
So we then have to check all of those rows that had 31 in the index,
and check them, right?
You have to use specific settings for the Bloom filter to get this exact result,
but yeah, that's how Bloom filters work.
They are much sparsher than an inverted index,
which has the benefit that you can maybe fit the entire thing in memory,
which is going to make it a lot faster because then you can go to the memory
and figure out which parts of the disk you actually might need to read.
But you will, of course, have some mild over scans because you have not,
you're not sure about your positives.
Then if we're talking about vectors, right?
Like dog is an easy thing to search for,
but if we want to search for the concept of a dog,
then we're starting to talk about vectors and embeddings,
vectors of course exist in this multi-dimensional space,
and so in that multi-dimensional space,
we can calculate nearness,
and we can sort of define which vectors are nearest to each other
from that we can create neighborhoods,
and then we can create layers on top of that that the query engine can walk
in order to find vectors closest to a specific point in that vector space.
So that's how a approximate nearest neighbor works.
For me, theus, and other databases based on the For me, these architecture,
which includes Loki, sort of follow different architecture.
So we have this like time-series database architecture,
which is a pendulumy, right?
We have these TSTB blocks, and they're a pendulumy.
So we just always take the last sample and pop it on the end,
and our samples are all at the same type,
and we have one file per unique time series,
so we don't really have to store any extra metadata,
right? We just put the sample into the correct file,
and it is where we need it to be.
This seems really simple and awesome,
but there's a problem with it.
Your number of time series is going to follow this formula,
which can explode really, really fast,
unless you really, really carefully guard your cardinality.
That requires everybody sending you data
to be on sort of the same page about what they're going to call things,
and how much cardinality they're going to use up, right?
We don't really necessarily have an easy way to control this
in large organizations, and so then you can deal with time series databases.
They're very difficult to scale and keep maintain performance.
But there is a nice thing about the way that that data is laid on on disk,
if we want to read, like if we're trying to draw a graph on a graph,
you think about what we just read that one file,
and it has all of the samples in it.
That's a really, really simple read operation.
It's really fast.
And also another benefit of having the same type all stored next to each other
is we can compress it really, really easily,
because it's the same type.
And so compression algorithms work great on column-oriented data,
which is sort of what you can think of a time series database,
block as being it's like a single column in a column-oriented database.
Clickhouse combines that column-oriented nature,
and the log structured merge tree, so instead of storing key value pairs,
like in traditional log structured merge tree,
we are now storing entire columns.
That's the main difference, right?
So we still have the background compaction.
We still have temporary buffers, and clickhouse is actually on disk,
so the data is immediately readable and flush to disk.
There's no memory buffer, but you do get all of the benefits
of the background compaction and the merge tree engine.
What this means is that say we have a table that has
1009 columns, it's 59 gigabytes of data,
in a row or into database, we would need to scan all of the data
to answer a question about every row.
We even will only care about one column.
If we care about every row, we have to scan the entire table.
That's a lot of data that we have to read from disk and transfer, right?
So immediately going to a column-oriented database reduces that
from 59 gigabytes to 1.7 gigabytes, so dramatic reduction,
just from going to a column-oriented.
But then because that data compresses even better,
because it's column-oriented, we get down to 21 megabytes,
and then because we can process this with multiple threads,
each thread only has to read 2.6 megabytes instead of 59 gigabytes.
So we're at a fraction of 100th of a percent of the amount of data
that we actually need to read, which is how we take queries
that take thousands of seconds and run them in under a second.
Here's kind of a view of how this looks in Clickhouse, right?
I'm kind of trying to visualize that these are entire columns,
instead of key value pairs.
The data is immediately available,
and eventually we get to this fully marked part
that's going to be really, really efficient,
and we might ship that off to iceberg or something.
Clickhouse uses those other indexes, but it uses them as skip filters,
so it first needs to find that I did some of the data,
and it makes heavy use of sparse indexes.
These basically just point to a large block of data
that's between two points in an ordered key,
and usually when we're talking about a observability
that ordered key is going to be time.
So this allows us to really quickly find, you know,
sections of data that we care about,
and then we can apply those other tricks that I talked about earlier
from open search as filters on this
to decide which rows we actually need to look at further.
All right, so I think I'm getting close to time,
so which ones should you choose?
Well, when I was first working on this talk,
this is what I was running in my home lab,
so those are pretty good choices,
and they'll also scale, right?
Although, permethias, there are some alternatives
that kind of scale better, and maintain the same API,
and I'll talk about those.
But yeah, this is a great starting point.
In our own cloud at Altunity,
we use Victoria metrics instead of permethias,
it's much more performant.
I'm also running this in my home lab now.
I've updated.
It kind of combines some of those tricks,
like the architecture permethias,
with some of those tricks for merge tree,
as Alexander told us all about that was awesome.
Loki, very, very similar to permethias.
It's just an append only thing.
It does not have full text, it does not have full text indexes,
like inverted indexes for full text search.
It does support full text search,
but then that requires standing the whole table,
which is slow,
but if you're just searching on labels,
Loki's really, really fast, really, really simple.
We also use this in our cloud.
Honorable mentions,
these are things I didn't have time to talk about,
but cortex from the CNCF.
Thanos, also from the CNCF,
Memear, which is graphanas,
sort of wrapper of cortex.
Times scale DB is a plugin for post-grace.
Solar droid, right?
These are all viable options.
Follow some of these tricks.
I didn't have time to talk about.
What else should you use?
Well, that first example that I gave, right?
It's more nuanced than just one recommendation.
I'm a huge fan of using what you have until it breaks,
so if you're just trying to keep some small data around,
and not right, like do full observability,
well, then yeah, post-grace can work for that.
If you really need full text search,
and that's your main use case,
there's really not a better option than a elastic and open search.
If you want to use one database for all of your use cases,
traces, metrics, logs, right?
Clickhouse is a pretty good choice.
It can kind of suit all of these use cases pretty well.
And if you need real time analytics and wide event analytics,
clickhouse is also a really good choice for that.
If you need lots of last sample reads,
e.g. you're doing a learning.
Then a time series database, like for me,
the use of the Kramer matrix is going to work really, really well for you,
because it optimizes for that last sample read use case.
And if you want to kind of overview of everything that we've just talked about,
here it is.
This is my last slide with this QR code, if you want to scan it.
And we are hosting Meetup on Monday night at the comments up.
So if you like this talk and you like talking about this stuff in much more depth than
then come join us.
I also have a bunch of stickers on the table over there.
And thank you.
And happy Creen.
Thank you.