I am a developer advocate at Altinity.
We do hosting services and support for Clickhouse.
So if what I'm talking about to you is interesting, come talk to me, follow me on LinkedIn
or come talk to us at our booth in building UA across the courtyard.
Let's dive in.
So they were going to talk about observability in one, or all you in one as I like to say.
I'm using Clickhouse as a unified database for all of our telemetry signals.
OK, so I spent most of 2022 and 2023 really talking about open telemetry both publicly and
internally at my employer.
And usually I start with a diagram that looks something like this.
It's very familiar.
I'm sure to all of you from the open telemetry documentation or just any blog post that you
write about open telemetry.
In this simplicity of this diagram, it's really useful for explaining the architecture
of open telemetry to someone who's never encountered it before.
But it is very misleading, and it misrepresents the complexity of how we're actually using
open telemetry in our real world applications.
So one of the things that's sort of proudly placed on the open telemetry website, so
as to not, I think, compete with the vendors that are supporting open telemetry, the open
telemetry project does not include any kind of database or back end UI.
That's sort of just up to you, right?
And then we also have this piece of information from the Grafana State of Observability
Report, thank you Grafana folks.
Six is the typical number of observability tools that an organization will have to
deploy in production, right?
And I don't mean like Fortune 100 organizations here.
I mean any organization.
So what I think we actually need, if we kind of take that first graph that we were looking at
and actually just reverse it, we get something that looks more like how we actually
would want to think about observability in the real world.
Some of, you know, on the right, or sorry, all the way there, on my right, on your left,
we have some of my favorite sources of telemetry data.
You're probably using some combination of all of those today in your organization.
We then need to consolidate, right?
We want to take that and in order to process that information and correlate it correctly
in the way that we want, build the topologies that we want, build the dependency graphs
that we want.
We need some way to combine all of that information.
And then ultimately we need to store it so that we can use it for things like alerting
and dashboards and analysis.
So I think this diagram is more realistic than the one that we just had.
And there are more challenges, right?
If we're splitting up our planetary data.
Now, technically, you could build a unified observability solution with a separate
ticket-based for each signal, but you might not want to, right?
But if you don't have that unified, without that unified interface, right, without one way
to view all of your data when you have to go to different tools for your different
types of data, different signals from different sources, you run into problems, right?
We're all very, very smart and good at our jobs, but the more times you have to, like,
correlate an ID from one tool to an ID from another tool, and basically do all of the
joins in your head, just every single one of those little points of friction is a chance
to make a mistake when you're in a really stressful scenario trying to respond to a
production out of it, right?
And it's like, we can't afford mistakes.
So ideally, we should be reducing all of the opportunities for mistakes.
And instead, we don't, right, like, we have this, like, set up where we need eight different
monitors to respond to a production incident, it's kind of crazy.
We got a lot of haystacks to look through, right?
But when we look at what actually goes into those haystacks, right, like, what are we actually
storing in our telemetry databases?
Of course, it's the signals, right?
It's the metrics, traces, and logs, soon profiles and events, right?
Open telemetry is adding more signal types, it's not three pillars, and that's awesome.
We need all of these different data formats to structure the data in the way that's useful
and compressible for the things that your questions were trying to answer.
So that's going to be the bulk of your data, and that's going to be the bulk of your queries.
But there's more stuff than just that, right?
We also need the resource metadata, resource metadata, resources for those not familiar with
the open telemetry terminology.
It's really just the metadata that describes the entities, right?
So the node that the process is running on, the process, things like that.
Maybe the EC2 region that the node is in, right?
Things like that would all go in your resource metadata, and all of that information is useful
because it allows us to tie together, right, those telemetry signals, and contextualize
them and make sense of them and what they actually relate to.
Then from all of those relationships, we can actually create graphs and apologies of
right.
We can create dependency topologies, we can create topologies like James showed to two presentations
to go that show sort of the entire architecture of our application.
These are really, really useful.
At scale, when we talk about really, really large organizations, one of the hardest challenges
is just zeroing in on the data that you care about, and it turns out that if you have a topology
as sort of the front of your filter, that's a really good way to filter down to the specific
data that you care about.
Of course, if we have graphs and topologies, those are kind of expensive to calculate.
So we're going to want to store them and maybe hang on to them for a certain period of
time.
So we're going to have some kind of snapshots.
If we have snapshots, it makes sense that we might have some concept of deltas, in
order to efficiently store changes in those snapshots, and then finally, right, of course,
we need configuration.
We need users, we need the dashboards, all the stuff that actually is probably fine
in like a traditional OLTP database, whereas if you're trying to use OLTP database for
these other things, you're going to run into some scaling problems.
You're going to run into some performance problems.
Okay, so we can't use OLTP, is there a silver bullet?
Is there something we can use?
Well, no.
If there was a silver bullet, honeycomb wouldn't have to build the custom storage engine
that they have for their really cool high-currenality stuff.
We need all of these things.
Right, we need the ability to do full-text search, that's really important for logs.
We need really efficient compression, that's important for everything, but especially for
logs and metrics, right?
We need to be able to the ability to query in real-time, more or less, and get real-time
analytics on the data, as it's coming in, we need to be able to alert quickly, like, on
the order of seconds, not minutes or hours, and any variations in that data.
We need to be able to relate the data to other parts of the data, and at end of our scale,
we're talking about a metric ton of data, right?
So we need this solution to be petabyte scale.
So while there is no silver bullet, Clickhouse comes pretty darn close to checking all
of those boxes.
So I think it's pretty starting to wait.
So let's talk about Clickhouse.
Raise your hand if you are already familiar with Clickhouse, we're using it.
Wow, awesome.
Okay.
That's not the response I'm used to, but I love it.
So click out really quickly, then.
Clickhouse is single compatible.
It is extremely massively scalable, both horizontally and vertically.
And it is really, really, really, really fast, like,
sub-second queries on petabyte's of data, and ingesting millions of rows per second
is absolutely no problem.
And it's really, really powerful for observability use cases specifically.
And I'm going to talk about why that is and how the architecture of Clickhouse
blends itself to observability use cases.
So all of that data that we were looking at a couple of slides
ago, right, the metrics traces, logs, profiles, events, all that fun stuff.
It follows a worm pattern.
Right, once read many, it doesn't change.
Right, a log line is a log line, a trace is a trace.
We never need to update it.
So we need to write it, store it, hold on to it for a certain amount of time,
whatever retention period policies are, right?
And probably query it, read it a ton of times during that lifetime.
Hopefully, if you're not reading it, then, like, why did you collect it?
But that's a different question, a problem.
And eventually it'll get deleted as a part of a bucket.
But we don't ever need to update it along its lifetime.
We think about this kind of data.
Maybe we think about balance trees.
Balance trees are really great for really, really quickly reading all
of that structured data.
But they're really expensive when it comes to inserting the data,
because you have to reorder and reorganize everything
and rebuild your entire tree every time you do an insert.
Not great for tracing data where you're going to be getting 60
spans for one word press query.
So we have a solution for this.
And one solution, this is what Alaska uses
as log structured bird trees.
So this is optimized for ingestion.
And how we do this is we store key value pairs,
offered a numbering, and we keep these really short little bits of data,
and then we can order them.
And then once they're sorted and ordered,
we can flush those to the disk.
We can immediately query them and use them as part of our read
operations.
And in the background, we can compact them.
So we can make them bigger and bigger, keeping,
meaning, hanging the order so that the read operations
are still really fast and efficient, but also
making better use of storage and better use of our nodes
and speeding up those reads even further.
This is really, really cool.
This is a great blog post from Bedstepford.
It goes into more detail on how this works in a last
to specifically.
It's a little bit different in Clickhouse.
So in Clickhouse, instead of storing key value pairs
and impacting them, we're storing entire columns,
which, if you think about time series data,
or basically any type of observability data,
this is going to allow us to do the types of reads
that we need to do as quickly as possible.
There is a way to update and delete in Clickhouse.
I'm not going to talk too much about that today,
but it's possible if you do need to do that.
And then we have the same thing that we saw in the last to great.
We have these unmerged parts getting
compacted into larger and larger parts that can be
queried more and more efficiently.
This is happening over the course of minutes
in the background, as the data is inserted.
So how does this help us?
How does this data structure help us
do observability better?
Well, one thing that gives us right, we mentioned,
it makes it very efficient.
We get very fast writes, because we can just dump
whatever data we have straight into memory
and then let the nodes sort of figure it out.
It's very time friendly, right, because we're ordering the data
usually on the time stamp, right,
that the telemetry signal came from.
This format lengthens up to that ordering very well.
We get very easy clean up through TTO policies.
Again, because everything is ordered by time,
we can just drop off and entire one of those petitions
and handle our TTO.
And since that data is all ordered,
it's very compressible, right?
And that makes it very cost-effective for us.
Clickhouse also has some more features
that lend itself to observability use cases.
I like to say that you can almost build an entire application
inside of Clickhouse using all of the features
that are available.
So we have materialized views.
These allow us to transform data from one format into another.
You might use these, for example, to take open telemetry spans,
or, I don't know, to take zip-conspans
and cast them to open telemetry spans.
It's the same data, right, just slightly different keys.
You can do much more powerful things with that,
but that's a pretty common observability case.
I mentioned we have time to live, so we can build
in the expiration of data, build in policies.
We can also migrate data from hot storage
to cold storage using things like S3
or other object storage, so we can use that to control our costs,
as well.
We can use this on premise, so we can keep data
where I was talking to somebody yesterday
who keeps 10 years of their load data in some cases.
I'm sorry, but you definitely don't want to do that with data dog.
And, right, yeah, so we tear it storage.
OK, and then integrations, right?
This is SQL compatible, so that just basically
opens up the door to integrate with pretty much anything.
There's actually two griffana data source plugins.
One is maintained by an alternative.
By us, it has some nice little macro helper functions
in it to help you create time series data
out of stuff that wasn't originally time series data
and make it look really nice in your griffana dashboards.
Jager, which is a popular tracing
to trace visualization tool, supports Clickhouse
as a back end, out of the box.
You just have to say, yep, I want to use Clickhouse
and give it an instance.
There's C-Loki, which is actually evolved into Korean,
which I'll talk about in a little bit.
And there is a Kafka table engine for Clickhouse.
So if you're one of those organizations,
the already has all of your observability data
on a Kafka stream, you just point that Kafka stream
that your Clickhouse database and boom,
all of your observability data is persisted.
So we end up with something that looks kind of like this.
Oh, right.
There's an open telemetry x-forter for Clickhouse.
So this goes in the collector or in your open telemetry
SDKs.
It's currently supports traces, metrics, and logs.
Support for the other signals will be coming soon.
And so through that, then we get access to not just
like those six, like my favorite things over there.
But all of the 90 plus receivers
from the open telemetry ecosystem are now available to us
as data sources that we can pipe directly into Clickhouse.
Cool.
OK.
So more benefits.
We get excellent compression, even with variable schemas.
So we're taking all of these disparate sources.
We don't have to know ahead of time what schemas we're
dealing with.
We don't have to build that into Clickhouse.
We can use sort of arbitrary maps and JSON columns.
And then we can still take advantage of some of the excellent
compression and really fast reads that Clickhouse is going
to give us.
It'll even inside a JSON map column
turn those into column or storage parts.
We get practically unlimited cardinality.
So CloudFlare gave a great talk at Monadorama last year
where they mentioned, it talks about moving from
permitious to clickhouse for time series data.
Anyone who runs a cardinality issues in permitious?
Yeah.
OK.
No one even wants to raise their hand for that.
But I know you all have.
Low cardinality in permitious, right?
Low cardinality, everything's relative.
And permitious, it means like dozens may be scores.
OK, in Clickhouse, low cardinality means millions.
So we're just talking about completely unrelated orders
of magnitude when it comes to cardinality.
That's really, really useful, too, for this sort of arbitrary
observability data where we don't necessarily
have control over all that on the front end, right?
We're part of an SRE team or some team that's managing all
of this data, but there's a ton of other teams generating
the data, and we can't control what they do.
So cardinality is one of those things that's actually really
hard to control.
It's going to drive your costs up with some of those vendor
tools.
But with Clickhouse, you can kind of worry about it less.
Not, you still do have to worry about it just a lot less.
And like I mentioned at the beginning, right?
It's extremely horizontally scalable, both for ingestion
and for reads.
So this is great for observability.
Because we can scale up to me our own needs, whatever those may
be.
There are some challenges, though.
It's not perfect.
It's not.
It doesn't solve all of our problems, right?
Nothing does.
One of the biggest challenges that people went into using
Clickhouse for observability is that SQL is not from
through all.
And people who spend most of their time dealing with
observability data tend to really like from through all, right?
Like it's a very productive, very tourist language
for the types of queries that it handles.
As far as fair, because both ataltinity and at Clickhouse
incorporated, there is work underway currently
to bring PromQL support to Clickhouse.
So you can just write PromQL queries.
Yeah.
Wouldn't that be awesome.
And then that opens up even more integrations, right?
Like think of all the tools that integrate with PromQL,
then you can just use those with Clickhouse
makes your Grapana dashboard easier.
It makes courses integration possible.
So this is really cool.
OK.
Another challenge with Clickhouse is that some people say
that it can be overly complex for small data volumes.
This data point, this bullet point comes more or less directly
from the Clickhouse in blog, on observability with Clickhouse.
I kind of disagree.
I've spoken to, we have some customers who are dealing with tens
of gigabytes of data on Clickhouse.
And because of all of those features that we were talking
about, right, they'd find that it's worth it,
even though they're not dealing with a scale
that Clickhouse was sort of originally built for.
So you can use this at any scale.
And another thing I would say is, I've got
in 150 gigabytes of observability data out of a demo
running on the Raspberry Pi for a few days.
So what is small, right?
Everything's relative.
And then one final challenge with Clickhouse,
it's just a database, right?
It's not a complete observability solution.
It doesn't have any, you're learning.
It doesn't have any visualization.
It has integrations with all those things,
and that's great.
But by itself, it is, by no means a term key solution.
It's just a storage engine.
We do need that, though, right?
We need that for observability.
We need both the database and a back end UI of some kind.
We need a complete observability solution, right?
So here's kind of a simplified diagram
of what a complete observability solution might look like.
You've got your application SDKs, right?
Open telemetry is awesome for that.
You've got your host and node agents, open telemetry
is also awesome for that.
Bala, which we just heard about, is awesome for that.
Then you need some kind of collection sampling
and processing, open telemetry collector,
sort of the default choice for that, just kind of make sense.
From there, you're going to go to maybe directly
to your storage, or maybe you're going to use some kind
of reverse proxy or gateway sitting in front of your storage
that can help process the data batch the data before it finally
gets persisted.
And then out of the storage right, we need to have an analysis
UI, so our engineers can actually make use of the data.
And they'd be separately from that.
We need something that's going to query the data on a ongoing
basis, polling basis, and alert whenever there's
something that violates one of our alert rules.
So this is a more complete solution.
How do we get to this with Clickhouse?
Well, there are some open source projects
making this possible already, so that you
can have a turn key solution with Clickhouse under the hood.
These are all really great projects, or you can build your own.
We're working with some customers who are doing that.
If you wanted to do that, we're here to help.
But if you just want some of the you can turn on and run,
these are all pretty great choices.
Two that I think maybe are less well-known are co-root
and query, so I'm going to talk about those
in a little bit more detail now, with my last 10 minutes,
and it will take some time for questions.
So co-root, and some of the creators of co-root are here,
if you want to attract them down and talk to them,
it's a battery-included, no code, observability solution.
So here's a spring shot of the dashboard,
and we can see some of those joins in your head
that I was talking about earlier.
We can kind of already see coming up in this.
We're using all of those different signals
to understand the health of these services,
whether we're getting that information
for metrics about the RED metrics,
whether we're getting it from tracing
and upstream and downstream relationships,
or whether we're getting it from explicit error messages
in logs.
Here we can see the topology that is generated by co-root.
It's kind of hard to see on the screen, I'm sorry.
But this is what's actually here
is the open flow material demo project,
running in Docker on my laptop.
I did not do anything, right?
I cloned to the repo and ran Docker Composer,
and I ran the single Docker Composer up command for co-root.
I didn't tell the open telemetry collector
built into the hotel demo project
to export anything to co-root, right?
Co-work used EVPF, and it got all of these relationships
using EVPF tracing.
You can enrich the data, right?
Co-work will allow you to point your collector
and send your open telemetry traces,
and then you get all the extra additional enrichment
of the application context, which is awesome,
but just out of the box, you get this.
And you can even see traffic going to in front of something
like an open telemetry collector, or an old agent.
So this is really, really useful as a first step
when you're setting up the rest of your observability pipeline.
You can see everything that you expect to happen
and everything that actually is happening
with pure automatic instrumentation using EVPF.
This is so awesome.
And then we can use that information, right?
I was talking about how we're combining all three signals.
In the screen, we can kind of see all of you.
You can't quite see the logs that could off at the bottom, right?
But we can see that we're inferring the health
both from the relationships that come from the traces,
that topology that we generated from the traces,
metrics which could have been exported as aggregated metrics
or could have been derived from traces or logs, right?
And then the logs themselves.
So it kind of brings it all together into one tool,
all correlated around the entities, the resources,
which are the things that I, as an engineer,
can actually reasoned about thinking about, like,
I understand what a node is.
I understand my process running in the node.
And that's usually the thing that I care about,
not like a specific CPU core, right?
So this allows me to contextualize all of the logs
and all of the metrics and all of the traces
for the one thing that I really care about.
And then navigate to the other things that I care about
using the topology that we derive from the traces.
OK, the other tool that I think is pretty cool,
based on top of the case, is Korean pronounced Korean.
And this is basically a translation layer that translates
between these various formats.
So you can use logQL, you can just primeQL,
temboQL, rate, all four open-time attributes sources.
And it creates APIs compatible with each one of these things.
Yeah, here's an example of some of the log ingestion
formats that it supports.
OK, so kind of a high level comparison between the two.
Right, core is more of a, like, batteries included
right-e-to-go solution.
It's got its own EVPF-based node agent that you install
on your nodes to get all of that awesome tracing data.
It does allow for OTP injection.
It's a place that's own open-time tricklector inside itself
that you can send as open-time entry data to.
And the scheme that it uses inside Clickhouse
is mostly the standard schema that is also used
by the open-time entry collector exporter for Clickhouse.
So they're kind of compatible with each other.
Core route does have some optimizations
to speed up some of the performance things.
Korean uses its own custom fork of the open-time entry collector
and uses its own Clickhouse schema that is distinct
from the core route and the collector exporter schema.
But it uses the Clickhouse materialized views
to create these projections so that we can translate
between all these different APIs and use right, like,
promQL for open-time entry data or whatever we need to do
with all this.
Oh, no, am I going to make it through this presentation?
Whoops.
No problem.
OK, problem solved.
So let's talk about the scheme is really quick and then questions.
All of these scheme is are doing some things
that are just sort of the, all of these scheme
is have these similarities, right?
Because these are things that just kind of make sense
when we're dealing with this kind of data.
We're using the standard compression.
We're using Delta encoding so with that we only need
to store changes that's going to dramatically reduce
the amount of storage that we need for all of our data.
We're using, I said at the beginning, right,
we need full text search.
How do we get that in Clickhouse with Bloom filter indexes
for maps, resources, and the full text of logs, right?
All of these tools are using the merge tree engine.
Clickhouse has some other table engines,
but merge tree engine by far the most common.
And definitely the best one for observability data.
And of course, you want to partition your data
on the time dimension.
And very common default is a seven-day time to live
for observability data.
That's typically a good amount of time
to make sure that you can respond to or diagnose
your last production incident.
If you don't have audit requirements that are longer
than that, that's a perfectly fine place to leave it at.
What is also a very easy to configure this
if you need to store something for months or years?
The open telemetry collector, export of for Clickhouse,
these are sort of the unique things about its scheme
that it uses.
So it's got maps for the metadata.
It's using those Bloom filter indexes
for really efficient full body text search.
And it uses a materialized view to calculate
the span duration from the start time and the end time,
pretty basic use of them through a lot of view,
that you may also want to adopt in your use case.
Queryn, just a really cool stuff inside its Clickhouse schema
that makes it more optimized, especially
for time series data since it's working
with so much permitias and time series data.
So it's using fingerprints for unique time series,
and then index labels with materialized views.
This allows you for really efficient.
This actually, like at the beginning, I said,
you can't do updates, but Queryn actually
does allow for updates and uses a variation of merge
tree called replacing merge tree.
And null engine is basically like an in-memory,
not persistent table, that Queryn uses
as it's sort of in-memory buffer.
And then it uses materialized views to protect out
from that null engine table to all of its other formatted tables.
So that's kind of a really cool way to deal with time series data
using some of the power built into Clickhouse.
So I'm going to talk about quickly how you might architect
us how you might use this in production.
And I think we'll have time for a couple of questions.
So you're probably going to have an open sum of your collector
or bail or something deployed as a demon set,
if you're running in Kubernetes or deployed on all of your nodes.
If you're not, then, like remember, a few slides
go ahead that like storage gateway, little yellow box on the slide.
You might use an open sum of your collector deployment
for that.
One thing about Clickhouse is that it really doesn't
like a lot of connections.
It can handle a lot of data with each insert,
but you should try to reduce the total number of inserts.
So you don't have partition explosions.
So you can use open sum of your collector
as a deployment as a way to sort of batch those inserts
and have fewer larger connections to your Clickhouse cluster.
Of course, in production, we're going to want to use replicas.
Clickhouse uses either Zookeeper or Clickhousekeeper
for coordination between the nodes in the cluster.
So you're going to want that.
And you're going to want probably exactly three
keeper nodes.
Clickhouse number of clickhouse nodes is going
to depend on your data volume.
And then the utility operator is a fully open source operator
for Kubernetes that makes managing clickhouse inside Kubernetes
a lot easier.
So you don't have to deal with things like upgrades
and persistent volume planes on your own.
You just kind of tell it, hey, this is the layout of my cluster
and the utility operator will figure out the rest for you.
Right, I just mentioned using PVC management,
rolling upgrades, and it comes to some built-in monitoring.
There's some grip on a dashboards.
And some Vermetius X-Waters built into the utility operator
to get you some more information on that layer,
sort of a between the Clickhouse binary itself
and the Kubernetes.
This is a topic for another day.
We didn't get into all of this, right?
So there's a learning.
There's a lot more stuff that you need to do that I mentioned
for a complete observability solution.
But hopefully I've given you some ideas
on how you might be able to build that around Clickhouse.
So why do I think this is really cool?
Why do I think you should actually do this?
Well, the first three points are kind of just
about having one tool instead of three tools, right?
When I'm going in using the open telemetry SDKs,
I love that it's one tool.
And instead of three different SDKs with three
different sets of documentation that I have to learn
just because I want some metrics, some traces, and some loss.
So if we can have fewer tools doing the same work of mortals,
that's usually better, as long as we're not giving up
anything else important to get there.
When we have all of our telemetry signals in one database
that resource metadata that's attached to all of those signals,
it's going to be so much easier for us to standardize that
and maintain consistency within that resource metadata.
We can, by having everything in one place,
we can have some background processes
doing post-talk, dependency mapping, right?
Like we can pre-generate these topologies on the data.
Without going in ahead of time, what those topologies look like,
and we can poorly across signals.
Thank you so much.
Thank you.
Any other questions?