Okay, can everybody hear me okay?
Yeah, sounds like it.
All right.
I think we'll get started.
Hi everybody.
Thanks for coming.
My name is Josh Lee.
I am a developer at Altinity.
We do hosting and services and support for Clickhouse, which is a little bit of a giveaway
about what I'm going to be talking about here.
The name of this talk is Tracing, a Cloud Native database, and Clickhouse is the Cloud
Native database specifically that we're going to be talking about.
So let's dive in.
So before I can talk about Tracing, Clickhouse, we need to just level set on what
Distributed Tracing is.
So just really quickly, right, for anyone who may not be familiar.
Distributed Tracing is a way of tracing the flow of a request through multiple services that
is running on multiple hosts.
So it's kind of like a stack trace, except across the network, right?
And what you end up getting when you look at an individual trace, I'm sure many of you
are familiar with this.
You get a waterfall view, looks kind of like the waterfall view and like Chrome developers
or something.
And what's interesting about this is actually really the only thing that you need, like,
the difference between a trace and a log, right, to get this from logs, you only need two
extra pieces of information.
You need the parent ID of the thing that came before it, the thing that called the
thing that you're logging.
And you need the end time, right?
Logs usually have a start time.
You include the end time, you can then calculate the duration.
You include the parent ID.
You get this nice tree, graph, structure, and we have tracing.
Right?
So it's not that complicated.
It's just logging with a few extra steps.
But it's very powerful in what it how it allows us to analyze distributed applications,
like a distributed database, like Clickhouse.
Okay.
So, distributed tracing is awesome, but, you know, ten years ago, the only way to do it really
was to build it yourself or to work with a vendor, like Dynatrice or something like that,
right?
And to open Solometry.
Right?
So, Open Solometry merged a couple of projects and is now sort of the fact to open source
way to do distributed tracing.
Along other things, right?
You can use it for metrics and logs.
But what the problem it really solves is distributed tracing.
And it's a couple of things, right?
So, Open Solometry is a format and it's a bunch of tools and libraries that sort of
conform to that format.
The thing that we're most interested in for this talk today is just the format.
We're not going to be using that many of the tools, but just know that they're available.
The part that's interesting to us, right, when we're talking about a trace, how do we actually
get that parent ID, right?
That thing that called the thing?
And the way that we do it in Open Solometry is with the W3C trace context
propagation specification.
So, basically, we just have this header.
And we just put the trace ID in the header.
And as long as the thing that we're calling over HTTP understands how to interpret that header,
it can use that trace ID when it's building its own trace spans, right?
Each part of the trace is called a span.
Okay.
So, I mentioned, right, there's a whole bunch of stuff in Open Solometry that are tools that
sort of conform to the specification.
Like, you can build an entire observability stack using this.
And you could even use Clickhouse as sort of the destination for some of that observability data,
if you wanted to.
So, let's talk about Clickhouse a little bit.
What is Clickhouse?
It's SQL compatible.
It's an analytics database.
It's really, really fast.
Injusting millions of rows per second is no problem.
Queryn overpetted weights of data.
And under a second is no problem.
And it's, I had massively scalable, but the test also claimed that.
So, I'm just going to claim even more massively scalable.
So, let's talk, I mentioned, right, you kind of use Clickhouse for observability.
That's actually how I came into the world of Clickhouse.
Working at an observability vendor.
I'm only going to talk about this very briefly because tomorrow in the monitoring of observability
Debt Room.
I'm going to be talking about this much more in-depth using Clickhouse for observability.
But kind of why it lends itself to the observability use case.
Part of the reason is because of this architecture, right?
So, the way Clickhouse works in part of the way that it's so scalable is when you're ingesting data.
It gets written into these partitions that are, that can be pretty small, right?
They can be small on memory just all of the data that came with a specific insert.
And then, as that data is sitting on the disk, right?
We have these little small parts that are very inefficient to read.
What Clickhouse does is in the background it merges them.
It keeps them ordered on disk, so that we can read them very, very fast if we're reading,
using the same ordering that they were inserted with.
And we eventually get much, much faster queries on our fully marched parts.
So, this led itself to observability for a number of reasons.
It's optimized for the right once read many use case of observability data, right?
Like we want to record all of this data, it never gets updated.
A log never gets updated.
Eventually, we want to delete it when we don't care about it anymore.
But we want to read it a lot in the meantime.
Because we're a partitioning that data, usually by the time frame,
it's very, very easy to clean up with things like TTL that you can just delete an entire partition
when you don't care about it anymore.
It's very fast, as I mentioned, it can deal with cardinality, like extremely high cardinality.
There's a quote from CloudFlare.
They use Clickhouse internally.
And they, right, they talk about how it's with prometius low cardinality means dozens,
maybe scores.
With Clickhouse, low cardinality means millions.
So, it's the cardinality is completely different orders of magnitude, right?
In terms of what we can do it.
And this is really useful at observability data,
when we don't necessarily have control over the cardinality of the data that we're storing.
There's a lot more, and like I said, I'm going to talk about this tomorrow,
but there are a lot of integrations with Clickhouse that make it really really awesome for observability.
Co-groups, signals, query, and hypergex,
Christmas, these are all tools of observability tools built on Clickhouse.
They're awesome.
All right, so here's what we're actually here to talk about today,
which is observing Clickhouse itself, not using it for our other observability data.
So, one of the cool things about Clickhouse is that it's actually very observable, right?
Observable in the mechanical engineering definition of the word,
of being able to understand how a system is working,
without changing the system without prodding it, right?
Clickhouse is just reporting all of the data and recording all of the data about its internal operations,
and we can just examine that and inspect that whenever we need to know what's going on.
It does a lot of this in these system tables, right?
We have the query log of all of the queries, the park log of the log of the partitions and the merging.
The trace log, this is actually not distributed tracing, right?
This is a stack trace log and can be used for profiling,
and then we have this open geometry span log, which may not be in your Clickhouse cluster out of the box,
but is very, very easy to turn it on and I'm going to show you how.
And then there's a lot more, right?
But let's talk about the architecture real quick.
So, like I said, Clickhouse is very scalable.
Part of how this happens is horizontally, right?
So, we can parallelize the queries.
Clickhouse is very, very good at deciding exactly which node need to be involved in a specific query.
It's going to distribute that work to the nodes that need to be involved that might have relevant data.
And then like we showed a couple of slides ago, right?
The data is all stored in a column or format ordered in these mergeable partitions.
And you can also use things like tiered storage, right?
With like S3 and object storage so that you can, you know, have your hot data very close to the compute,
ready to go and have one term storage that is cheaper, but maybe a little bit slower.
And then we have clickhouse keeper or zookeeper, you can use either one,
clickhouse keeper sort of the clickhouse native one, zookeeper right, of course, we're all familiar with.
And this is going to handle things like consensus, replication, and all of those things that keep the clickhouse clusters coordinated
and working together, right?
And we were going to talk about this, how it's kind of merging in the background and improving our query efficiency over time.
And then when it's actually processing a query, right?
So we say we have an insert, we have these three steps, we're going to parse and plan the query.
This is actually not a distributed node, but it's kind of the same idea, right?
We're going to parse and plan the query, we're going to load the relevant data and then we're eventually going to format and respond.
Let me show some of these steps in our tracing in a little bit.
But one way that we can speed this up, make it work on even more data, right?
It's to just paralyze it.
We can do that inside the single node with more threads, and we can also add more nodes for example scaling.
And this is what allows us to do those, you know, petabyte reads in under a few seconds or a second.
Similar, similarly, right, with aggregations, except we're doing some extra steps in memory before we're responding.
So, guys, this is really, really good at aggregation, right?
Like, oh, that database has calmed our databases.
These really lend themselves to, I don't need a specific data point, but I need to aggregate all of these things across billions of rows.
One story that I like to tell is when I worked at an observability vendor, and I need to know which customers were using a specific new feature.
And we hadn't instrumented it.
And I asked my late engineer, hey, do we know which customers are using this feature?
He was like, well, it would show up in the metadata of their spans if they're using it.
So let me just query all of the spans for the last three months for all of our customers and tell you what percentage we're using this new feature.
And I got my answer in about seven seconds.
That's pretty cool, right?
So here's some more.
How does click out right on how click out is kind of doing that?
Thread aggregation with hash tables and then eventually merging and giving us a result.
Okay.
So, I showed that architecture slide for open geometry earlier.
And one of the things we saw was the SDKs, right?
And we have all these awesome SDKs for 11 different programming languages in open telemetry.
One of them is for C++.
Clickhouse is written in C++.
So it's like, oh, if we wanted to trace clickhouse, we would just install this.
See, wait, how do we, which is that mean we have to make our own build of clickhouse.
That sounds crazy what we could use auto instrumentation.
If we were in Python or Java, but clickhouse isn't in one of those languages that has auto instrumentation with open telemetry.
But like I said, it's all just built in.
We just have to turn it on.
Clickhouse already has open telemetry tracing built in if you turn it on.
So what does that look like?
Well, you turn it on.
And you get something like this.
Now, this is a pretty simple query, right?
This is a table creation query, but just kind of showing a very basic trace.
So we can see this in this first span.
Right, we're receiving a network request essentially.
We're using the TCP handler.
And then we're processing a query.
Nothing's actually happening here, right?
You can see this black bars where things are actually happening.
And kind of without the black bar means we're waiting for other processes or functions to complete.
So then we step down into the query interpreter and we interpret the query.
And the query interpreter sees that this is a table creation query.
And so it does its DVL action, right?
Then we return to the query.
And it takes, you know, what, two and a half milliseconds for the network to respond here.
So this is kind of cool.
This is a single trace.
Let's look at a more interesting query.
I think maybe this one.
Yeah, so here's a much more interesting query, right?
And we can see some of the parallelization that's happening here to speed up this read query.
It actually ended up returning zero results.
But, right, it's really, we can see how awesome this format is.
Now, this is useful in it on its own, but it's even more useful in aggregate.
Because in aggregate, we can start to identify patterns, right?
Like, oh, this particular sub-process is always slow on this node.
What may be happening on that node that we need to look at.
And so we can kind of use this to sort of optimize both our infrastructure layout.
And maybe we might see something about how merging is happening.
It might tell us that we need to change our queries, right?
And reformat them and make them more efficient or we might need to change our schema.
And make that conform better to what clickhouse can sort of spread out.
These aren't slides anymore, so yeah, there we go.
Okay.
So how did I get that?
Well, oh, that's hard to read.
Okay, as you've done.
So I said, it's just one flag to turn it on, right?
And then we get this table.
And this table is the open telemetry span log table.
These are fully formed open telemetry spans ready to go ready to use in your observability tool of choice.
Where we were just looking at Grafana, but you could ship these anywhere that's open telemetry compatible.
To get these, all we need to do is tell Clickhouse, hey, I want you to start some traces,
even if you didn't get a trace context from the thing that called you.
So we can turn this on on a session, you can also step this on a profile, right?
And then based on that probability, so if it's set to one, it's always going to create a new trace for every new query.
If it's 0.5, half of the time, it will create a new trace.
And you can, you know, reduce some of your data because if you have a lot of if you've high throughput of queries,
this can generate a lot of data, right?
You probably don't want this turned on for things like a profile that's doing a health check, right?
That would just be kind of redundant and wasteful.
But say you have a user who's like always breaking your Clickhouse cluster.
You can create a profile for them that turns on all the debugging.
So you know exactly what they did and can tell them not to do that again.
And we can also do it for an individual from the client, right?
So the client can provide a trace ID here.
I'm just spoofing one.
I'm like, this is just a random string that I came up with.
And I'm like, hey, use this as your parent trace, right?
If we were coming from an actual like Python application or a job application that's calling Clickhouse,
that SDK would include that trace ID.
And then that would allow us to contextualize all of this distributed stuff happening inside Clickhouse,
with all of the other stuff that we had going on before we got to Clickhouse and anything downstream as well.
This is the beauty of trace propagation.
And then once we've got that data in that table, right?
What do we actually do with it?
How do we get it out into our observability systems?
This query is from the documentation.
It's a little bit out of date.
You've zipped in up there.
It's like an older trace format.
I think Yeager is still technically supportive, but it's not on by default.
So the documentation is out of date.
We're going to work on fixing that.
But what this shows is that we can use a combination of Clickhouse features,
a materialized view, right?
And a URL table engine, and essentially export all of our trace spans over an HTTP endpoint.
I wouldn't recommend actually doing this in production.
There's some issues you're going to lose.
Basically, you'll lose traces when there are errors,
which is when you most want those traces.
And you might also create availability issues for your cluster using the URL engine
if the endpoint that is trying to write to isn't highly available.
So what I would recommend doing instead, like for production,
is just straight, right?
Make your own receiver for the open symmetry collector.
Make your own program that basically just scrapes that table.
And we used a view actually for it, so I can show that.
Right? We created this view.
Instead of Zipkin format, this is open symmetry format.
You can also turn this into a materialized view.
Just going to take that deal with those issues that I've talked about a second ago for production.
But you can create this view,
and then build something that scrapes this,
and you have your trace spans.
Okay.
And I mentioned briefly about that we could kind of put the client side traces into clickhouse.
I actually have a call to action here, because this is not complete.
This is the beauty of open source, right?
We do get trace propagation with these clickhouse clients.
But there are clickhouse clients for many more languages,
and there are community contributed ones.
Most of them do not include this open symmetry trace propagation.
So please, if you're using one of those client libraries,
and you feel compelled to contribute,
consider adding open symmetry trace context propagation.
That would be awesome.
Please help.
Okay, real quickly some best practices,
and then we'll take a few questions.
So this is awesome for distributed queries.
I was showing queries right now and I think we'll know it,
but this is most useful when you have distributed queries
running across multiple nodes,
and especially when you want to identify issues
that are specific to a shard or two node,
you can use it to identify root causes of slowness,
because problems are going to cascade
and traces have the order of operations embedded in their structure.
So it tells you the root clause,
basically in the structure of the log itself.
You don't have to figure this out.
You don't have to do any investigation.
And right, so I kind of mentioned the times
when you would want to turn this on,
especially for problematic users,
problematic queries, things like that,
not for routine things like health checks,
and so on.
Thanks.
Yeah.
Thank you.
Any questions?
Redoison.
Good job.
Sorry.
Hi.
My question is,
how clickhouse is working on the client side
to provide more scalability in the number of client connections
that can connect to the cluster?
Because I know there's sort of like a hard limit
when you install a cluster,
and then you have to kind of make the changes,
but is there any work for kind of
make the number of clients connected to the clickhouse cluster
be more like changeable in a way
that you can have like thousand clients connecting
to the cluster without any issues.
Because yeah, nowadays it's not like very.
Yeah.
I know that this is not respect.
I'm not aware of some of my co-workers here,
so we want to stop by the Osakam booth
and ask them they might be more aware of future work,
but the way it stands currently,
you definitely want to reduce the number of queries.
You can insert a lot of data with each query,
and clickhouse has a strong preference for
big inserts versus many inserts.
And think, right?
They want to reduce the reading side,
so kind of read like thousand times
like using thousand connections.
Oh, that's not it.
So just scaling up the reading?
Yeah, actually the amount of clients that will read that data
because right now it's just like when you install.
It's like, I think it's 100 or 200 something like that.
Okay.
Kind of limpets.
And then if your application kind of goes there,
every time you create a loop and then,
like you can scale and then,
these scale.
Yeah.
For reading, I don't know.
For writing, you can use a reverse proxy,
like the opens, elementary collectors are great.
Reverse proxy to put in front.
Up there goes cluster, maybe there's something similar
that could have like a read cache on the read side.
Robert, do you know them?
Yeah.
Okay.
Come to the booth and talk to our CTO and he'd like to thank it.
Any other questions?
Oh.
Hi.
It's slightly left field question,
but when you're collecting a telemetry data,
do you hit any privacy or a consensus?
Sorry, what?
Any privacy concerns when you're collecting all the data?
Absolutely.
Now, in Clickhouse itself,
it is going to include that query statement in the trace
span, so if there's PII in that query statement,
that's something that you're going to have to build
some way to redact that.
If there's no PII or anything that you care about in the
actual query statement, it's not going to get caught up
by the built-in clickhouse instrumentation,
but it might get caught if you're using open
telemetry auto instrumentation.
So it's definitely something you'd want to build filters for.
Are there any specific strategies that you have for that kind of reduction?
Less familiar with the current state of the art right now in this.
I do believe that Red Hat was working on an open source tool
specifically for handling correlation and filtering of this type of data?
Cool. Thanks.
Yeah.
I thought I saw a hand go up.
Maybe I'm imagining.
Other questions?
Thank you.