All right, so welcome to my talk. This is Scott Pachink, click house in production with X-ray.
So this is me, Palo Marcosolta, I'm a software engineer working at click house, core team.
The core team is the team responsible for developing in C++ database itself. I consider myself
a systems developer, so I like low level stuff, I'm passionate about video game development,
and I'm interested in compilers, build systems, new program on languages and things like that.
All right, first a few questions. Who knows what could reloading or hot patching is?
Which you're handy? Okay, almost everyone, right? Who knows what LLVM's X-ray is?
Quite a few, not bad.
All right, so who's ever a cat to unpack something in production and which they can have a new look
trace to figure out what the hell's happening? Everyone, right? So it happens every day.
So let's go to what is hot patching, the basics. So hot patching means changing the code
while the process is still running. So in our case, since we are talking about a native application,
we are talking about modifying the machine code while the process is still running.
This allows us to have fast reputation and we can well, the process is in production.
We can patch and cover different behavior. This is often done in video game development.
It's, yeah, I don't know why. It's often called reloading, rather than code patching,
because I can use a different laptop, maybe or I don't know what's happening.
I mean, my screen is not flickering.
Okay, okay. So there are a few examples of open social libraries that are using this sort of architecture,
in which they hold reload the code. One of them is one time called by C++.
The second one is from Fungus. It's a very simple C straightforward library called CR.
And life++ is the most advanced solution that I've seen out there.
And well-injing is actually using this live++ code reload code.
So a code patch, code patching, it's been there in the Jesus, in the compilers for a while.
Yeah, okay.
All right.
So a hot patching has been out there in the compilers for a while. There are different ways in which you can,
there's that. One of them is by using the instrument functions parameter in which it generates instrumentation code
for every single function, entry and exit, and just by implementing those two functions.
You can hook to all the entries and exits.
I am using QR codes that you can see once every two seconds.
But don't worry about them because they will be one last slide in which you can take all the links together.
Well, hopefully will.
Anyone knows how to hook a patch on each DMI cable?
All right. So the other option that it's been out there for a while.
It's called a patchable function entry in which you can ask the compiler to introduce notes
into either before the function or after the function starts, but before the function body starts.
So this is done. So you save some room filled with notes so that you can override those instructions with your own code.
Okay. So let's move on to x-ray. x-ray consists of three main parts.
The third one is a compilers inserted instrumentation points in a very similar way to what we saw here with the notes.
The second part is a runtime library for enabling and deceiving tracing a runtime.
And the third part is a suite of tools for analyzing the traces.
We are not using this set of tools ourselves in clickhouse because we don't want to have to deploy into production all those tools binary and everything.
So we fight to do a more seamless experience and we have developed our own analysis tools within the same clickhouse binary that we used for everything.
So x-ray has existed since 2018 and it's available on Linux in all those architectures.
Luckily for us at clickhouse we are using x64 and r64 so we are good.
So this will be like the simplest example that you may have for using x-ray programmatically.
First you need to utilize the library then you patch this function patches all the functions.
So there's another function that allows you to patch one single function but this one patches all of them.
You set a handler and then you call whatever you want to in this case dummy full function and you do the cleanup.
This is a dummy full which print something and then the handler needs to be decorated somehow so that you don't call recursively the handler while it is handling something right.
It takes two different parameters the first one it's the function ID and the second one is the entry type which allows you to distinguish whether it's function entry or exit.
We can compile that example by using dash f x-ray instrument and then by default ldm will introduce will instrument all the functions that are larger than two hundred instructions.
Since our friend food is not larger than that I use the instruction threshold equal to one to force the compiler to instrument it.
So if we run this we can see that the food is here right before that we see the function ID which is one because in this case there is one function and that's it and the type it's here which means entry and then we see the type one which means exit at the at the end.
Now in order to do this ldm well the compiler ldm is also adding a new section into the elf binary this section is called x-ray instrumentation map which is basically a map of function ID and memory addresses memory function addresses let's say right.
So this introduces some over some it at some size to the final binary in our test we saw that for x64 it was like 4% so it seems worth it.
And checking out this full function with the debugger ldv we can check this is before patching we can see how it introduces some knobs at the beginning of the function and then some other knobs at the exit.
These pieces are usually called slets so I will call them entry slets and exit slets there is a jump first instead of all knobs because you need to do it in such a way that you can patch atomically.
So you start actually patching the knobs and finally you patch the jump to make it atomic.
Then this is after you can patch everything if you see the difference the knobs have been come a call to x-ray function entry similar.
And then you see that there is a move of one literal one which is the function ID into a register which will be used by function entry and then it will call the handlers that we have set up for that particular function.
Okay, how did we integrate this into clickhouse first what is clickhouse is all up database.
A very fast one hopefully all up stands for the A stands for analytics are supposed to OLTP that T is for transaction on such a sparse grace my SQL and the such.
Those OLTP databases store the data in rows all up that this is store the data in columns so that they can run analytics much faster because of cache coherency and everything.
It is started in 2009 the index metric by LXA millovidov who is also the founder of clickhouse ink and the city of it was open source in 2016 and there are patchy 2.0 license it is developed in C++ even though it has some third party libraries in C and in Rust it has a super strong focus in performance.
And internally in the company we use it we do a lot of dog fooding so we use it for store all the CID results for a lot of all cell mobility we use it for pastila which is a serving text snippet service.
So how did we integrate X-ray into clickhouse first when thinking about how to do this we wanted to be able to to is to patch or to hot patch this in a regular release.
So we need to make sure that whatever we did had an negligible impact in production and this nob thing that we did that we saw before actually enables to do that.
Then X-ray already provides a runtime library which allows us to patch whatever function we want to runtime which is very convenient and since this is a database we thought why don't we use SQL statements to to as a mechanism to interface with the developer to know which functions we want to patch.
We run a POC that was done by our intern Alina Barakova and I want to say thank you big thank you because I try to mentor her to the best of my abilities but as you can imagine for an intern it was an an easy task to grasp.
So see that great work I'm very happy with what she did and then I took the book and I rework it to have a production ready thing that we could release and that landed into clickhouse in 25.12 this last December so it is quite fresh just being a few minor improvements since then.
All of rank have been back ported.
Now this is the cheese sheet of how we can make use of these in SQL statements.
Peoples already have a symbol stable that you can use to install spec which symbols are within the elf binary.
So what I did was adding a new column for to have the function ID the same function ID that X-ray instrument is a map is using.
And by using that we can we can check out all the functions that we come patch.
Then we system instrument at or remove symbol as a string and then we can set three different colors that we will see in a minute and then whether we want to patch at entry or exit and the parameters actually depend on the hunter.
Then once we have added any instrumentation points we can check which are enabled by by taking a look at system instrumentation and finally.
Checking out trace log we can see which of the instrumented points were hit with which parameters, what time, etc.
So let's go through a real example of how this works.
Let's say that I want to patch this query metric log start.
I am very fond of this method because it's one of the first ones that I introduced into the search code.
So by running this query we will get a result such as this.
There are two functions that match that thing.
The first one it's that little method that we care about.
The second one if you are familiar with C++ mangling which crafty it's actually a lambda name because that function contains a lambda.
So by default click also ready smart enough to assume that you meant the usual start query if you only use a start query.
Otherwise you can use the full model name and I'm patched a lambda.
Right let's add three instrumentation points the three that we have the first one will be a log.
We will patch it at entry and then the only parameter that we take it's our coded string.
This will be logged into our usual logging system along with the stack trace so that we know where it will come from.
This is very useful to know whether your path is taken this way or this other way.
Then we have slip which in this case we are patching at exit and then it accepts either one single parameter to slip for a fix amount of time or you can pass two different parameters to slip for a random time uniformly distributed between those two values.
To try to to exercise some stress to see whether something happens right.
Then profile is my favorite handler because it allows me to profile deterministically.
Click also ready has some clean profiling but you know for doing development and even in production is very useful to have deterministic profiling to catch all the outliers and everything.
All right so once we have.
Once we have instrumented at where points we can check them out in system instrumentation you see log at entry with the parameters same for slip.
Same for profile for profile there is no entry or exit because if you want to profile you need to patch both entry and exit or wise it doesn't make any sense.
And after you can finish patching and testing everything that you want you can remove either all of the instrumented points you can remove one of them or you can remove all of all of the instrumented points associated with specific function as in this case query metric log blah blah.
Finally you can take a look at trace log to see what instrumented points we are hit in this case we see profile at entry the duration nanoseconds we cannot see there because of the PDF but it says null okay.
At exit we already have the full function executed so we can actually get the duration nanoseconds.
This is fine but it is not very human friendly so when thinking about how we can make this better we thought about perfect speed scope and these sort of tools all of them use.
The exit current trace event format which is a very straightforward this one file that looks like that but we can export using a SQL such as this one it's a big each SQL I don't want you to focus too much on that I just want you to know that it's.
And this is what it looks like in perfecto I patch this slip for nanoseconds.
Function and you can see the duration the thread you run you can see all the thread from up to bottom the query ad belongs to the cbid the start super convenient for the the ministering profiling.
In terms of feature work we are thinking about adding more handlers as we think that they are needed.
We discuss about if it makes sense to add some retombation to always creating with lua brands until it is but this will probably.
Open a new kind of worms because the moment that you need to grip on the bindings for everything you allow people to run whatever the one it's probably risky.
And the most important thing is to educate others that this thing exists especially people from support team and everything so they can debug production issues more.
Collect the feedback to improve this and some caveats you cannot really mix x-ray and sanitize bills because they are symbols clashing in little clan RT.
And this is not an issue for our release bills because we don't release sanitize bills but for the CI and for all the tests that we have.
We cannot have both of them so we cannot test sanitize bills along with x-ray.
You need to be aware that if you mess something up.
You do it big time because you are doing it in production and it's a hot patchy machine called so nothing has happened but if anything may happen it will be.
You're doing rich things and even though profiling is very cool profiling a function as we saw there is no easy way to profile all the functions within that function because you will need to know beforehand all the functions that you need to patch and it's difficult to do so you may patch all the functions and only enable this for those on your same thread but it's it's messy and we don't want to affect the rest of the program right.
So for now we are going to keep it that way.
Okay finally there is a community slack that you can join if you want to we will also have a click house community dinner tonight in Brussels.
You are welcome to come there would be finger food and stuff like that we also have our technical blocks which are really good.
And this is the final QR code that you may want to check out because all the links that I gave in all the slides are there.
Two minutes.
Any questions?
I just want to say that time span is and this is like CS time so I hope that you appreciate the commitment to the talk.
Yeah the question is regarding the performance difference between the touching and if statement or changing the machine code directly.
So I wish again I haven't tested that personally but you know that any statement is going to involve a one branch play with the branch predictor and everything.
It looks like it's not going to be much always but once you add that into every single function that is larger than 200 it starts to accumulate right.
So we wanted to have something as low level as we could and this is the best that we could find and I think it's really cool x-ray it's a cool technology to do these sort of things.
Question.
Yes the question is if we are running this in production and the answer is yes.
Yeah I hope that's fun.
I wish I could eat the guys not listening to the talk.
This is something that.
This is something that only developers can do in our production thing and there is a media of security checks that you need to be granted in order to do this.
There are different users who have different capabilities and this will be like the admin god mode capability so I think that we should be good but of course if you're.
I mean everything that you may patch it's either going to look which is nothing sleep or profile my effect a little bit but it's supposed to be meant for developers depending something in production not for a regular user.
I don't know this is why we kind of said we're talking a shame for this scripting will be cool but you know if you allow people to do whatever they want they will do whatever they want.
Another question.
Actually only allow you to patch on a call if you want to jump and then developers to control flow to like patch on.
So could you repeat then?
Can you only the entry to the entry set?
Uh-huh.
I just call in and I also have to jump and then developers to control those who actually want to do that.
Yeah the question is rewarding whether you can only code patch the function entry or exit or in the middle of the code right.
It is right now you can only do it with either a entry or exit because this is where X ray it's introducing the notes.
I guess that no.
No no, the question is if I always let the start but tonight it's like if function is about tonight jump for another function I would know that.
Ah ah ah.
Alright so the question was whether you can jump into a different function which fixes the behavior let's say of your wrong function.
Yes, it could be done, it's not done because right now there's no handover that does that but there's nothing
Parenting you from doing that. I mean you call diam into somewhere else and one some other could which will actually be like
I will our code patching for something that wants to change behavior like in video games or whatever
It could be done. It is not done and
There was another question over there
All right
Thank you