Yeah, hello, welcome to our talk on zero code distributed traces for any programming language
with EVPF, so that's the topic.
A few words about us, so I'm Fabian, I am engineering manager at Grafana, also active in open
source, a contributor to Prometheus, and we have Raphael.
Does that work?
Yeah, so Raphael, I'll work at the Raphael and Raphael, it's a software engineer, also
back running open source, mostly involved with the QT project in the past, and yeah, here
we are.
All right, and the way the structure at this talk is I will do more of the introduction
over here and say like what problems we are trying to solve, and then Raphael is the actual
engineer, you know, we'll talk more about how we solved it, and he will do the main part
of the talk, I think, so let's see, but for introduction, the tool we are going to use
is named Grafana Bayler, it's an EVPF based audio instrumentation tool, it's open source,
it's on GitHub, it's currently under the Grafana GitHub organization, licensed under Apache
2.0 license, however we have started a process to donate it to open telemetry, it's currently
being reviewed by the open telemetry technical committee, and once this review process is
done, our intention is to move it over to the open telemetry project, within open telemetry
it will likely not keep the name Bayler, it will be named something like open telemetry
EPPF instrumentation or something like that, right?
So that's the ticket, like if you want to follow the status of that.
Yeah, and I guess what is Bayler, I mean, if you want to show a tool, like the best thing
you can do is a demo, even if it's back and tool, so there's probably not too much
to demo, but what I did, I had a couple of go services running in the local Kubernetes
cluster on my laptop, installed Bayler as a demon set, told it to instrument everything
in my default namespace in Kubernetes, it found these go services, so there's a couple
of them, and what we see here is a pretty standard, you know, graphana dashboard with
the three most important metrics that you would like when you monitor HTTP services,
reclust rates, error rates, and durations, how long things take, right, and what, so this
is all metrics, right? It has nothing to do with tracing, but what tracing comes in is,
if you, for example, see that the error rate here of my product service is more than 20%,
you might want to learn what's the reason for that, right, and in a distributed application,
of course, the root cost might not be in the product service itself, but some are downstream,
like whatever, it's calling, and so this is where tracing comes in, so you go to your trace
database, run a query, for example, you would say I'm looking for any stand within HTTP response
status code of 500, then you get examples of erroneous calls, right, and if you look at
such a trace, so it's maybe close that window so that we have a little bit more space,
so you see as indicated by this red symbol here that there is an error in the product service,
but if you scroll down you see that this product service also calls the pricing service a couple
of time, and that's also an error down here, right, and so if you open the details, you can
you know have a look and see, you know, that it's an HTTP request and which part is called,
but now it cannot scroll down anymore, but you get the picture, right, and so the reason
I am showing you this is, you know, to just let you know that this talk is about distributed tracing,
and also the previous one, and there's a lot of talk about distributed tracing,
metrics are still, I think, the core, like the most important signal, right, metrics are what you
have on your dashboards, what you define your alerts with, what you use in your SNOs, and so forth,
and metrics with EDPF works pretty great, so we have support for a lot of natural protocols,
HTTP, HTTP, HTTP, HTTP2s, and so forth, right, a huge list, we are adding more and more to it,
it works for any programming language, so I listed a few here, but actually I think as of today,
I'm not aware of any language where it doesn't work, so if you have any language where you
wouldn't get proper metrics, open a issue, we will figure out why, but it's actually pretty
pretty stable, and of course there's not only network metrics, we have process metrics like whatever
memory usage you use it and that stuff, right, so if you do tracing, I mean because if you
have seen if you actually want to go to the root cause of things tracing is pretty useful,
tracing with EDPF is harder than metrics with EDPF, right, so why is it hard? So basically,
you need to do three things if you want to do tracing with EDPF, so we have a service here,
it gets incoming HTTP requests, does something, and at some point performs an outgoing HTTP request,
the first thing you need to do, and then HTTP requests is coming in, you need to figure out if
it already is part of an existing trace, which you learn by looking at the HTTP headers,
there's a trace header with the trace ID and the parent span ID, and this is done, right,
we can look into headers of incoming requests and parse these and figure out the current span
context. The second part, the service does something and you need to keep track, you know when the
service does an outgoing request, which incoming request is the context that this belongs to,
and this is depending on the framework that you use, this is hard to do with EDPF, so we solved this
for a couple of frameworks, first of all if you have just a single threaded application, that's
easy, right, if you have the standard Java server model with one thread per request, that's also
easy, if you have multi threading it gets harder, so we solved it for goals, so if you start a couple
of goal routines and they are scheduled and so forth, we track these goal routines and can still know
what the original context was, so multi threading with goal works, we instrumented no JSAsian
groups and so forth, so there are a couple of frameworks where we reliably know which outgoing requests
belong to which incoming requests, but we have to be honest, it doesn't work in all cases,
so for example, if you have a fancy, reactive Java application with some kind of framework,
there's a lot of scheduling back and forth on thread pools, we might lose track, so it's basically
depends a little bit on the framework that you use, whether we can track these requests or not, right,
and then the last part is, if you figured out the trace context that belongs to this outgoing
request, you need to add the trace ID and the parent spend ID to the outgoing requests, so that the
next service downstream knows which trace this request belongs to, and this is also hard to do,
but this is something that we've solved in the previous couple of months, and this is what
this talk is about, right, and now I'm handing over to Rafael, who will try to do all the details,
like what we did, what proper problems we ran into and so forth, so take fun,
all right, so as I've been just explained to you, we need people, what one is to propagate
the context, so this is the sequence diagram of the demo that Fabian just showed you, and we have
all these requests going on, they are not known dashed arrows, and so what needs to go around
basically two things, the spend ID and the trace ID, as you know, spend ID, identify each individual
request, and the trace ID is used to tie them together, so we have one ID that
changed everything together, so that's one of those what it looks like, we got
header, HTTP header that gets injected, it's defined by the W3C, it's called trace parent,
so it's made of the vendor-infold trace ID, the spend ID and the flights, and then we end up
with these big string there, that gets injected into the HTTP header, and how is that done,
like how do we pass that around with all the instrumentation, or not, like how can we inject that?
Well, the obvious way is to use an SDK, right, there's some protocol there, you know, some API,
and we do it, but, you know, you promise this, you're a code instrumentation,
it's a programming language, so that's not what we're going to be talking about,
and that's where Bala comes into play with EBPF, just being curious here, who's familiar with
EBPF and knows what it is, they could be, okay, that's best trait, for those who don't know,
very, very, very constrained terms, it's a Linux kernel infrastructure that let us run,
we get small programs, they are very safe, they go to a very fire, and these programs,
they give us some kind of visibility inside the kernel, and we can attach them to different
parts of the kernel, they're different kinds of EBPF programs that do different rows, and we use that
to correlate the requests, and also to inject a trace parent. So, basically, we have too simple
requirements to do that, one is probably becoming a request, as I said, and writing memory,
so writing trace print, had that level, EBPF level were basically writing process memory,
and that's, yeah, that's where the problem begins. So, it's kind of difficult to write process
memory with EBPF, so, I mean, this is just a verbose code, it's an example of how we do it with
go, we don't have to really understand it, the takeaway here is that EBPF programs, they don't have,
you cannot use a lead C or anything like that, you can only use a set of predefined helpers,
one of them is called BPF ProGrid user, and in this case, we're using this to inject the trace
parent's string for a go application, but this helper is a bad idea. It was added back in 2016,
to the Linux kernel, but the main goal was just to help people messing with EBPF to make experiments
or, you know, debug things, and because you're writing user space memory, you can crush your program
or do bad things, and also you need capsis, that means which means you need root,
so, you know, security implications, and yeah, you will see this helper in a few
auto instrumentation projects like bail out or tell go, but mostly in malicious skills,
if you're going GitHub, so because of that, it was locked down in August 2021 under the Linux
security mode, meaning that for production, it's really difficult to use it, I mean, it's been
used, it works, but, you know, it has a lot of downsides, so we're starting looking for alternatives,
like, what can we do? Well, that EBPF probe right user, it works in one kind of EBPF program
called probes that get, they get attached to, it's like a breakpoint, making it very bad analogy,
that you get attached to a function in your program or a kernel function. There is a different
kind of program called classifier program, BPS program in me, that lets us see network packets,
so it's no longer, we're not longer dealing with functions or kernel functions, but it
sits really low level on the network stack, close to the, you know, the data path, close to the
network card or the, you know, lowest level, and we can analyze the contents of packets, and
these kind of programs, they let us change the contents of the packets, so right packet to memory,
so not program memory, nor user space memory, but just messing with the packet, so
please, the idea then, what we're trying to do, we see all these packets going through
the on EUS path, so the outgoing packets, so that those are the outgoing HTTP requests that
fragment was talking about, we see, okay, do they have a HTTP header there, are they a HTTP,
request cool, then we need to punch a hole on that HTTP header and then write the trace
transparent string that be, that be string, too easy, right, but as you know, famous last words,
because it's never that easy, so our first attempt was to do something like this, so here's like
a TCP packet, it has all these headers, the Ethernet IP, TCP, and then on green there,
you have the HTTP header, there is a helper, it can be used with the programs called change head,
so the idea was to extend the head of the packet, and then we push, we copy, we find where the
HTTP header starts, we push it up and making room for our trace parent, and then we write it,
cool, not really, because turns out the kernel, it keeps track of the original offsets or of
where these headers are, the Ethernet IP and whatnot, so when we adjust it, the kernel also adjust
the offsets, and then we manually copy and push it up, we mess with that, what happens then is that
when we try to send out the packet, it gets dropped, so it doesn't work, so we need to do something
else, and what we do instead is we change the tail of the packet, so it's a different helper,
has a bit of downside that we need to copy all the payload down of that packet, and we inject the
trick, where it's right, no, so we did that, the packet goes out, so just a digression here,
so the way TCP works in really small, not shell, is that for to make sure the nothing gets
lost in the way, we have something called TCP sequence number, so it basically you send a TCP
packet out, and then your peer replies, okay, I've received these many bytes, this is the sequence
number, so if you're my traffic, I've already sent out, let's say, 100 bytes, then I have the
original packet here, that's 50 bytes, I'm just making these numbers up, and I send it out,
in total I have already transmitted 150 bytes, and then I get a anac packet saying, okay,
anac 150, when we do this, this happens at the really end of the stack of the network stack,
we exchange the tail, the carrier doesn't know about that, it's just messing it, so now we're
no longer, let's say 50 bytes, we have 60 bytes, we send it out, and then in total we have
already transmitted, instead of 150, we transmitted 160 bytes, and we get anac 160, and then
the packet traverse up the network stack, but the carrier says, oh, but I was waiting for 150,
now this is bad, drop it, so what do we do? Well, crazy idea, I mean it, first attempt it was an
experiment we tried to, okay, kind of fudge with the kernel, so we keep track of how many bytes
we have extended throughout the TCP connection, and then when we get anac back from our peer,
TCP peer, we adjust that, we subtract those bytes again, and we fool the kernel, that works,
but you know it's not very robust, if you need to have like bailer running there, keeping tabs
on what's going on, and if bailer goes away, then you messed up all your connections, and you know,
it's not great, so we figure out a more robust approach, so I was telling you about
the BFF as different kinds of programs, so far we've been using one that's product types,
cat CLS, or the classifier program, but there's a different kind of BFF program called socket
filter, so that's the first one there, SKMS MSG, so this program sits very high up on the on the
that a path of the package on the network stack, we cannot write package data, so we cannot
mess with the contents of the packet, but we can add space in the packet, so it lets us use a
different kind of helper, so called message push data, that lets us expand the packet in arbitrary
offset, so a BFF is all about which program you're running and the set of helpers this program
is allowed to use, so we do a two-tier approach, right, we use the socket message
program to punch the hope we want, and then the kernel is aware of that, because once the kernel
is reached the part of the kernel where it starts dealing with the offsets of the headers I talked
about before, it accounts for that, and then once we receive that down the data path on the original
classifier program all it has to do is to write a transparent string, so that works pretty nicely,
but obviously it does some downsides here, first of all we're looking at strings writing data,
it's all plain text, or we need to be able to parse that, so we doesn't work in cryptid traffic,
this might be, it might be a way out in the long term with KTLS, so that's TLS support in the kernel,
then we could attach EBFF programs to strategic points there where we could still see the data
before it gets encrypted, but it's not the case yet, and we don't have HTTP to support it in
beta yet, we're working on that, it's a bit more complicated, so we have an alternative,
so they are complementary, we want to be robust, we want to be able to propagate the trace context,
and we encode the trace parent in the IP header, so that's the other idea, so the IP header,
it has a few fields we cannot use, so in the right-hand side I have an IPv4 packet or the layout,
and this is IPv6, so IPv4 has a section called options, and IPv6 has also a section called options,
but it works a bit differently because it's just a bunch of embedded headers in the layout,
so what we do here, so I'm going to show you IPv4, but it's the same logic applies for IPv6,
so here's like a pseudo-sympified version of the same packet, we have the ethernet header IPv4,
TCP, and payload, and then the classifier program can also use a different helper called
adjust room, this helper is specifically made for operating in this headers like IPv4 or ethernet,
and it's only done for that, you cannot adjust room on payload or TCP, so we basically insert
a space for a new IP option, so right IPv4, the option part of the packet goes
right in sequence, so we insert that option, and then we encode the transparent there, but
we don't have a lot of space there, so we cannot encode everything,
can I encode the big screen, or anything like that, so we have to be a bit clever about that,
so we don't care about vendor info, we do not encode the span ID, we do not encode the flags,
we need only 20 bytes, so basically the way in IPv4 it's encoded is we write the option ID,
so we use an option called stream ID, it's rarely used, so hopefully it doesn't conflict with anything,
and then we encode the length, so we need 16 bytes, that's the trace parent that we want to
propagate, and then some padding, because it needs to be a multiple of four, but then you won't
ring okay, what about the span ID, right, we're not passing it along, well we don't have to,
because it's just a unique and random number, so on AWS, we need two kinds of programs,
I was only talking about AWS, injecting things, but now we need an ingress program, so on the
other end of the data path, when we're receiving the packets, we need to be able to see if there is
one of those options encoded in my packet, and be able to parse it, and then on ingress,
we generate the span ID at that stage, and we derive it from the TCP length on the TCP packet,
the length number, and the EC number, the inflame was that number they gave us the problems before,
so eight bytes each together, 16 bytes, and we have our span ID, and Baylor remembers that, and
it keeps steps on things, so there are obviously caveats with that, we require EBPF agents or Baylor
to run on both sides now, whereas with injecting the trace parent on the HTTP header, it plays
nicely with all the instrumentation, and manual instrumentation, because it can just parse it,
we can maybe depending on your network topology, it can be stripped by
network layer features, it doesn't work with out seven boxes, because of that as well,
unless it's instrumented, because then we can re-inject it, so you need to test it, usually works,
but you know, famous as words as I say, we are looking into TCP, encoding these things into TCP options,
and TCP header, we had a few setbacks in the past, but we figured out a different way we're working
that to see if that works, because that would solve this, they usually don't get stripped,
and this is a, well, there's a summary of comparing both approach, so the original HTTP header approach,
it's a multi-tier approach, two kinds of EEPF programs, it plays nicely with center open telemetry
instrumentation, because you can inject manually your trace parent, Baylor, we know about it,
we can Baylor, we then inject itself, we are SDK, we know about it, so it's like very,
pretty much transparent, not suitable for HTTP traffic, and it's still working HTTP support,
IP header, it's not compatible with standard open telemetry instrumentation, but, you know,
can work with encryption traffic and requires the instrumentation for all services involved,
and finally, I mean, we just had a Baylor release now to that zero that brings all of these goodies
there, but you know, plenty of work to do, so we encourage you to get involved,
it's everything is open source, you can find those in Github, and also, we have on Refinite
Public Slack and the Baylor channel, and yeah, I guess, that's pretty much it.
Any questions?
Can you also, if you go, I'll just keep it quiet, so we can hear the questions.
Yeah, I'm really interested in, like, this is, I didn't hear you, okay, can you hear me now?
Yes.
Okay, this was awesome, thank you very much, and I'm already using this, I really appreciate it.
I am interested, particularly in things that can auto, or can already be instrumented,
auto instrumented using the traditional stuff, and obviously, on the Baylor docs,
you guys talk about instrumented for example, Java applications, is there something that I
am missing that there's an advantage to using EPPF in those situations, or is it more just
in cases where you can't use that auto instrumentation, because you can't get the agent in.
Great question, so the short answer is, for Java, it makes more sense to use the Java
instrumentation agent by the Open Telemetry project, because that's a feature really,
by the JVM where you can hook in, get a lot more details, a lot more internets, and so forth.
So if you have Java, and you have the chance to attach the Open Telemetry Java agent to your
JVM, then that will give you more insights than Baylor instrumentation. There are a few cases where
you cannot do this. The obvious one is modern Java, you can compile to native executables,
so if you do that, then there's no JVM, and then you are lucky that you have Baylor as an
alternative. The other thing is a little bit more subtle. Usually, you cannot have more than one agent.
I mean, you can't technically, but agents prevent that, because they're conflict. So if you
already have whatever some proprietary agent attached to your JVM, and then it refuses to
let the other agent attach as well, there might be situations where all also EPPF is the only way out.
But if you're able to just attach the Open Telemetry Java agent for the specific case of Java,
I think it's the best option.
That's really great. I'm so thank you.
One, three.
Okay. First of all, it's great to great presentation. Thanks. I have a question about if I don't
want to propagate the spans forward. For example, it's a street called. If you start to mess up
your headers, you need to also try to adjust the design things and so on. The designs of the headers
and so on. For example, I want to match the size of how long did it take from and to end.
So from request to response. So can it be used only for just creating that the mapping goes
guys like request and response without instrumentation and just put the spans out to some systems to
track those.
Sorry, can you repeat the question? It's really hard to hear from here.
Okay. Is it possible to map the request and response without propagating the
response inside. So for example, it's a street calls. I want to just just track the amount of
time which I spent in the history. And for sure, I don't want to have the mess up with all
headers because they're signed because there is a very signature. But I want to map the zero
instrumentation for tracking from request to respond time just to know how much it took and I would
put it to some database to map it.
If I understand your question correctly, you asking if you can map the request and response
without propagating context. Is that right? Yes, with a few caveats. Only in the say, I would say
only in the same node or because Bailey's what the agent is what keeps the context. So it
correlates the request and response. It has the data. But if you are shipping a request to a
different, followed or different node or whatever, that has a different agent running there,
then you would need to somehow transmit that information. That's when this comes into play.
If that answers your question.
Yeah, and by the way, can also map the data from different nodes. So we have, for example,
the by by IP headers like reports and all that stuff. So if you have data from different nodes,
yeah, then you need to complex propagation. Otherwise, you need to, as far as I know,
you need that value Yeah, next.