Thanks for coming to my talk.
Thanks for staying for so late.
There's just one talk after mine, so keep going.
This talk is going to be about this really tracing and suicide swift.
And as my previous talk, this also originated from Google
with some of code.
But in my case, it was back in 2020.
So you can see me pitching to the swift forums.
And that's when I started working on Swift's stupid tracing
with my mentor Konrad.
And since then, I've been a commentator of the Swift's
stupid tracing library.
And I also created a couple of spinoff projects,
including an open telemetry tracer.
Your general for today is I will give you a quick overview of Swift
of Swift observability.
Then we'll take a quick deep dive into Swift's stupid tracing
the library itself.
Then I'll show you how to make tracing even more
exciting if you pull it together with logging.
And then we'll take a quick look at open telemetry
and Swift's hotel.
And then a couple of next steps.
So as Tim mentioned earlier, there's three signals
in observability and Swift.
Currently there's logging metrics and then
last but only this real tracing.
Throughout this talk, I'll be using a fake server
that is a product kit, not API.
It's written in server sites, Swift, of course,
using the hummingbird project.
And you can see there's just two end points
one to get all products and another one to get a single product.
Here we can see.
This service running in VS code.
And by default, this is as much observability as we get.
We can see the server started, but after that we're kind of on our own.
So let's gradually add more observability to this.
First up, we have logging.
Logging is way to record specific points in time.
And you can attach more information to those points
through metadata.
But the downside is it's kind of hard to understand a full request.
So for example, if I fetch the products,
I may log when I start carrying from the database,
but I may not know how it turned out.
So add logging.
Some additional logging here.
What I'm doing is I use hummingbird's log request middleware.
And this already gives me additional logs.
And see, so for each request I'm sending a guide log statements
and includes a metadata like the path and request method.
We can also add our own custom logging.
For example, here logging if the post-crescue request failed.
And which it did in this case, unfortunately,
and we got a log statement for this.
Next up, we're going to add metrics.
Instead of logs, metrics are aggregated.
We don't, again, look at a specific request,
but we rather look at the system as a whole.
And this is great for providing high-level overview
of how our system is doing.
And we can build stuff like alerts on top of it.
To add metrics to hummingbirds,
I'm just adding the metrics middleware
and behind the scenes I also set up
the Swift metrics library to export metrics for parameters.
So in this case, you can see there's the rate
of active requests per endpoint.
And I just hit the endpoint a bunch of times.
That's where we saw a spike.
And then it went down again.
And now let's get to the actual topic of this talk.
Distributed tracing.
So that's the third signal that we're adding to a product catalog API.
Distributed tracing is again specific to one request.
But as opposed to logging, it kind of gives you an overview
of the entire lifecycle of that request.
So you can see each operation that makes up this request.
You can also see both high-level, like Span A in this case,
is a high-level operation.
But then you can also get detailed and see which sub-operations
this is made up of.
And you can also tell how long each operation takes.
So this is great for detecting bottlenecks, for example.
If you see that stuff is running in sequentially,
instead of concurrently, for example, you may want to shift things
around to make the overall request faster.
To add tracing, you can see the pattern now.
And again, just adding a middleware behind the scenes,
a set of tracing to export to a two-coloured digger.
And here is digger in action.
So here we can see a single trace.
That shows that I hit the specific products and points,
and if we draw it down into that,
we can see the span that was created.
And it includes again metadata like the P method.
We can also create our own spans.
So in this case, we just wrap our post-gres skills,
statements in another span.
And then this span, because it runs inside the request handling operation,
as a child span of the one that I've been created.
So in Yager, we can see that we started with the products,
and points got a span for that.
And then we switched over to the SQL query span.
We can also add additional information to our spans using attributes.
For example, here I included the SQL statement that we actually running.
And we can also see this in Yager.
For example, here I fetch a product that didn't exist,
and in the SQL statement span, I can also see the SQL query.
In this case, this was still all running within one service.
So that's not really distributed tracing.
Which is fun, but it could be better.
So here I'm calling out to a different server.
In this case, an open feature-based feature-flegging system.
And I'm just checking a feature-flegged to see if I should fake a failure in my server.
And then if we hit the endpoint again, now stuff is getting interesting.
So here we can see that I still started with my products and points.
But then I'm calling out to this thing called flipped,
which is the feature-flegging system that I'm using.
And we can perfectly see how much time and when we call out to this service.
To take things even further, this is now integrating the product catalog
in an entire microservice architecture.
So it's not just the product catalog service anymore,
but it's still part of this microservice.
And in this case, for example, there's a couple of microservice involved.
So we start by putting something in our shopping cart,
which then calls out to different services,
which in turn call out to product catalog,
which calls out to feature-flegging.
And then on this slide, we can clearly see that after all this initial work is done,
we then, from the API, call out to this recommendation service,
for example, to get recommended product based on what we put in the cart.
But if these two would take up a bunch of time,
then maybe this could indicate that we should figure out a way to parallelize this instead
to speed up the overall response time.
What we also get is a nice service diagram.
So this clearly shows us the dependencies between our services.
So here we can see who's calling the product catalog.
Now that you saw how distributed tracing works in action,
I'll take a quick deep dive into the Swift distributed tracing library itself.
If you've ever worked with Swift log or Swift metrics,
this is pretty familiar to you.
It also provides just the interface,
and then there's implementers that actually implement the trace server that is being used.
Same as logging in metrics, it has three target audiences.
We have library authors, instrumentation authors,
and application developers.
Let's start with the library authors.
These are the ones like coming bird, for example,
I think I should peak line post-gross Neo.
And their job is to start spans where it makes sense.
So for example, hummingbird starting, we serve span.
They don't know or care about the specific trace that is being used.
They just use this abstraction provided by Swift distributed tracing.
Then instrumentation authors, on the other hand,
they care about which specific distributed tracing system is being used in the end.
But they don't start spans themselves.
They just take care of exporting those spans to the system.
And then last one release, we have application developers.
So they are the ones that choose a specific trace or implementation.
Then they just use libraries that have built in support,
so that could be adding hummingbird's middleware, for example.
And then optionally, as you saw, we can also create custom spans,
but that's not required to get distributed tracing.
So I've talked a bit about the trace of protocol.
And this is what instrumentation authors would implement.
If you can see it has a copper requirements, first of all,
it needs to know about the specific type of span that this trace occurs.
And it most importantly has the ability to start spans.
And then whatever a span is might be different in each specific implementation of a traceer,
but for library authors or application developers,
this specific type of span doesn't really matter.
Speaking of spans, spans can be mutated,
but only until they're finished.
And you have to end them, like finish them,
by calling the end methods in order for them to then be exported.
So you start a span, for example, for the postgres query,
and then at some point when the postgres query ends, you also end the span.
So you have to start in an end date,
and at that point, the specific trace that would export it to a tracing backend.
Spans are also uniquely identified, where I think of the service context.
The service context is basically a type-safe dictionary,
there's nothing fancy in there.
And then the specific trace implementation puts the IDs in one's to use in there,
so it could be a span ID and a trace ID, for example.
The service context is not in a task local.
And this is how we get automatic parent-child span relations.
So for example, on this slide, I'm creating a span called parent,
and then inside, I create another span called child,
and this is using task locals under the hood to just give me a,
to establish this connection from parent to child.
Again, this works great for local,
but if we actually want to take tracing distributed,
then we have to get the service context across boundaries as well.
So it could be async or process boundaries.
So if we take the example of a client server from earlier,
this means that the clients, when it makes a request and creates a span,
it needs to inject these identifiers into the HTTP headers,
and then the server needs to extract those,
and then the server creates a span based on this restored service context
that was propagated.
This propagation is done using the instrument protocol.
The instrument protocol is inherited by the trace protocol,
so if you're building a trace protocol yourself,
you also have to conform to these methods.
The instrument itself is acknowledged about the thing that carries it,
so in the client server example that were HTTP headers,
but it could also be different kinds of metadata.
What it does know about is the specific values that it wants to transport.
Then let's take things another step further,
and integrate tracing with logging.
So this feature is built on a thing called metadata providers
with log, and basically what a metadata provider is
is it goes from void to logging metadata.
In case of tracing, we use these to then transform the task local service context,
so we can just pick up into specific log metadata.
That allows us to associate log statements with specific spans and cross reference.
So here we can see this in action.
So just by choosing my metadata provider,
I get these metadata fields in my log generated by hummingbird,
and then I can just clearly see if I search for a trace ID,
which log statements belong to this request.
And then I could also take this trace ID,
take it to the eager, look up the particular trace that this belongs to.
Then let's briefly talk about open telemetry.
It's an open observability standard.
It supports logging metrics, distributed tracing,
and hopefully some profiling as well.
It has this thing called the open telemetry protocol,
and this is how different components
that speak open telemetry's language talk to each other.
And it's supported by a variety of observability tools.
So if you have something that speaks auto,
you can pretty much use any observability back and you want.
Speaking of, if you do want to do that in Swift,
we have Swift Auto,
and Swift Auto provides these OTLP exporters,
currently for distributed tracing and also for metrics,
logging is in the works.
And it's done by being an instrumentation author,
so it implements the Tracer protocol.
So it knows how to start spans in an open telemetry format,
and then it knows how to export those spans
to an OTLP compatible background.
This is how we can bootstrap Swift Auto.
So here we can see,
which has created HGRPC SpanExporter,
which means that we sent these spans via OTLP over GRPC.
To the observability back ends,
we have a bunch of other stuff in the set out,
like batching the spans to not export them,
as soon as they ended,
but first trade a batch and then export their batch,
and then we just create our auto Tracer,
and then this last line is very familiar
to work with Swift Logos with metrics,
that's where we just bootstrap the system,
and at that point,
everyone that creates spans,
including libraries that you're using,
but also the expense that you create on your own,
will start using OTLP under the hood,
without needing additional libraries,
like hummingbird, open telemetry, for example,
or postgres, open telemetry,
it's just postgres and hummingbird supporting Swift
distributed tracing as the abstraction layer,
and then you as the end user say,
I do want to do,
I do want to use the open telemetry Tracer,
and that's how it works, as easy as that.
Next steps,
I quickly teased the logging integration in Swift Auto,
so we do have the metadata provider,
which puts trace and span IDs inside your log statements,
but we want to go step further and also export,
because open telemetry supports logging metrics,
and distributed tracing,
we want to export with log statements via OTLP as well.
Then we want to roll out Swift distributed tracing
into more libraries to get even more of this metric for free.
So, for example, earlier,
for me wrapping the postgres statement myself,
instead it would be nice for postgres,
and you would have built in support with Swift,
used to be the tracing, so I don't have to do that anymore.
Same for the Async HTTP line,
but at least for this one,
I already created a poor request,
so check it out,
and then finally we want to get to a Swift Auto 1.0,
which will then include log exporting as well.
Thank you so much for listening.
I have some handy QR code here,
if you want to check out Swift distributed tracing.
Thank you.
Anyone?
Last call?
Okay.
Can you put your slides to everything?
Yes, I will do that after this one.
And the example that I've been using
for Microsoft's demo will be on GitHub as well.
Let's say the question was
if distributed access and Swift already support distributed tracing,
they don't add this point,
but it's kind of teasing it.
So yes, it could work.
I played around with it in the past as well.
I don't know of future plans of distributed access,
but I feel free to ask Conrad about it.
And then so the action you had one?
X-ray.
Yeah, I do have a support library called Swift Auto X-ray.
Sorry, I have to repeat the question.
The question was,
This will be the tracing works with Amazon X-ray already.
And the answer is yes, I did use it a previous job.
And for that, I just had to create an additional support
library because the way X-ray does context propagation
is a bit different.
There are trace IDs look different from the standard
that's open-time metric users.
So like briefly go back to this set up.
Here we can see that I'm using the auto random ID
generator and the W3C propagator.
If you want to use X-ray, you just configure it
with an X-ray specific ID generator and an X-ray
specific propagator and then you're done.
And these two are implemented in the usual auto X-ray
repository.
All right, that's it.
All right, more questions in nature chat
or meet him in the hallway.
I'm sure he'd love a high five.
Great share color.
And thank you so much for the detailed talk.
I love when my two worlds collide.
I'm a recovering Kubernetes maintainer.
So it's always great to see cloud native crashing
with Swift.
Amazing.
Thank you so much.