Let's start it. Last talk for the day is Gotham, who is a long-time permutacy member and
a lot of other things and who learned yesterday evening that he's going to fill in with
his talk. So thanks for doing this and welcome Gotham.
Hello everyone. I'm Gotham. I'm a permutous maintainer. I've been a permutous maintainer
for like seven years now and then I joined to Fana as an engineer, bidder all of the
permuties, a lot of engineering for five years. Then I got bored. Had a little bit of a crisis
so I became a product manager. I did that for two years and realized I still don't know
what product managers do. So yeah now I'm going back to engineering and this talk is
me kind of struggling with the whole. I'm not writing code. I'm not doing anything useful.
So I decided to tinker with my home lab. Yeah. And I found out that yes, I did find out that
I was doing this talk yesterday. So I wrote the talk today. So I have no idea how long
this is going to take. But I hope it takes long enough. Okay. Let's go. So home lab monitoring
with EBPF, but what do I mean by home lab? You know, I googled it. Can EBPF help your home lab
from blowing up? I don't think so. I hope not. But then I asked Gemini and it generated this
image, which is kind of close. And then I decided to use good old Google, which showed
me this image, which looks really, really professional. Turns out my home lab is
nothing like this. Unfortunately, I don't have pictures of my home lab given I only
found out yes. But it looks something like this in a random bookshelf with a lot
more wires attached. Sometimes, as in right now, it also looks like this, where I just
have random boards at random places just opened up. And I just do lazy to put it all
back together into something neat. Yeah. So I run a lot of different boards, I run K 3 S on
these boards. This talk is not about the home lab. This is about monitoring the home lab.
But what am I actually monitoring? So I am self hosting a few applications. How many
have you here? Self hosting applications? Wow. Holy shit. You are going to love this
talk. Yeah. So self hosting is basically you run the services on your own servers, on
your own infrastructure, and you are not going to pay like big tech or anyone else to
kind of provide you a service. You also own all your data. Yeah. Like I, as I said, I wanted
something technical to do, because I was kind of having a crisis. So I decided to self
host a lot of the applications. And what are these applications that I am self hosting?
They are kind of all kinds of applications. There is a nice website called awesomeselfhosted.net,
with all of these list of applications. Yes, I did not read it three times. But I did. Yeah.
Yeah. So I am using something called beaver habits to track habits. I am through a shame
of how bad my habit tracking is. So I am like, okay, I am not going to show it, but here's the demo.
This is a service written in Python, Django, and something else. And then I use this is a screen
short of the bookmarking service that I use. So the cool thing about this is it not only bookmarks
the page, it also downloads it onto the server so that if the page goes away, I still have a copy
of all the cool articles that I read and I can search through them. And I can use this to circumvent
pay walls and share articles that I have access to with my friends. So basically, I copy the article
on to read it and then I share it with my friends. That way they don't need to pay for all the
services that I am paying. And I also share super, super legal files with all my friends.
Yeah. And the best is this thing called use memos. So I used to have Twitter and I used to use it a lot
and then one day I deleted all my social media. And then one I had really amazing idea that I
wanted to tweet out and I was like, okay, what if I had like my own private Twitter that nobody
reads, but like in like save all the interesting ideas that I have. And this is what I write.
But yeah. So I run all of these services. They're mostly in Go, but there's some in Python,
some in Java, like varied set of services. They're run quite well. And one of the things I want to
give out a shout out to a stale scale. Even though all of these services are running in my office
or like in a room somewhere, I can still access them on my phone and on my computer and I can
like act as if they're all on the internet. So that's really cool. And you can also use stale scale
to expose your office lab, home lab services to the internet. For example, this QR code will give
you the link to read a service that I'm self-hosting the bookmarking service for the instructions
to run tail scale on Kubernetes, for example. It's going to be super slow because you're not
apparently supposed to use it for anything useful, but it works. My friends don't need to pay
for all the stupid shit I do. Cool. So I'm running all of these services and I'm a PM and
I don't promise this export. So I was like, I want to monitor all of these services and I want
to have good alerting on these services so that when they're down, I get page. I want to be on
cotton. You see. So what monitoring am I particularly talking about? Like is it like monitoring
the infrastructure like is there enough CPU? Is there enough disk? Is there enough RAM? Kind of thing?
Actually, not really. Like this has been done enough infrastructure monitoring with
permutation to find out there's like a thousand tutorials that you can use. And I
you basically follow one of those tutorials. And I'm now monitoring infrastructure. That's fine.
This is not the cool part. What I really think is cool is I want to do application
observability, which is basically all these different applications that I'm running that I have
no control over like source code. I mean, I have full control over the source code, but I can't go
go and tell them, hey, you know, you need to be on call for this service. So have good metrics in
alert. So yeah, I want to be able to understand when my journal that holds all my stupid stuff
goes down. Like I want to make sure that this journal is responding quickly. All the services
that I and my friends depend on are working. Yeah, and I want to be alerter than they're not working.
Yeah. And one of the reasons I wanted to do this was I wrote a blog post on why what I don't
like about being a PM. One of the things I don't like about being a PM is I don't get to use my own
tools anymore. It's kind of funny. I was reading this and I was like, I'm an idiot. Because it's
just like I wasn't on call anymore and I sorely miss it. Yeah. Yeah. Yeah. So I wanted to use all these
tools that I'm working on as a product manager and kind of be on call for this home lab service
that I'm running. Yeah. I want to have like fancy dashboards, like requests,
various durations and how long things are taking. Like I want to kind of be able to do that
for all the home lab services like beaver habits here. All right. So I've then looked at all my
services that I'm self-hosting and I found that 90% of the services they don't give a shit about
metrics like they don't care. There's no metrics coming out of these systems. These are these are
people who love building tools and they're built tools for themselves and I don't know how they
monitor it but then don't expose any metrics. The few tools that do expose metrics,
they have all kinds of different formats, different metrics names, different,
somebody was exposing durations in nanoseconds and I was like, who the tell case about
nanoseconds? But yes, they're completely different. So if I were to rely on those metrics,
it meant building one dashboard, like a dashboard for beaver habits, a dashboard for go cap.
You know, ten that each application had to have its own dashboards and had to have its own alerts.
It was like it was getting complicated. And finally, I preferred running doulang services back to
by SQLite and there is no Java agent I could just drop in and automatically I have good metrics.
But yeah, this is a whole light up for myself. I could have picked all the self-hosted Java applications.
So yeah, I mean, this was a good enough challenge and I have a few requirements for the solution
where I was going to pick. It needs to be very, very easy to instrument and I need to have one set of
alerts and dashboards for all the services that I'm running, like it's just one alert that I write
and then the alert when it fires it tells me what application is done. It needs to be Kubernetes
native because that's kind of what I'm running. And the most important thing is I wanted to give
conference talks, so I wanted to do cool stuff that I can block and talk about. I said this is
the most important thing because turns out I did not choose the easy to instrument option. I chose
the cooler option. Yeah, anyways, so for me to have one set of alerts and dashboards, all the applications
need to emit one set of metrics. So basically, I settled on the open telemetry semantic
conventions. There's a link to it here and also post the slides. So basically, the hotels,
semantic conventions, they tell you for databases, for HDDP, for GRPC, this is the metric that you need
to emit. This is the unit and these are the labels. So every application, if it follows the
hotel semantic conventions, we'll emit the same metrics and that way I can have one alert and then
basically one dashboard and all of that. So it came down to two solutions on how you can instrument
your services for open telemetry. One of them is the open telemetry operator, which is a
Kubernetes operator that you run. What it does is you say, hey, this is a Python service in an
annotation and it injects the Python auto instrumenting service, like auto instrumenting agent.
This is a Java service and it injects the Java instrumentation. So you get like in-depth instrumentation
without actually you doing the work. You still just need to tell it, hey, this is a Java application.
It, one of the, back in the idea, it has like one key language support, I say one key is because
compile languages, can't do auto instrumentation. So it doesn't support that, which means goal
language is out. Also for the languages that it supports, open telemetry is so early that Java
is great. Everything else is weird, like some languages do not conform to the auto semantic
conventions yet. So they still do milliseconds and stuff, seconds and things like that.
Having said that, if you use the hotel operator, you can add custom instrumentation inside the app
and then that kind of merges with the auto instrumented data and it's easier to extend the
telemetry that's being generated. Yeah, the two, one more drawback is you need to say, hey,
this is a Java service, this is a GoLang service, this is a Python service and you need to restart
the pod for the instrumentation to kind of happen. The alternate is Bella, which basically is an
MVP of agent that runs in the kernel. It looks at all the network calls, like HTTP calls,
passes it and it's like, okay, this application is making 500 HTTP requests and 200 of them are
succeeding and stuff like that. So it runs in the kernel, there's a help chart, it runs as a
day mindset. It only does not do in that instrumentation, it only does like request, like HTTP
GRPC requests, network level instrumentation, like which service is talking to which other
service and it does like CPU memory RAM. So it doesn't actually do like garbage collection
and stuff like that. But the good thing is it works for any language, including Rust and all of those.
But because it's running separate from the process, you can't actually extend the telemetry
being emitted in the process itself, like, you know, this is like two separate things. But you
don't need to restart or change anything, you run Bella, it hooks into the process, it hooks into
the kernel and it starts emitting matrix logs and traces in the hotel compliant format.
So yeah, I, there's two commands that you need to run to deploy Bella as like cool, I'm run this,
and that exposes prometious matrix on 1990 and I was like, I'm going to scrape this and I'm going to
send it to prometious. Okay, and then I looked up this dashboard made by my colleague,
the open telemetry service dashboard that uses the hotel semantic conventions to build a nice
dashboard that you can't really see, but like first meet really cool.
Yeah, I installed the dashboard, you still can't see anything because there is no data.
I was like, holy shit, what's happening? This is, uh, so I, and friends are there was no data,
because Bella lied to me. They said it is compatible in any Linux environment with kernel
greater than 5.8. And it said, oh yeah, you need BTF and BTF is enabled on most kernel, whatever.
In the Raspberry Pi kernel, there is no BTF. So I had to recompile the kernel with this particular option,
but it took me forever to figure out. So I said, like, you know, it makes, it makes for a cool
dog course. So I blocked the bottle. So this is definitely not the easy way. So yeah, I recompiled my
kernel and I blocked the bottle. And now I went back to this guy and it works. So live demo,
you can kind of see, like, you know, I used the service. Some requests to like P99 was like 700
milliseconds. You can see what endpoints are the slowest. You can see that get link metadata
was taking super slow, but there are no errors. You can see the logs, the traces, all of this
is kind of working because of this hotel service, open telemetry service dashboard. So yeah,
it works. So I decided to do the next best thing, which is alerting. So that when the service goes
down, I get a pinch. I get woken up at 2 a.m. because my journal is down.
Yeah, I'm going back to engineering. It's okay. And then I was like, no, I'm not going to do any
kind of alerting. I mean, just any kind of alerting, I'm going to do SLOs. Yes, I'm going to do SLOs,
as a SLOs or better. So I was like, I want 99% of the requests to succeed for all my applications.
I was like, this is the SLO. And if that doesn't happen, page me. Yes, this was what I did.
And I slept extremely peacefully because, when I was sleeping, I was not using anything. So
when I'm not using anything, even if the service is dead, there are no metrics, which means
it's 100% up. All the requests are succeed. Okay? So this is not good. But I was like, okay,
I mean, again, this is not a useful because, you know, when you try to use it, it'll throw a 500
and you get notified. I mean, you can see it's down. You don't need to get notified. Like, you know,
okay, fine. But there was a more interesting failure case. So while I was writing my blog post
on Recompiling the kernel, I broke the kernel. So the service was not running on the system anymore.
There's no system to run. And then there is no pot running. There are no metrics. And then
there are no metrics. It's 100% up. So I got no alert. So I was waiting. I'm like, yeah,
I broke my Raspberry Pi. I'm going to get an SLO. I got no SLO. It's like, fuck. So yeah,
so I was like, I realized the best form of alerting is black box exporter. So this is a very simple
exporter. So basically, you say, you say, hey, Prometheus just checked with the black box exporter
if the endpoint is up. The black box exporter calls the endpoint, HTTP 200. It's up. So if,
so basically, because Prometheus checks every minute, there's a request every minute to the service,
I think I do it every 15 seconds or something. And basically, black box exporter, if it's down,
it's like, okay, I can't actually open this website. I can't reach it. It's down. And I get an alert.
It actually works. Yeah, I also kind of use this thing called awesome Prometheus alert. It's an amazing
repository of alerts that you can download. There's a nice black box link. I just copy paste those alerts
and they work really, really well. And you can see it actually page me in this black box probe,
HTTP failure, that box probe failed and stuff. But I want to tell you something else that I do with my home lab.
So I use this thing called air gradient, which is an open source air quality monitor. You plug it in
and then it exposes Prometheus metrics. And every time I fart, I get an air coil. Hi,
Hi, we'll see alert. Yes, I was small office. And I was sensitive alert. It's okay.
But yeah, the TLDR is, if you're using a cello for home lab monitoring, I don't think it's very useful.
Unless you have a backup synthetic monitoring alert, I send it to my SLOT in this mean. Yes. Okay.
And I wrote an internal article about like all the things I loved about SLOS as I was trying
to create an SLOS. And again, I kind of like forked a lot of holes on some of the SLOS like I realized
I created bad SLOS and then I realized half the teams inside to find a created bad SLOS or half like a few of them.
But yeah, you do SLOS do need a backup alert. Like when, for example, if there's no pod,
you need an alert, if the pod is crash, looping, you need an alert. If you can't scrape something
you need an alert because for a SLOT fire, there need to be metrics and you need to make sure
there are metrics being emitted. Yeah. And now I'm going to rant about push versus pull because
open telemetry is push, put me this is pull. But I think it's relevant. We have internally
idealized as I was writing this internal rant on SLOS. Critical alerts on prompts
scraped failing. So I'm not able to scrape something pod crash looping and Google Google load
balance errors. For me, the most interesting thing is this prompts scraped failed. So if you're using
prompt years, you're running three replicas of a service, prompt years knows that three replicas
are running and it tries to get metrics from all three. And if it fails to get metrics from one of
those because it's dead or it's not like responding, it's it up equals down and we have a
prompt scraped failed alert. But if you're using open telemetry and push, there is no equivalent
of this because there can be 10 replicas running. Two of them are like dying and throwing errors
to the users. But for some reason, if they fail to send any requests, you're still going to be up.
So I don't know how to solve this. But this is also the big advantage of pull. So use
for meetings. Yeah. So that's basically, yeah, now I have alerts that work, which is black
box, it's border, SLOS are shit. You don't really work for my use case. Yeah. Cool. The caveat
is that bail is still little rough. Like, as I try to do this, I found some a lot of minor issues.
But I also found a couple of interesting issues where, like, you know, magically bail
was like filtering out a Python process because of some edge case. But the really good thing is
the theme is super responsive. I mean, because I'm DMing all the members saying, my home lab
monitoring is not working. But they fixed a lot of these. But I like, yeah, it's still super early.
But I think what they're building is really cool when you run one command and you have consistent
metrics everywhere. Yeah. The next experiment in my home lab, which runs in my office,
it's office lab, is going to be this cool thing, where Ali Baba, Data Dog and Kesma, they did this
cool thing, where if you're using go, you don't need to instrument anything in the code base,
and you can kind of run in with one command. So it walks the whole code base, injects instrumentation
wherever required, and then compiles it. So you get the same benefits of, like Java, agent,
like auto instrumentation, even with go. Now, this is new, this is not the easy path to do, mainly
because every time there's a new release, I will need to manually compile this, each application
with this special command, and then deployed in my own Kubernetes cluster. I will need to write
all this automation, but that will be a future blog post and talk. So yeah, that's the next
experiment. Cool. Thank you.
Yes, questions.
Thank you for the talk. I wanted to ask you to elaborate more about push horses, pull,
smithx, for metrics, because I need to roll the parameters and just use the pulling.
Now, I'm learning more about open telemetry and investing more time into it, and that was the biggest
red flag initially that I'm like, this doesn't make sense, this is going to cool issues.
Yes, it is going to cause issues, so don't use skillet.
I think it's like a fundamental trade-off, it's much easier to implement push, because you can push
anywhere, but with Prometheus, you need Prometheus to go and scrape all these applications.
So it needs to run in the same cluster and stuff, so like, yes. So again, in Prometheus,
we're working on implementing the best of both worlds. So the big advantage of push, pull,
or, for me, is it knows how many replicas exist, and that they should be sending metrics.
Now, you can use open telemetry to push, and if you know that there are 10, but you're only
getting metrics from 8, you can say those two are down, and that's something that we're
implementing in Prometheus, like kind of combining the best of both worlds. But today, you need to be
kind of careful and have a lot more alerts to make sure every single application that's supposed to
be running is sending you metrics. So yeah, we're going to fix it the easy way, an implement
the equivalent of up equals 0, but until then, just be careful.
Yeah.
More questions? How many of you are going to use bailar to instrument your home lab now?
Okay, cool. It actually is the funny thing, yes, you don't need there. You just need black
box exporters, but what you can use bailar to it's cooler. You can do another blockpost or
leg conference, talking out yet. All right, then thanks a lot.