My name is Roman. I'm a software engineer. I like go, play in video games and listen in
podcasts. I leave in Ukraine, in United Kingdom, in Austria, and now I'm living in Poland.
And in open-source, I've created things like click house data source. The first click
house data source for Grafana and some balance in proxy for click house as well. And for
last few years, I'm working on Victorian metrics. So, Victorian metrics, it's an open-source
solution for monitoring. It collects metrics, processes metrics, provides interface for
in Grafana for clarity in this metric. So, if you didn't know about such a project,
visit them on GitHub, it's open-source, Apache to license. Yeah, please take a look.
And what this talk is about? It's mostly inspired by experience of maintaining the project
and especially open-source project. And it's about complexity of distributed systems.
It will be useful for people who like or at least did try to make dashboards in Grafana
and of a mirror with Grafana and Prometheus. Okay, so let's go.
By the way, disclaimer, all of the content in this slides are generated by a human, not by AI.
Rod map. He provides a transparency for all the people to be able to use this project.
Like, for example, Prometheus is very good example of good open source project. It has all of this.
And if you didn't put a star on that GitHub repo, I encourage you strongly to go and start
Prometheus. But this is how it usually looks like in real world. There could be a brilliant software
solving many, many problems.
Scode there may be adds a readme file for 10 lines of code, like how to start it and that's all.
And maybe this project is indeed cool and people will start using it. But if the project itself
doesn't have this transparency, doesn't have the commutation, users will have the more questions they
generate. And if there is no documentation, maintainer becomes a documentation of the project.
And this is kind of a bad situation because everyone gets upset.
The developer doesn't have much time to develop the project itself. He has to answer questions.
Users are upset because they don't have the answers quickly enough, so they likely to leave the project.
So what we need to do, we need to convert black box into some centransparent by adding all this stuff inside the project.
So users can answer on their own questions with help of documentation,
or with help with other community members. And then the developer and maintainer could do what
he wants to do, like, improve the code, add more features to fix box, etc.
Okay, so why this feature is here? Well, first of all, I find it title very interesting because
it has the world world monitor in it. And if you read it, like a shepherd and her dog, monitor
the ship. I really like this title and it's pretty elusive because you can think of
shepherd as a devops, the ship as a infrastructure or applications or services, and a dog,
a tool that shepherd uses to monitor all this stuff. And it's pretty cool, but now imagine
that you have not like 10 or 20 ship, what about 1000 ship? Will it be enough to have one dog,
to monitor 1000 ship? Maybe shepherd needs to run a distributed system of the dogs to monitor
all this ship. And this is like, like, imagine, we need to monitor 1000 ship, and now we need to
monitor distributed system of the dogs. And this becomes complex, very fast, and when I need to
demonstrate complexity of distributed system, I'm usually showing this picture.
So, this picture of Cortex, it's an open source monitoring solution based on the primitives,
and it has many, many moving parts. Like, at least me personally, I find it complex.
If I will have to run it, I will have to understand all this stuff. And if something goes wrong,
I need a way to understand where it goes wrong and fix it. And maybe I will not have much time
to do this. So, what we can do to help people who use this project to make it better?
Well, from my experience of maintaining the two metrics, we can do the following things.
We need to write good documentation. We need to instrument it with helpful logs and help
meaningful metrics. And we need to provide a learning rules and dashboards. And this is exactly
what we will be talking about right now. So, vector metrics is also distributed system.
It also can be compact. There is no a silver bullet. If you need to monitor hundreds of millions
of active time series, it will be complex. And there will be questions on GitHub as can why
something doesn't work good. And here's how we do with that. Well, one of the tools is the
Cortex for another sport that we provide. So, this is how this dashboard looks like.
It's a shift in with every distribution of vector metrics, like every user can install it
and gets all this numbers here. But does it actually explain to the user how
vector metrics works? Well, if you first time log in into this dashboard,
the only thing you can say about all green and all good, right? But okay, so what we can do with this?
If you will try to Google how to make a good observability, how to provide
understanding of the system for users, you will find some recommendations like
recommendation of red method by the networks. They will recommend you to
have at least three signals describing your system. And it's very simple. This signals are rates,
errors, and duration. For example, you have to monitor a web server,
web server accepts requests. This requests represent rate. This requests could fail,
so we have errors. And this request have latency. This is duration. So, if we put all these three
signals on the dashboard, we at least have some characteristic of the system,
with this can say if it's okay or not. The reset technique by Google called for golden signals,
and this is the same as red, but plus one signal saturation, which usually stands for measuring
the finished resources like CPU saturation, network saturation, disk saturation, etc.
So, we of course have the signals on the dashboard. So, vector metrics can accept
right? It accepts rates. It can produce errors, of course, and there is latency for
red request, for example. So, we have all these four golden signals on the dashboard,
but the question is, are they actually helpful? Are they explaining the system?
The problem of these signals is that they don't answer on the question, why? I'm having errors.
Why? My latency is so high. For user, the system still remains a black box. It doesn't
explain anything. It only shows the problem indicates it. So, what can we do? Well,
Rapana doesn't provide, isn't a troubleshooting system. It doesn't provide you all these tools.
So, we remet it with the technology of our time, but let's let's try to do something,
and we will try to utilize this info buttons on the graphana panels. So, in the term metrics,
cluster of the board, we heavily use this help tool tips, and they're mostly next to every panel,
and here's how they look like. So, when user clicks on the info button,
there is a description, what this panel means, how to interpret it, interpret it. Sorry.
For example, the lower the better, and you can immediately say that is it good or no,
and then there is extra description like what you can do if it reaches the threshold,
whatever, and for the explanation given into the real mechanism, how this thing works inside.
You can go to the GitHub discussion, you can go to the documentation, et cetera, and understand
something how it works. So, the troubleshooting guides it, we build, using the dartboard,
looks like this. For example, user opens a dartboard, he sees this first of the signals,
and there is a normally with that signal. So, in this way, in this example, we have data points
in direction right, it shows deep anomaly. What user should do? He should click on the info button,
and there will be explanation what this panel means, and it says, hey, if you see problems with this
graph, please go check the insert metrics. And the insert metrics is just another section on the
dartboard. So, user can go to the very insert section, open the panels, and there will be
probably another anomaly. In this case, there is a storage connection situation panel,
that shows the anomaly. So, user can click on info button again, and there will be explanation
of what this panel means, and what needs to be checked. So, it suggests that when insert talks
to some VM storage, this is a state full storage, charging vector metrics, and probably you
need to check either the situation resources of VM insert or VM storage. So, we follow the advice,
and we check the resource situation of the VM insert. And we see that actually, there is no
increase in situation, it's opposite. VM insert resource usage goes down. So, probably it's not
VM insert full. So, let's continue following the advice and check VM storage metrics. And the next
anomaly of VM storage, explaining that, hey, that's probably caused by background merge,
you need to check the situation of CPU and IO. So, following this for steps already gives some
understanding for the user, that it's not only writes into the black box, that inside the
reason VM insert, and VM insert talks to the storage charts, and one of the connections to the storage
chart was saturated. Something with this one single connection, or with one single storage chart.
So, this kind of gives a context, a direction where user can take a look.
Yeah, so that's the idea, is to hide the complexity of the distributed system, hide the complexity
of the code and metrics itself, you don't need to run arbitrary expressions, you're just looking
on the graphs and following the devices, and you can indicate the malicious agent behind this.
Okay, so we maintain this dashboards for a couple of years, and I would like to share the top
cool features that we started to use. With this time, maybe you will find it useful from the
practical perspective. So, first is, of course, this helpful tips, I encourage everyone to put
them on your panels and explain in a prompt way what the meaning. For example, we have some
notion called slow queries, you may not have idea what it is, but the graph itself will
tell you that it probably depends on this common line flag settings. If you want to change it,
somehow, it is very easy to find it. What else? We do not show all the resources, all the components
that we monitor, we show only outliers. So, the term metrics cluster is a distributed system.
It can be small, like three instances, it can be very big, like 100 instances. If I would like
to show the memory usage of all the components, if I would put 100 of them on this graph,
it will be a mess. I can't read it. I'm just a human. So, what we do here is the two things.
Well, first, we show relative usage of memory. It's a percentage usage, because different
components can have different limits on memory or in CPU or whatever. And also, we not show
in every scene, we show the outliers are components. So, we show metrics consists of three
component types, and we show only those who consume the most, only those instances that consume
the most of the memory. And looking at this panel, helps me to understand pretty quickly, if I'm
okay, like if something is 90%, I'm probably not okay. And you don't need to check 100 lines,
you know, the tone distance that you are not okay. What you can do next when you found the outlier?
Well, there is a very cool feature in Grafana. You can click on this panel on the line, and
it will take you to another panel that will represent this metric in a different perspective.
So, in this case, we go from the relative representation of outliers to the absolute
representation of all the components. This feature is super cool, but you know what is
bad about this feature, anyone knows? Do you know that this feature exists?
Yeah, this feature doesn't exist, and Grafana doesn't, sorry, it exists, but you don't know about
this, and Grafana doesn't help you to show on the panel that you can actually click on the line.
So, I would think maybe we can enhance this, and I created a feature request to Grafana,
which about it's create an alternative use for a panel. So, for example, I want to show CPU usage,
and would be nice if I can click if I can switch between two modes, a relative mode,
an absolute mode, and this will be a display in the title of the panel. So, if you like this proposal,
this is a ticket 9, 9, 8, 6, 1.
Okay, let's go next. The geometric components are mostly configured with common line flags,
and we try in very hard, so user don't need to configure that. We have this meaningful default,
with optimal work of the components, but there are always some corner cases when something needs to be
tuned up. And usually this corner cases is the main source of misconfiguration, of course.
So, in order to spot instantly, if the reason is configuration, we expose every common line
fact in form of a metric, and this metric called flag, and it shows you the value of the flag,
the name of the flag, and if it was overreaden by a user. So, we have this panel with a non default
flag, and I can instantly see that someone said like they want to select 250 million of unique
time series per request, which probably could lead to problems with memory usage when you do this.
What next? We have a user notation, so we expose the versions of the component in the form of a
metric, so we then can build an annotation query, which will show on the graph when version of any
component has changed. And it's like an operator. If you see the annotation on the graph,
and something went bad after that, some performance degradation, you don't need to think, you just
roll back, and then you investigate the something, what actually caused that. We also use annotation
for restart, and restart could happen in many cases, but it could be also out of memory exception,
if something crashes because of lack of memory, you have to take a look on that, or if you change
the common line fact, you also had to restart it, so probably the user changed configuration when
they did this. What else? Of course, the term metric components produce some logs, warning errors,
and etc, and we expose that in the form of metrics as well. And even if I don't have access
to the logs of the components, I still can check them on the panel in Grafana, and they have
this label pointing to exact line of the code, which produced this error. And since this,
the source code is open, I can just check which line of the code produce it, and I can guess
what is the error without having to take a look on the logs itself.
Yeah, so all this dashboard goes out of the box, and we recommend in our best practice
system to use the dashboard, and we encourage everyone to use it, and we also ship it with the
alerting crews, which we think are very helpful to understand if system is okay.
This is how alerting crew looks like. It contains the similar context, the summary and description
point in what the problem is, and how to solve this, and also they contain a link to the
dashboard, the same dashboard that I showed you. So if you use our alerting crews and our
dashboard, and you receive the alerting crews firing, you can just click on the link, and it will take
you to the panel, which explain what's happening here, and you will get the context what happened
before. You will have the information tool seep with recommendations, and yeah, I think this
interconnection between alerting crews and dashboard is pretty helpful, at least it helped me
to troubleshoot it much faster than before. All this is also depends heavily on the documentation.
I ran this command two days ago, and it says that we have 90,000 lines of documentation in
the project itself. It doesn't mean that we add documentation every day. We constantly refine it,
trying to make it clear, but yeah, this is what it takes to describe the distributed system.
And when we accept pull requests, we also require from user not only to make a cool change
in test. We also require to make a documentation change, because if you introduce a new flag
or change the behavior of existence, this should be reflected in the documentation.
Because otherwise, it will go unthink pretty fast. And other features that we use in the
documentation, they'll start it using not so long ago, is the version label. So when we
introduce a new feature, and we're mentioned to upstream our documentation render sit instantly.
And some users when reading through the documentation, they can find the feature as it doesn't
still exist in their version, at least. So we have this macro in markdown that's
called available from, which points to exact version when this feature was introduced, and
user can get from the documentation when this flag was added. Of course, the result automation,
you don't need to put exact version, you can put a placeholder in it, and then on release,
it will be automatically replaced with the actual version for you and published with the latest tag.
Okay, so as I said, we cross-refer our documentation to keep it fresh. We'll use the
documentation links in our dashboards, in our other includes, in our code, in every public platform
where we help users to answer their questions. And this is what helps us to keep it up to date.
And we also do really care about broken links. If something was answered on the GitHub
like three years ago with the documentation link, it still should work. Amen right now. So if
you find any broken links in our report, just let us know. We really try and to keep them
alive. And yeah, we do care about all this that I showed. All this is available in open source,
and we really want you to use that. And we really do care about this because we use it every day.
We do put the same dashboards and alerts and calls internally. So the term metrics provide the
enterprise support to customers, and those customers can send data to the laboratory to our
cloud. So we basically receive in the same metrics of the vector metrics component to our cloud,
and then we can reuse the same dashboards that I showed you, the same alerts and refer to the same
docs. And this is how it usually looks like. We have support engineers who receive the trigger
alerts. These are software engineers, the maintainers of the vector metrics itself. So when
they receive an alerts, they can immediately see if this meaningful alert, if it's a false positive,
or not. And if it is a false positive, maybe it needs to be changed in the upstream.
Then they go to the dashboard and they check the same panel, and they also can apply any
modifications to it, maybe make it more clear to change the description, etc. Then we apply changes
in the upstream in the first place, and then from the upstream it goes to the internal system.
And yeah, then the system back again to the green state. So this is why it's called
monitoring of monitoring, and this is how we're used our own dashboards and alerting tools.
Yeah, so you can check our dashboards in our public playground. All this again is public. You can
try it. You can see all the descriptions. You can play with expressions. You can check our alerting
tools and documentation. And that's it.
So thanks a lot. Are there any questions?
Yes, there's a question.
Thank you for the brilliant talk. I would like to ask you about, I would like to ask you about
monitoring of monitoring, actually, where some kind of harbids or something like
can show you that monitoring is actually broken.
For example, if we have automations that shows that monitoring is broken,
yeah, well of course, like if you check our best practices, the commutation is such as that
every monitoring should have a monitoring of monitoring. So basically we're on two
these sort of metric systems that monitor each other, like cross monitoring. If something dies,
there's another system we'll let you know. Is there one more here?
And do you then have also different infrastructure for these two instances? Because we have
at the moment this problem that we would like, for example, Loki with an object door back end,
but Loki is also collecting lots of our object door solution. So we don't really want to
depend all Loki on the thing that Loki is monitoring.
Yes, so the question is, do you need to have a different infrastructure to cross monitoring?
Yes, that would be the best way how you can do this. They need to be independent for sure
if you can afford that, if you can do that in front of me. Yes, I recommend to do this.
But you also can use like, um, kills which mechanism if I'm not mistaken, like if,
yeah, you're right, basically, but there are cheaper ways to do this to at least notify you
that you're monitoring is broken. More questions? Yeah.
When you had the list of errors, a number of errors or warnings in the logs,
could that cause an issue with cuttingality?
So the question is, if I have many logs with errors, does it cause issues with cuttingality?
No, because the number of lines with the errors is limited, it's finished. So if we count the number
of unique logs, it probably could be 100 at atmox, but I believe it much lower, so it doesn't
go. It just looks dangerous, it is not in reality. Anyone? Oh, up there?
Hi, thanks for the talk. I really like the section with the tool tips on the dodge parts,
but to be honest, I would never want to thought of this myself, and I guess my question is,
once you first started getting users onboarded on this dodge part, and you're
building a general, how did you make sure that these features and this documentation is
discoverable enough for people to actually RTF them? Thanks.
The question was, how do I find that people actually use the information tool tips,
and if they are helpful? Yeah, so we don't have any way to know that, but we have a lot of
question on the GitHub and public platforms. And here's how we can make our support better.
So we first think that we ask, is can you give a screenshot of your graphana? And we also have
troubleshooting checklists that we also have a link, and we share it with user, and there is a
steps like what you need to do, like check this panel, check this panel, read the stuff, and
yeah, there is no clear way, but this is how we usually help people with let them know that
this is the list, and then community is supposed to take it off from there.