Hello everyone, I'm Javier Lasse. I'm a software engineer at Reddit for a bit more than
three years now. And today I want to share with you a little bit of our story with
observability in the QVID project. We'll start with a bit of a background about
QVID. Dobs is a really really challenge that we faced. What was our approach to solve those difficulties?
A little demo on how we are trying to do things now and a bit of the lessons work.
So, for those of you that don't know QVID, basically it's a API and a runtime that you can install
over Kubernetes to manage and run virtual machines as native workloads. We are a no consort project.
We have over the five point eight thousand gate of stars more than three hundred contributors,
one thousand plus four and ten thousand yards. This project is vendor than adopted by
many organizations such as Reddit, Microsoft, Fire, Nvidia, ARM and many others.
Basically, QVID works with a lot of components. On the master node, we have, for example,
the main controller to manage virtual machines, the API to end over requests. But on each work where
we also have a native installed, that's responsible to end also the VM container.
Here we have a bit more detail. We have open names of these components. The idea here is to
just understand that QVID is composed by many small parts. What was then the challenge?
So, as the project progressed, QVID has more and users, more vendors. And obviously, we
those the requests for observability features and new signals kept increasing. In the beginning,
it was rapidly growing. And as many of the projects you probably work on in the initial stage,
we usually won't care that much about having metrics, right? We just want to develop features
that are needed. And sometimes, it will be nice to have this specific metric to track this.
So, let's just put it somewhere. Let's just make it work. So, we can track it. Let's just
step by step applying some glue here and there. So, we can have the necessary observability
stuff that appears at each time. So, we call it like the initial wild west approach.
We have a lot of components and usually teams responsible for each of those components.
And each developer would just go to the code and, for example, prompt use metrics.
As they just had the documentation and thought, okay, this seems like the correct way to do it.
Let's just start it this way. And, okay, it's working totally fine. And a few years after that,
this led to a high cognitive load. For example, when I start working on the project,
I focus more on the observability. Sometimes, people would just request me to have some
observability feature. For example, we want to add the stability levels for metrics.
And what would it take to do it? I will need to go across several repositories.
I didn't know where the metrics were being defined, how they were being defined. And it was
really hard across those projects. For example, to keep a mental state of how each of those projects
implement the observability signals. And alongside with that, we have a lot of business logic,
intertwined with the observability logic. Usually, the you just go to the file where we want to
track some specific signal and just start the logic to create a metric there and just define the
metric there. But that doesn't scale for when you have the dedicated observability team.
And also with that, we noticed a lot of inconsistencies in the naming conventions, the variables.
And as I said before, it was hard to add new features for across the repositories.
So our approach and this might be not the ideal solution for some of you. But we try to
mobilize observability in a separate package. And as I said, these are all the repositories
that our observability team works with every day. So what we thought was the best solution for us
was to try to make something strict that was did not provide all the flexibility. It is like,
for example, the parameters library gives, but would be the right fit for our project.
We submitted the initial proposal for the code refactoring. And the goals were mainly
to decouple the monitoring logic from the business logic. To encapsulate the monitoring
best practices and the patterns that we wanted to follow in this library and just have it as
the dependency for all those keyword components. This will mean also that it will be easier for us
to keep the code and the utility is easier to maintain and evolve. This will only need to
change them in a single location. And we could also have a structure and many tools to
accurately and use the generic, for example, documentation, linkeding holes, define, for example,
allow the nigh and opting lists for our customers. And all of these without having to change
the code in multiple places. We started with strict interface and dependency model. We want to
really limit what we've been across all those repositories and try to move most of it to our
to our dependency. So I don't know if you can see it very well, but here we have the operations
just to create the metrics. And the idea here is to have a list of all the metrics that are
actually being exposed. And we have something similar to the registration that promised
is library S. But this is very limited and it forces certain parameters to be passed and to be
defined. And this is all, for example, on the single the package, for example,
package monitoring, metrics, virtual controller. And then we have multiple files there. And you
can see in the bottom that the logic to create the metric and are all the information that we need
for the metric is defined also in this file. So this can be just responsibility of the observability
team and not intertwined with other not-so-collated code. Here we have a very similar example
about this time for the implementation of our collector. So in approach, we list of metrics
then the definition of the metrics and then the code for the collector. And the registration
of the metrics, all the metrics from this component are always registered here. They cannot be
registered anywhere else. So we have these central focal point where we can quickly see what
everything is being created. And we make sure that nothing that does not appear here might
be being created or something like that. So this is our single point of truth.
Here we have the link if you want to check later for the biggest PR. This was on the
core QB platform. The initial PR to move from the old implementation to the new implementation
also you know like some examples. This is the first one that we have on the initial
application file for the virtual operator. In this example you can see here what was mentioning
in the beginning. It's probably something that you might be used to see. We just define the
metrics that we want in the beginning of our file. We have probably a need to function that
in this case must register all those metrics and then during the code we are just populating
the metrics directly there. Here we have an example because on the goals I mentioned that
we wanted to be more careful with the generation of documentation both for the tooling and for
customers. Here in green we have an example of four metrics that were being created in the code
but since they were a bit all over the place they were not being picked up by the code
that was generating the documentation. When we have this new system that mainly over metrics
were registered in that single point, automatically we were able to catch those metrics and
add it to the documentation and provide extra value for the customers. I also mentioned a bit
of the valuations. Here on the photo valuations we have for example the ones that are also
provided with the promitius libraries but for our use case we wanted to have additional valuations
those you can see on those functions in the middle. They are incorporated in our observability
package. Then each project just can just create the test defining the custom valuations that
they want and then just call linked alerts and pass the list of alerts and expect no problems.
So we tried to, as I said in the beginning, as much as we could from each of the repositories
and tried to move it all to the operative observability package. This also provided as
ways to make it easier to write unit tests. Since the monitoring logic is now very limited to
those that monitoring package and we have strict interfaces for how to create those metrics,
we can just simplify the ways that we write tests. For example the collection for virtual machines
in for just received the list of VMs so we can easily define a specific list of VMs that we want
to track and just write very tiny specific unit tests for those behaviors that we want to track.
Previously many times we saw that the metrics didn't have any unit tests because people would just say
how we are just testing the business logic and the metric follows all that business logic so
it's probably fine that was the initial thought that people have about making sure that the
metrics were right. As I said in the beginning there was just trying to create them and go to the next
feature that the customers were looking for and this mentality only changed as the project to grow more mature.
And this also allowed to have more clearly bounded ownership over the observability code. Usually
incubate the for example the curcubate there's like six different teams for a single repository.
So most of the teams have very clearly scope permissions to review and approve code that
she's certain packages and just a few approvers that can approve for the whole project.
What we found in the beginning was that each time I was going to open a public list it touched the
packages from like three different teams so I will always have to ask the general approvers for the
project to review and approve the PRs and this usually took a long time because they were
always busy and the PRs always took as I said a lot of time to move forward.
With this approach we were able to simplify all these processes because everything that is
related to defining the values of matrix and testing matrix were just on our team and when
sometimes we needed to change something in the logic to pass more data to fetch more data from the VMs.
Then yes it made sense that for people that were responsible for that code to also take a look to
also verify that everything was working right. But if we want to change matrix accommodations
other new matrix from the data we are receiving in some collector that's more of our knowledge
than their knowledge. So it didn't make sense that they were forced to review that code and this
was a big advantage to our velocity. Obviously there were still a lot of challenges in the transition
to this new format. There was a lot of over-have. I think that first PR it was one of the biggest
but we had to open at least like two or three PRs across all those repositories and they were like
to eight probably so at least 24 PRs just for all these these effort. But we found that later
the velocity that we are able to move now was is worth all that initial work and some of
those PRs were also very very complex. The matrix were scattered all over the code. The business
logic was very intertwined. So we had to be careful with moving all that logic to the new package.
And one also our challenge was also advocating developers on the things that should not
do anymore. Like sometimes we saw that they were still creating like new matrix that I thought was
useful around those all the blocks of code. So we also created a winter that verifies that these
practices that I'm sharing here were being followed by them that they were not using
promitius Hesister, Muskhegeister calls across the code and even operator observability calls
to Hesister matrix and to define were all in being done in that specific package.
So now a bit of demo I created I created a new operator with operation SDK it's very empty just
to an API for the same cache those of you work with Kubernetes R and with operators should know
this is like a default example. And for example let's imagine that this project was being used
by some teams usually what we will do is just install that monitoring linker and try to see
where it will catch definitions of matrix that we don't like and that we want to remove.
So here we will consult the controller and see here it's just a handy minute function with the
metric inside the master register. So we want to delete this piece of code obviously it's not
being called I just started this year for example purposes but usually we want to remove and then
move to package monitoring under the metric sections that we will take a look in a second.
And then when we call it monitoring linker we expect it to be queen.
As I was showing on those small examples during the presentation we have the main setup for the
matrix and we want to register all the matrix that are defined across the specific files in this package
and also the collectors. We see that here the formatics we just you only have operators matrix
and just these custom resource collectors. Here we have the example for the operators matrix.
For example right now we have a reconcil count, a reconcil action and old count at many of
we are also what was the idea just for this example. The reconcil count is totally normal
metric as I said we just enforce like some fields and for example we added also these extra fields
I was mentioning the beginning that when we wanted to have new features we need to go to all
repositories to add support for all those features because for example these extra fields
does not come easily with the parameters library but is something that is on on the Kubernetes
code and just we just know for example to add the stability stability level so this is possible now
and we cannot for those tests on operator package the validation to make sure the stability
level is present and the values are in a given list of values in this case we can see that this
metric is in beta the reconcil action for example in alpha and this old count that I mentioned
was for example in deprecation and we can also state the version in which it was deprecated
because one of the problems that we also faced a lot of times was matrix with incorrect names
because sometimes developers would not use the best practices for parameters in terms of the
units for the metrics or the names for the counters and we wanted to rename those metrics but
a lot of customers were already using them so we added these new features to be able to state
for them what's the best ability level of the metric and in case it's separated the version so we
can later remove it and he plays it with a new metric here you have the example for the resource
collected it's very simple the only difference is we have a callback function and for the callback
function we can just access the a collector client variable that is able to for example list
for all the resources in the cluster so that we can count how many custom resources were created
and the idea is as I said was tried to have all these logic separated from the other logic code
so we can do this in a neatly tight scope and the same for the holes if you notice the
structure is very similar to the metrics you want to make everything as most consistent as possible
across the files and across all the repositories so that when we change projects even if it's a
storage project a networking project cork you will we know that everything is implemented the same way
it's easy to hit the environmental model and we don't need to spend a lot of time going all over
again through the code to see in this repository our things implemented and yeah we just have
the normal alerts not much differences here you can see that we are using the hook definition from
prometius and the same for the recording holes and we also add some utilities I've show you one
in the slides about the unit testing and here I'll show you that we also try to make it simple
to generate documentation the package already provides like a basic basic template to generate
documentation we just need to set up the holes and build the alert box and it would just
generate it to in this case by default it's just a mark one without definition for for the alerts
and all of its fields for example for the metrics I just I put the extra template
and here you can see that it's defining the a certain structure for the depacated version
and for the stability levels and here we just call view of metrics box with custom template
and we show that with something very similar and this it was actually useful later
because initially we just wanted this mark round that list of all the metrics but now we are trying
to feed the for example open shift systems AI with the metric limitations and we found that for
example mark round it's not the best input for it so it was really easy to just go to the
template and change these to to XML and just have all the tags we need with the fields helps
and even the labels we are working now also going through all the labels that metrics
have trying to add some more detail descriptions for what the labels mean the values they can have
so when later can go to those AI systems and request I want to track this information or generate
these dashboards everything about the metrics and their labels is documented neatly here
and it's easier to to return the correct answers for them
so a few of the lessons that we learned like obviously we should try to review technical
that early many times we think like okay I'm a good developer I'll take care of putting
all the beautiful and everything very well structured but then we need this feature for tomorrow
and this feature for the next day so sometimes we we prioritise some things we thought are smaller
usually metrics in the beginning of projects tend a bit to follow on those categories but
and that's why we try to create these consistent and strict dependency models so everything
it's easy to do but still enforce to have the right best practices obviously encourage also
observability mindset and probably we are also in the monitoring observability room we understand
the importance of monitoring but we need also educate our developers and other teams of the
importance of this might not work for you but in my experience I suggest something like this
library based approach every everything neatly stuck with some toys separate from the logic
to code and obviously continuous iteration and advances you need new features and try to see
what works best for you and for your team
thanks a lot for your talk are there any questions
thank you so much did you manage to fly I can deal closer is that good did you manage to find any
efficient ways I still I think it's starting to evolve maybe I'm like do you hear me now
by the way let me tie now it's okay did you manage to find any efficient ways of actually
measuring the cognitive load before and after your approach to understand and find the improvements
yeah you ask about ways to measure in the performance let me go back yeah it's easy sorry I
are really and I can't repeat the question maybe like
the velocity of code development the velocity of now
it's a question
but going to yeah so repeat the question yes so the the question was about if we have measured the
differences in cognitive load before and after the changes so like cognitive load is a bit of
an ideal right and for me it's about the field that I have before and after and after the
changes I was mentioning that before every time I had to change hypothesis to work and I don't
know metrics I felt that I needed to take like some time to just to all things were being
specifically implemented in that repository and afterwards now I go to any of those repositories
and I know exactly where things are being created or how metrics are being registered
and you mentioned about measuring I didn't bring any like data on the velocity but I know that
on the gd that we used to track our features we were able to do more story points
per sprint than we were previous capable of that because we are now able to develop the
features faster but also because the PR's are much quicker review than approved because we can
rely mostly on our team to do it so what do you think about group state metrics approach
compared to this one because you could totally separate metrics from your business logic at all
so we tried to take inspiration from both the approach that Kubernetes main code to
conduct and both cube state metrics I think cube state metrics you know like the best example
for this because the usual is just everything formation on the resources that are available on the
on the cluster for example when I show the the collector approach cube state metrics is a lot like
that right because you can just go to to the resources and fetch information there so it's a
more of a collector based approach but we also took a lot of inspiration from the Kubernetes code
Kubernetes works a bit different from cube it because they have more of a
structure the package we go back home and just take everything from there but I think we try to
to go at the way we would Kubernetes and many of the things that we did here they are also doing that
all right then thanks a lot thank you