Hello and good afternoon and welcome to the talk of the art of the twight given it is
of liberty three core strategies.
I am Prathikpanda, a site celebrity engineer at Red Hat.
Hi all, I am with Alibullah, site reliability engineer at Red Hat and this is our first
time at first time.
So, this talk is more based on our experiences of managing kind of large fleet of clusters
and that kind of covers the strategies that you can use to ensure that your fleet of
clusters is managed properly on the observability site.
So, moving to the agenda what we will be discussing today is what is fleet wide
observability at first then looking at the observability challenge at scale looking
at the three strategies that we will be discussing and implementing the fleet wide
observability with an example and then looking ahead to the future of observability
and then concluding with the Q&A.
So, starting with what is fleet wide observability it would mean the observability
at a scale where you have unified metrics, logs, alerts, traces anything and then you
are able to perform your specific requirements at a fleet.
So, looking at the key aspect that you would take for a fleet wide observability it
will be the multi cluster correlation, multiple clusters exist in the same symptoms
should have a correlation.
Then the centralized monitoring giving you a view for everything at once across your fleet
whether it is your Q&A cluster or maybe your application running out those or any
places that you might be running.
A centralized monitoring view unified plane of glass for them proactive insights that
would include the early detection as well as the remediation of issues that could affect
your fleet before they actually are kind of dangerous.
Standardization would mean that we have a specific format for everything that you are capturing
everything we are monitoring as and then we have scalability that means as your fleet grows,
your monitoring should be, your observability should be growing as well.
Moving to the next slide is more on the observability on this challenge at a scale.
So, the first challenge would be a high-metre volumes.
Let us say a single part that can expose at minimum 20, 20 to 30 metrics and then adding
a container moving those metrics to 25.
So, let us say a single node cluster could easily expose 5000 metrics running about
10 pods and that would mean that you have quite a lot of metrics to work around and what is useful
what is not can be a challenge and that would mean that you are stuck at a phase where
you do not know what to work on.
Then data silos that would mean that there could be multiple systems that are running,
multiple clusters running different kinds of workloads, different configurations,
both everything having different kind of logs, metrics.
In a fragmented way that would mean there are multiple challenges to work on at a unified manner.
Then there is the performance state of when you work at with metrics as well for any smaller
system and when you scale those up the challenges with the cardinality stuff.
What happens is when you scale up those metrics the cardinality would give you a lot amount of data
and that could lead to higher costs and then the specific thing that you have been looking
is not visible to you.
Then you have the operational overhead.
So, operational overhead would mean that if you manage different environments, different
configurations there are a lot of different stuff that you that could be going on.
And there needs to be specific teams that needs to monitor on those and that could kind of lead
to kind of a fatigue between those teams and they are kind of managing this at a scale.
So, looking at these strategies one that is metrics, then what that would mean identity
find what matters actually.
So, if we look at the Kubernetes metrics or metrics that are really useful, we have the infrastructure
metrics that would include the CPU, memory, disk usage, network usage coming from the core
infrared cells, any cloud provider or bare metal anything, then you have the application metrics
request latency is error rates through quotes coming on top.
We will be cumulative specific, we thought restart what is the node utilization that you could
be looking at or any resource specific utilization.
And at the top is something quite important is your specific SLO or SLI based metrics.
That is the custom metric that you are working on and that is where you provide your service
on.
So, if we look at a simple diagram that could help is how can you make that metric could be useful.
So, this is the flow that might not be relevant at all times, but can provide you with
series of questions that can help filter out if you need to include it in your
observability or not.
Let us say is it actionable?
metric should be providing you with action to work on, let us say a pod restart.
So, a pod restart which is occurring multiple times over a course of a few minutes.
That would mean you need to look and see what has happened and may be increase the resource
for it or see if the configuration is correct.
Does it provide context?
So, if a metric should be, should not be existing in isolation, it should have a context
rate.
Let us say there was a spike in CPUs.
It should correlate with a metric that should give you a part that this spike was coming
from something.
It is a higher workload demand or higher usage on your service itself.
Is it predictive?
So, prediction here would mean that let us say the metric would allow us to know what possibly
could happen if this metric is still continuing to interact.
Let us say there is high CPUs spike that you would see for a specific pod.
That could mean or high memory spike.
That could mean a future potential out of memory issue that could be going on into
the application that is running on that pod.
And then is it available in both real time and historic view?
Something that is important for a good metric could be is when you look at real time,
it should indicate that you have an action to perform on that you can work on this metric,
figure out something and then clear out the root cause.
And then historic view is something that would give you with an inside and analysis part
that if over a period of let us say few weeks we see that at this point this metric was
coming out we identify what was the cause and then we try to remove it.
So, this kind of covers what could a metric be useful.
And then of some things that we can discuss on or related to metrics is what could
mislead and how to avoid it.
The first would be over collection that we discussed like lot of metrics what is useful
needs to be defined on what are you providing the service at.
Let us say any metrics that have a high signal, high value are the prioritized metrics
that you should be collecting rather than all of the metrics that you are picking up.
Like of standardizations, multiple metrics and have the same there could be metrics
that kind of convey the same thing but in different ways but you need to figure out which
metrics are more better and then move them into a consistent monitoring framework.
Ignoring cardinality that has the risk associated of let us say the performance as well as
cost.
So, you need to be selective on the label usage itself you need to know which labels are the
ones that you would be using and if not look to avoid them and then reactive monitoring
if you try to alert on everything that has that is based on the reactive metrics.
Let us say CPU spikes that might not be the best way for your fleet to work on.
Instead you should be looking to prioritize on proactive and absolute event metrics.
Let us focus on the next strategy which is from noise to actionable signals.
Transforming from noise to actionable signals is very crucial for maintaining reliability,
reducing alert fatigue and for incident response.
It basically involves refining raw system data to draw insights that can help in decision making
and remediation actions.
Now the big question what actually makes an alert actionable in fleet wide observability
and to get an answer to this question there are a series of questions that you might need to go through.
Let us say you have an alert trigger.
The very first question should be is the alert relevant to the systems SLOs or business goals.
If not then it is best advised to exclude it from the observability or just put it to silence test.
The next question does the alert provide clear and sufficient context.
Let us say I have an alert trigger.
Is it just telling me something or it is actually telling me what component is expected?
What component is failing?
What are some outcomes that I might expect to see in some moment of time?
The next question is the alert actionable in real time?
Does an on-call engineer will be able to solve this upcoming issue or is it just a warning?
Or is it just a system in for that is being shown to me?
The next question is the alert prioritized appropriately?
Do I know what is the severity of this particular alert?
How is my customer affected at the moment?
If not then I better be reassigning and re adjusting its priority.
The next we can alert be automated.
I would say this is a bonus golden question.
If there are certain actions that can be automated and help me reduce the resolution time.
And of course then at the end we get an actionable alert notified.
Take this as an example.
Let us say we have QBAPI down alert.
Now we all know it triggers when the QBAPI servers are not reachable by my observability or by monitoring by within 15 minutes.
I know that it is affecting my SLOs of business goals.
I know it has certain clear context.
I know QBAPI is badly affected.
I know it is actionable in real time.
I know it has a very high severity.
My customers' workloads will be badly affected.
I know there are certain actions I can automate.
For example, I can automate the process of restarting the QBAPI reports.
So I think going through this will help you get an answer whether an alert is actionable or not in free flight.
Now we should also keep that in mind that all these questions might not be relevant at every stages,
but they will definitely help us optimize towards mature observability scenario.
Now let's journey to the path of effective alerting, which basically considers two key principles.
Solid alerting framework and insights and analysis.
In alerting framework, we consider a few things such as early detection, which basically covers proactively identifying potential issues before the escalate.
Intelligent alerting, we use some AIML based anomaly detection so that we focus on real issues.
Custom thresholds, we move towards dynamic thresholds instead of static thresholds so that false positives can be eliminated as much as possible.
Context rich alerts enriching my alerts within sites such as logs, remeditions, steps including SOPs that will help engineers to reduce the MTTR.
Self-feeling mechanisms, setting of auto recovery actions such as restarting of pods and essential services.
Next we move towards insights and analysis.
These cover are few methods, first being trend analysis, where we analyze the performance degradation over a period of time.
Capacity planning, using historical data to foresee resource needs.
User behavior analytics, it is a mechanism where I try to detect anomalies at an early stage as much as possible and then try to forecast any upcoming potential issues.
MTTR reduction, providing insights, logs and correlation of logs and alerts helps to reduce the MTTR.
And of course, incident post-mortems, learning from incidents so that we do not see the repetition of similar issues in upcoming times as well.
Yeah, so that covers the second strategy that we had and moving to the third is the correlation that is connecting the dots.
So, the connection on this is through the context and the journey that is the logs and the traces.
So, logs as we know it is the detailed insights to any specific events more like the why it happened and traces is the visualization of the request.
The traces might be or might not be relevant when you kind of work on a fleet level.
It depends on what specific goal you have, what specific service you are working on and combining or maybe using these is what we get on a correlated insights.
So, when we did this, this kind of gave you the idea on the three strategies and how do you the complete fleet wide observability picture look like.
It is a kind of the common area between metrics, alerts, the logs, metrics being the what is happening alerts when does it need attention and logs or traces why is it happening.
And when you kind of focus on the correlation part, it will help you combine signals to find the root cause to find the solutions that you are looking for much faster and kind of help us resolve the issues much faster.
Now, let us see how we implement fleet wide observability and this will primarily involve leveraging the concepts of SLOs SLAs and SLIs.
Now, how we decide SLOs in fleet wide of less than 95% of our clusters are unable to meet the requirement of getting upgraded within two hours.
Now, we check if an SLO is being breached, we trigger an alert for that and then we investigate and resolve accordingly.
Once the incident is resolved, the issues are all in place, we do some post incident actions such as revisiting our budget burn and then reviewing and adjusting our SLOs and then the cycle goes and goes on and on.
Some benefits of SLO driven approach, scalable reliability management. Now, this because this provides a uniform environment for trouble shooting and monitoring, decreasing cross service dependencies and providing better user experience.
Proactive issue management as we just discussed with error budget framework and alerting and insights we are proactively working on issue management.
And of course enhanced user experience wherein we are focusing on customer impact and we are trying to reduce the downtime as much as possible.
A line goals across teams as uniformity comes in, we have unified metrics, we have prioritized efforts, everyone knows what they have to do and of course this means that goals are aligned.
And of course this overall indicates that our environment is continuously improvement, our observability, the fleet wide observability as such is evolving in an ineffective manner.
And of course business driven observability, we are able to align our technical and business needs and we are providing end to end visibility which is further increasing our customer impact because at the end of the day if customer is happy we are happy.
Looking ahead, the future of observability, AI driven insights, AI is going to play a huge role in observability by enabling to provide resolution to problem detecting anomalies and of course identifying issues.
Cloud native and cross cloud provider observability is going to evolve to support and enhance more of the cross cloud workloads providing more insights at the time do not increasing the performance overhead.
Automated remediation observability is going to move towards proactive monitoring rather than passive monitoring wherein identifying issues recovery actions are going to be all automated requiring less and less human intervention.
Open standards and interoperability open source is going to be the backbone of observability supporting vendor neutral and flexible integrations.
This is all we had today in a kitty folks we would like to have any questions or any discussions you might want to take.
Any questions and please we do five minutes Q and A if you have the time stay seated because this half the room leaves now we will not be able to understand the questions and answers.
Does anyone have a question?
Hi. So you said that we should trigger the alert when the SLA is bridged shouldn't we trigger when we measure that it will be bridged sometimes so that we are proactive working on the incident before the SLA itself is bridged.
I am going to answer that with a question how do you know that you are close to an SLA being bridged.
That's a good thing. So what I would recommend is the approach I have seen being implemented is let's say I talked about a two-hour window right.
At my internal level I realize that up to let's say one hour it is okay if the alert is not progressing but if the one hour beyond that is something that might become troublesome.
So I set up my alerting at that level only let's say control upgrade is delayed is what I get triggered at.
So I am already getting alerted per cluster level when the upgrade is in between failing.
So the thing that I talked about the SLA being bridged it is at level let's say when you are getting continuous alerts of different different clusters that the upgrade is between halted.
So that is already set up in a proactive.
Do we have more questions?
So you mentioned that we should enhance our alerts with logs right alert are based on metrics.
How can we identify which logs to add to those specific alerts and if you can give an example as well?
I think I can take that. So that depends on the type of alerting that you will be focusing on and also on the of the ability tool that might be using.
There are certain observability tools that kind of provide the scenario the view at a unified level let's say let's say that has logs metrics everything at once.
And that has the capability and I think that has the capability as well to kind of do something like a log based alerting as well when you do the metric based.
So both metrics and logs kind of combine and do the alerting as also if you work on something let's say at a very cluster level and then you are integrating something like
pzduk it it would mean that you reference something on an SOB based level that the engineer kind of works on to kind of figure out from that alert what the logs means.
What the logs are at that point of time. So it's kind of an enrichment of alerts can be depending on what this scenario is and what the actual work load being affected.
If you are standing all of the metric logs at once that could mean you can enrich it if not if it's on a more more let's say a wider level that would mean the engineer needs to do a bit of research before you get the exact cost for that alert.
More questions or there in the back.
Hi, a very basic question. Do you have any publication of like what are the basic SLIs indicators or what are the basic SLOs for different parts of your Kubernetes cluster as well as your whole fleet.
Some kind of examples that we could use in our deployments and how do you define those SLOs.
Now, please take a mic because there's people watching the videos and streaming.
Okay, so to answer that that will depend upon service to service.
For example, if you talk about just Kubernetes let's say we talk about upstream Kubernetes.
So you can refer the Google's guidelines on that. Google has already published on what is being happened and where.
And since we work on managed open shift that is our service. I don't think that is publicly open as of now.
But for definitely you can refer to the ones that are available by Google and Google SRE book is the Bible for managed services when it comes to Kubernetes.
So you'll find everything there and a bonus atlashing also has some of that available. So it's it's just all available openly.
Yeah, and something to add is like it's a continuous evolving process as well like we mentioned.
So something that you might have let's when you take it at a very basic SLO level you try to improve it based on what your team is requiring or what you are actually providing out.
So it kinds of goes on evolving as well.
All right, one more question.
No, then thanks again.