I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to
anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't
want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody
I don't want you, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to anybody, I don't want to talk to somebody, I don't want to talk to somebody, I don't want to talk to people, I don't want a thing, I don't want people, I don't want駄 resources, I don't want people, I don't want to talk with anybody, I don't want anybody, I don't want to talk to anyone,
what we can learn from formula 1, incident management.
Very excited about this topic.
Can I get started in a minute or so?
OK, let's wait a minute or so.
So, first of all, let's start by.
Thank you, everyone, for being here.
Seeing so many people.
In the talk, it's about incident management and formula 1.
So, I'm assuming you guys are very interested in incidents, not formula 1, right?
That's my assumption.
Okay.
Let's give it a minute.
And other colors like the N1.
Like this?
It's better?
Yeah.
I can speak louder.
So, let's give it a go.
So, thank you, everyone.
My name is Ricardo.
And this talk is in the monitoring and observing room,
but it is slightly different from what you've been having up until now.
So, I work for a product company.
So, incidents are the thing that we focus a lot,
and it's our very important for us.
So, as most things, in software engineering,
which is a very recent engineering practice,
when we compare to other types of engineering,
there's a lot that we can learn from other areas, right?
So, for example, if we think about patterns like both heads,
it comes from the shipping industry.
When we talk about canary deployments,
it comes from the mining industry.
So, if we start looking at other industries,
what can we learn from that?
So, today we're going to start analyzing
and a specific incident during an F1 race,
and we're going to see the things that they put in practice,
that they do, and what we can learn from that.
So, either we like it or not.
F1 is one of the most demanding engineering practices in the world, right?
So, they can win or lose a championship by 1,000 of a second.
So, that's how critical that is.
So, 1,000 of a second is the difference between
becoming first or second.
So, they're really precise on things.
And they're basically striping the human being to an engine, right?
That's all they do.
And all the aerodynamics is to make it go fast.
That's basically it.
So, when they have an incident, what do they do?
So, let's pick a specific incident that happened in the 2020,
and let's see the bunch of things that they did to actually
make it successful.
And at the end, I'll promise, I'll show you what this success was.
So, I'm going to show you like 15 to 20 second videos.
I don't know if we have how you or not,
but all those videos have subtitles so it should be fine.
So, let's start by setting a stage and understanding
what the incident was.
So, we don't have sound, but I'll narrow it.
So, basically, this is a race in Hungary in 2020.
This is Max for stopping.
For those who don't know who Max for stopping is.
He's currently a four-time old champion.
At this point in his career, he didn't have any world champions yet.
But it was probably the first year where Red Bull was actually
a challenger to Mercedes, and there was some kind of pressure
to actually perform.
This is a third race in the championship.
So, basically, what happened?
So, Max is challenging Mercedes, and Louis Hamilton,
who probably most of you know, for the world title.
He didn't have his best qualifying, he qualified seventh.
So, if he wants to continue to challenge,
he has to do a very good race.
He needs to get him somewhere in the podium.
So, he has to at least get four places during the race.
On Sunday, during the formation lap, which is the lap
that the drivers used to get their tires warm.
So, tire temperature is a very critical thing in Formula 1.
So, if it's too cold, they can try fast.
If it's too hot, they're burning rubber too fast,
which means they need to go a lot more times to change the size.
So, what happens?
During the formation lap, cold weather.
So, Max is trying to put his tires as hard as possible.
But there's also a lot of rain in the track.
Max goes too deep into a corner, and what happens?
Brakes lock, and he goes into a barrier.
So, if you notice, and I'll just put it again, just for context.
If you notice, the moment he breaks, or the moment he crashes,
there's a timer counting on top.
And I'll explain what that timer is in a second.
So, he's breaking tires lock.
He goes into a barrier, 22 minutes.
I'll explain in a second what that means, right?
So, first thing that we need to have on incidents.
We need to make fast decisions with limited information, right?
So, in a middle of a set one, there's an outage.
You don't have time to get all the data, right?
You'll have partial information, and you need to make fast decisions.
Let's see what they did.
Let's see what was one of their problems.
So, they're trying to make a decision here.
I'll let the video play.
It's like 15 seconds.
So, setting the stage again.
They have 22 minutes to get the car ready to race, right?
And they have to make the decision.
They do a pit lane start, which means that they put the car on the pit lane.
They have all the tools there.
They have all the mechanics.
Everything is there, but they will start last.
So, all the cars will go out, and max will start last.
So, it will means that he will probably have to overtake over 20 cars,
just make it on a podium.
Or, they put it on the track, but they have to fix it there.
So, they have to take all their tooling there.
Everything there, mechanics, tools, front wing.
But they lose four minutes.
Why is that?
Because four minutes before the race, all the four tires need to be on the ground.
So, it means that they just lose four minutes,
and they have to take the car off the grid.
Right?
Here's a problem.
And they need to do a decision.
They only have to let me treat data.
So, a formula one car, quality has thousands of sensors.
They have a lot of telemetry there, but no one looked at the car.
At this point, no one.
No one, no one.
They assume there is a suspension break, front wing to change,
and they need to make a decision.
And they make a decision.
So, we're going to the grid, and we're going to do everything there.
So, when we're talking about fast decisions,
we'll leave it till information.
So, we need to make decisions that are decisively,
even if we don't have the full picture.
So, what we want to do?
So, we want to minimize the impact.
So, we need to act fast to stop small problems from getting too big.
So, in this example, it's probably not the best examples,
but if you have a set one, most of the times you need to make a fast decision
before things get too big, and you're over your head.
You also need to protect your customers, right?
So, maybe you're approaching in a finance industry,
and if you don't do something, your customers are losing money, right?
So, that's something that you need to do quickly.
Maybe you need to hold your whole system down so that you can lose money.
So, you have to preserve revenue.
So, at the end of the day, most companies live on revenue.
So, you'll have to preserve the revenue company is making.
And, of course, you want to maintain customers' stress.
You want to show your customers that you're in control.
You know that you have an incident and you're doing everything as possible
to get to resume service.
So, as engineers, we need to become comfortable with this discomfort, right?
Incidents will happen.
It's not a matter of if they will happen.
It's when.
They will always happen.
So, we need to become comfortable with the fact that we will need to make decisions fast
and we don't have the full pictures.
And we need to keep in mind that many of those decisions will be sub-optimal
because we don't have the full picture, and sometimes we'll be completely wrong.
So, we miss the mark.
And that's okay.
It happens.
Another critical point in an incident is clear communication.
So, let's see an example.
Yeah, the sound makes it better, but I'll explain it.
So, I'll explain it a little bit.
So, basically, they're taking the car to the grid.
They don't have anything there, so they need to get everything in there.
So, parts, mechanics, tools, everything in there.
So, with the sound, it's more clear, but basically, they are communicating with each other.
What they need, where they need to be, where the car needs to be, who's engineers need to stay in the garage
and need to be there.
And again, they have less than 22 minutes to make this happen.
So, it's super critical that everyone communicates super clear.
What they need, what they need to do, what tools do they need, what are the spare parts.
And one needs to look at the car and understand what broke, what they need to change and whatnot.
So, clear communication.
It's all about sharing the right information with the right people at the right time, right?
So, if we have clear communication, we are ensuring faster resolution, right?
We need to communicate clearly so that we are all on the same page.
Who's here has been in a situation where you're trying to fix an incident,
and there's two people looking at the same thing at the same time,
all messing with the same services.
I'm changing this, and then suddenly someone changes it as well.
Yeah, that happens, right?
If we do this in under 32 minutes, we're trying to change the suspension, it's going to be a mess, right?
So, we've all been there.
So, this is super important.
Also, we want to reduce the errors, right?
So, I'm sending the wrong message to my colleagues, so he's doing something that he's not supposed to do.
We make things even worse.
We want to improve coordination, right?
So, I'm doing engineers.
We are working, I need you to do something, you need me to do something.
We need to be super clear on what's needed, what needs to be addressed.
And of course, increase transparency.
If communicate in this clear, everyone understands what's happening, right?
So, we're working on having good communication, help everyone, what's going on,
and how can we make this resolution faster?
And of course, we want to keep everyone in the loop and maintain trust during incidents.
I'm asking you to do something, I trust you.
This is the clear direction.
And of course, for non-technical stakeholders, it's super important that they understand
what the hell's going on, what we're doing, and that we are working to make things faster.
Next up, clear processes.
Let's see it again.
So, at this point, they need spare parts, they need tools.
They don't have it there, right?
It's not their garage, so they need to bring a lot of stuff on track.
So, they are basically communicating what they need, right?
And, at the end, there's the director of the garage saying, guys, we have clear processes
for this, like, please follow them.
It's super critical, right?
They're probably going to have dozens of engineers trying to get a suspension off and suspension
on.
If they don't follow those procedures, they have no chance of getting this work.
So, clear processes are the fine steps that will guide us through incidents.
So, they will allow us to have rapid response.
If you have a clear process on how you manage incidents, it will be a lot easier.
So, if you have people that have a fine test during an incident, it will be a lot easier.
It allows you to reduce chaos, right?
Again, the same example as before, if we have two people working on the same thing,
looking at the same dashboard, messing with the same services, it will be a mess, right?
So, you want to reduce chaos as much as possible.
You want consistent execution, right?
So, you don't want to go from one incident to the other,
and then everything is completely different, right?
You want some sort of resemblance of consistency from one incident to another.
And of course, you will improve collaboration, right?
Everyone understands exactly what they need to do to address this incident.
So, you have to plan before incident happen.
So, how do you do this?
You actually have a plan in place, and you train, right?
So, if you, I don't know who here is an F1 fan, but if you see before the race,
there are people just testing out how to change a tire, right?
But just there, taking tires off, putting tires on.
Taking tires off, putting tires on, because it's super critical,
like milliseconds are important here.
And of course, clear steps make for a faster response.
So, if we all understand during an incident, what our role is,
what we need to do, it will make the incident resolution a lot faster.
Right? This should be an obvious one, right?
But during an incident, teamwork is critical.
So, this is here just an example of a small snippet of all of the engineers
just working together, right?
They're basically taking suspension off, putting on,
the max is just looking at the car.
They're bringing a front nose from actually front nose is fairly straight forward to change.
During a race, it's tricky because it takes a few seconds,
and every second counts.
But for here, it's basically just a 50-game mistake, right?
So, it's not that bad.
A suspension is a whole different thing.
So, teamwork.
It's all about collaboration and share responsibility, right?
It's not a matter if I fix it,
or you fix it, the matter of getting things over the line.
So, you need never skills for faster fixes.
You need people to, who will be like,
who will be the engineers fixing thing.
You need someone who will handle comms.
So, you need a diverse skill set to actually get this over the line.
Like teamwork makes the dream work, right?
So, this collaboration will actually be what,
I would allow you to have a smoother transition from,
I have a problem, it is fixed.
Everyone plays a part, like I said before.
It's not all just about fixing this thing.
It's all about communication, making stakeholders accountable,
making customers understand what's going on.
And of course, it has the side effect of,
we all learn together, right?
So, if we're working together, we understand what's happening.
So, who here has read the Phoenix Project?
Right?
So, you know, Brent?
So, Brent is the antithesis of teamwork, right?
Brent is the guy who fixes everything.
It's the guy who knows the whole system,
who does everything.
There's no teamwork there, right?
So, and it's this idea.
So, this idea is that we don't want Brents.
We want people like Brent who has some knowledge,
but it's about a whole team fixing a problem, right?
So, teams that work together,
solve problems faster, like even the most experience
and most knowledge engineers in one organization.
When the system gets big enough,
one person will take longer to fix it,
and if we don't do it together.
Of course, teamwork builds a more resilient and effective response
to incidents.
Brent will have to go on holiday sometime, right?
Brent gets sick.
So, if we don't have all of these things in practice,
we will have serious problems.
Something that is easier said and done,
incidents, we need to keep calm, right?
So, this is probably the video where the audio
would make the most sense,
because you will listen to how engineers communicate with each other,
and they will be as much as possible calm, right?
So, at this point, they have five minutes to put the car on the ground, right?
So, they're finishing up the suspension,
they're putting the nose on, they have to put the tire on,
they have to bolt it, it's not only about having the suspension on.
Someone will be driving this car for 60, 70 laps,
over 300 kilometers per hour.
So, if they mess up, the guy will crash at 300 kilometers an hour
against the wall.
So, they not only need to make it happen,
they need to make it happen with extreme precision,
so that they don't kill anyone, right?
And you will see that, in some of these communications,
there's someone who's communicating with the FIA,
which is a regulation, but regulatory body,
and he's constantly asking for feedback.
And it's super calm, guys.
We have five minutes.
How are we on track?
Are we not on track?
Super calm, not like pressure,
shouting, and all this bugging.
So, during incidents, we need to make the best
on thing-compose and performing under pressure, right?
If we're calm, we can think better, right?
I know it's easier said than done,
but it will allow us to make it better, better decisions
than just being stressed, right?
Calm teams will better, like, a calm atmosphere
if I'm not pressuring my colleagues, I'm asking, of course,
we're up late, I need to understand what's going on,
but I'm not shouting, I'm not rushing, I'm not bugging him,
every five seconds they'll, is it fixed now?
Is it fixed now?
That's what we want.
Right?
We have to stay tough, air-cooled, right?
So, we need to build this mental strength to allow us to,
as much as possible, keep some cool, right?
And although we need to make quick decisions,
we also need to, we need to strike a balance between
quick and too quick, like, try to not rush a lot, right?
So, a calm mind allows us to make better decisions during incidents,
and we have to train to handle this stress incidents.
So, the best way to stay stuff, to stay tough, to stay tough,
of course, in incidents, but you train them before, right?
Stuff like, chaos engineering, like,
for a game day is where you actually break stuff on purpose
and try to test your processes or, like,
building that muscle where, when you have an incident,
I've seen this before, I know what to do, right?
Technical proficiency is also super important for resolving an incident.
So, here's an example of some, so this is at the start,
so they still have 20 minutes, and they're basically just discussing.
Can we even do this, right?
Can we do, can we change this suspension in under 22 minutes?
And they basically say, we have to do it faster that we ever did before.
We never did it this fast, right?
And they, again, they train those things, like,
how fast can I change this suspension, right?
And they've never done it before.
So, it's all about technical proficiency.
Do the people that go into an incident, actually,
know the frameworks that they're using.
Know the service, know the whole context.
They know what the trade-offs are, right?
So, this is super important, right?
So, strong skills allow us for efficient diagnosis
and resolution of a problem, right?
So, skills actually matter.
So, people need strong skills, not only about the programming language,
the frameworks that we're using, but the whole context of the company, right?
Why is this service configured this way, right?
So, there is some context.
So, over time, we need to get that context.
Knowledge is power, right?
So, the more the team knows, the more the team will be capable of understanding what the problem is and how to fix it.
And of course, we need to stay sharp, right?
So, we need to keep up to date with both the industry,
but also the how things are in our company.
So, stronger skills mean that we can be stronger at an incident response.
And of course, the right tools will empower our teams to resolve incidents effectively.
And last but not least, post-modern.
So, this is the only video that is slightly different from the others,
because they don't show the post-incident review,
post-modern, whatever you call it in your organization.
This is just an example of Mercedes actually going through
how they make some decisions after a race.
So, I'll leave it a link at the end.
So, Mercedes has some nice videos where they analyze the race
and they share with the whole public saying,
we did this decision, this didn't pan out.
I won't step back too much time,
because we don't have audio here.
But a post-modern, more than anything,
it's all about learning from incidents to prevent future problems.
And that's the critical part, right?
Post-mortems are about learning.
Nothing else, right?
So, you need to use them for learning.
So, how do you learn?
You try to find the root cause, right?
So, I have an incident, I mitigated or fixed it.
What causes this?
What led to was having this incident?
The idea is to learn from a sex.
Something didn't go well.
Let's go through the problem.
Let's see what happened.
And what did it go well, right?
So, in the case of a car like this max,
it was this driver error.
It did it break too late.
There was some problem with a car.
So, trying to understand so that it doesn't happen again.
Super, super critical.
No blame.
This is critical.
If you assign blame during a post-mortem,
people will shut up the next time.
They won't speak, right?
It's super important that you're learning.
This is a learning process.
It's not about the race driver that broke too late or
there was a problem because someone didn't fit it.
And it's okay.
Yes, maybe it doesn't matter, right?
Post-mortems, post-incident reviews.
Whatever you call it in your organization,
will have teams learn from incidents and get better every time.
So, the next time that it happens,
we know, so we will not make the same mistake again.
And of course, we need to fix on fixing problems,
not finding fault.
I don't stress this enough, right?
Please don't point finger points.
So, why is this all of this important?
Right?
I think that today, what do you want?
We want to prevent incidents.
So, we need to learn from incidents,
but if possible, what we don't want are incidents, right?
So, we don't want them at all.
When we do have them, we want to make them mitigate them
and fix them as fast as possible, as soon as possible, right?
And ideally, what we want to do is to avoid them to happen again.
So, having the incident over and over again,
it's costly has a lot of problems.
Unfortunately, many organizations don't have many of these practices in mind.
So, they don't practice these things.
And the idea for this talk is to show like,
one of the top engineering practices in the world,
they do it, and why do they do it?
Because it happens, because it helps, right?
It actually works, right?
So, and then you might ask, okay, where should I start?
And my answer is always post-mortems, right?
It's one of those things where, if done correctly,
done with the learning mindset,
it will feed on all the others.
Because during a post-mortem, you will understand, okay,
the incident took longer, because we didn't have clear communication.
The incident took longer because we had engineers on call
that actually were knowledgeable about that service.
Maybe we need some kind of process, right?
Because there was a lot of confusion.
Some people didn't know what they needed to do.
We had two engineers working on the same thing at the same time.
So, post-mortems will actually inform all the others.
So, if you have to start somewhere,
you probably should start with post-mortems, right?
And probably the one that it's easier to adopt
that will get you the biggest man for a buck is clear processes.
There are a lot of ETL frameworks out there.
You don't need to reinvent your own.
Just from board one, try it.
Make the adjustment that you need, right?
So, there's a lot of industry literature,
where you can just read and say, okay, here is a framework
that I can just follow that will help us a lot.
So, you might be asking, all of this for what?
So, what did they achieve in 22 minutes?
Again, the video has a little bit of a dramatic audio,
but...
So, you've done it.
So, they went in under 22 minutes from a broken car
that wasn't able to race to actually fighting for podium
and finishing P2, right?
So, not only they managed to get the car on track,
they managed to give him a car to overtake five cars
at this point max rose to third in the championship
and redwood rose to second in the championship.
So, if they didn't do any of this,
if all of this wasn't successful,
we probably have dropped in the driver's championship
and redwood should have dropped in the...
Constructs as championship.
So, in any two minutes, what people...
Miracle, yeah, maybe an exaggeration,
but again, they had never done it before,
and this allowed to redwood to continue in the race
and actually fight for the championship.
They didn't win it, right?
Max finished the third actually this championship,
but he won next year.
But this was the role that I needed to go to actually...
to actually get there.
So, just a few links.
So, the first one is for this actual incident.
It's more, it's roughly eight minutes, so it's a fast way.
So, if you guys want to follow the whole process
and what was happening.
So, if you want more information on this...
Grand Prix itself, it's the second link.
And the last one is the link for the Mercedes AMG.
YouTube channel.
They have a lot of post-incident reviews
where they go through what they like.
We made this decision, but we messed up.
And if you search the internet a little bit, you will see some...
So, they don't show the incident review from each team,
because they will be discussing confidential things,
but there are some videos that are zoomed out,
and you can only see, and it's intense.
Like, they go to huge scrutiny
to actually understand why did we mess up on this race.
Was it a problem with the drivers?
Did we mess up the strategy?
Was that a problem with some part of the car?
They go into an extreme detail.
Mercedes has this habit of actually explaining some of those decisions.
Like, we messed up on the strategy, right?
We made a pit stop at or like, and they go through, okay?
This is why we did that, and that actually didn't work.
So, it's actually a good learning experience
where not only they do it internally, but they actually do it
with the whole public, as I was saying.
Guys, we messed up.
This was our reasoning.
This isn't work.
This is how things work.
And this is all from my part.
Thank you very much for being here.
And if you ask any questions,
hello, hello.
Is it open?
Yeah.
Can everyone hear me in the back?
It's not all over the country.
Yeah, yeah, yeah.
It's not all over the country.
So, in terms of communication, how do you, like,
when that happens?
Yeah.
Can everyone hear me in the back?
It's not all over the country.
Yeah, yeah.
It's not all over the country.
So, in terms of communication,
how do you, like, when a separate one is a range of zero
incident happens?
How do you, you know, notify people?
Like, is it an alert?
Is it you?
Notifying people in the channel?
Yeah.
Oh, my one.
So, the question is, if you have a set one,
incident, how do you alert people?
Yeah.
So, you want it as much automated as possible, right?
But it won't happen every time.
So, what you want to do,
it's over time, start with a problem.
Maybe you have an incident, and you didn't have any alert, right?
Maybe a customer told you, maybe you,
some ZP, he was actually browsing your side,
found it out, right?
And that's the worst, right?
So, you want to do it over time, you wanted to make that,
please leave quietly.
What you want to do is to make that automated as possible.
Start from corporate, right?
So, maybe you found out that a certain cohort of customers
that do some predefined action in some time,
that fails, okay?
Can I actually replicate it?
So, you have a learning on top of the things that you already have,
and somebody that to me and my team
has been working a lot is on synthetic monitoring, right?
So, trying to simulate users all the time, right?
Making that explicitly clear and try to pick up things as soon as possible.
All right.
Thank you.
Any more questions?
What?
Please, again, leave quietly,
if you have to leave in the library.
I don't know.
I'll let it out, okay?
All right.
So, in the stakeholder phase,
you said that there should be always procedures
for, would say, most of the emergency,
but for example, in our environments,
it could be that there can be different things
that we haven't even thought about.
How should we be able to stay calm,
if we don't have procedures on that?
Yeah.
And how to prepare for things like that?
Yeah.
So, the question was,
if you have to leave, please do so quietly.
So, the question was about processes and having processes
and on the stay calm phase,
what if something happens that I've never seen before?
Like, I've never prepared before, right?
So, how do you deal with that, right?
Again, what our teams usually do,
they do a lot of game days and chaos testing, right?
So, we're basically putting our teams
not on the stress, but regularly saying,
here's a new thing, right?
So, here's something that they're used to handling.
They're not used to handling on a Tuesday at 5 PM, right?
So, we're basically keeping that,
so it's all about one,
introducing things that will likely happen,
so they are getting accustomed,
so it's about just introducing things,
just introducing stuff, right?
So, when something new happens,
that they never seen before, and I was like,
okay, it's just another day at the office, right?
So, then we'll see.
And only it's like,
if then using the post-mortem is okay,
we've never seen this before,
what can we do to not have it,
or to make it easier?
Do we need better observability?
Do we actually need, maybe it's the process,
we need to refine the process,
maybe it's communication,
maybe we need to train for this,
but even if we have all the automation,
all the scripts, all the buttons,
whatever, it's still hard,
but if we train this,
like, let's on a Tuesday,
let's break the system and try to fix this
on a flight one purpose, right?
So, it's all about that thing where it will happen,
right?
It will happen, something that you never prepare for,
it's the nature of business,
it's keeping that muscle alive of continuously
breaking stuff and fixing it,
and making it like,
it's just another day at the office.
Any more questions?
Yep.
Have you had challenges at your current employer
of convincing management to,
carve out specific times of duties,
practices these chaos engineering type things?
Absolutely.
How did you navigate them?
Yeah.
So, one of the challenges that you had specifically
when you're talking about product companies
that work with end user, right?
So, the first time that you talked to someone
higher up saying,
I'm going to break my system on purpose
and they're like,
no, you're not.
So, you are not doing this.
So, the, again,
being completely transparent,
the best strategy that we found
is to actually somehow correlate this
to a number.
It could be a metric,
it could be a dollar value, right?
So, one of the practice that we use
or to make it crystal clear that chaos engineering is important,
is saying, okay,
let's average out all the set ones that we have
and let's make a wild gas estimation
how much did this cost, right?
And it cost us,
again, in our context,
a minute of downtime depending on the incident
and on the time of day,
could be costing the company millions, right?
So, millions of dollars.
So, if we somehow say, okay,
we can train the team better
or we can invest in this tool, right?
And instead of taking us to our success,
it will take us 10 minutes.
It makes the conversation more easier.
There's a lot of upfront work
that we'll have to do to actually
analyze data,
giving those higher number values.
But it's making that understand
especially for product and customer facing the value.
And the best thing that we've used
that with success with executives
is putting a dollar amount.
It's money.
At the end of the day, for a product company,
it will be money.
And if you can do that correlation,
people say, okay, I got this, right?
I have no idea what we're talking about,
open telemetry,
but if you're telling me that instead of taking two hours,
you can do six to ten minutes,
go for it.
And it will cost you somehow,
I don't know, two million dollars to do.
Yes.
If I have a 10 minute outage,
it's costing me 20 million,
no brainer, right?
So, it's somehow getting to the point
where you're talking a language
that's an executive,
understand and it's usually about money.
At the end of the day.
Anyways.
I think we have one question.
No, no, no, no, no, no, no, no, no, no, no.