Hi, I'm going to talk about the Essinox problem that casts Manslope Shadow, which is a story
about Essinox and TechDepth.
Before we start, do we have any real customers, real users, past or present?
OK.
As the start of this disclaimer, that these clients are my own reflection of the events
and nothing contain them is official or the communication.
If you are interested, please talk to our support if you need information or need help with your systems.
So, where am I? I'm a software engineer and a product owner for content management and data collection in the client tool steam.
That's where real, run it inside or satellite or meet.
And directly we support much more use cases like OpenShift or Podran desktop.
And our team is directly for real.
We don't have any true upstream, we target the real while supporting Federa and Centos indirectly.
So, let's start with keywords about what to talk about.
Well, first is sort of the insights that the software is associated with hosted by Red Hat,
a figure of charge for all related customers.
Essinox, which is actually a architecture for Linux and it's part of RL.
And insights client tool that collects data over a host and uploads it to Red Hat insights.
This is how Red Hat insights work.
As you can see, there are some systems in the inventory.
You can see, we detect this, some CVEs, we know that we can remade it.
Some of them, and you can see, you can do wild stuff with that.
Remootly update packages, change configurations to ensure your host is compliant and secure.
And Essinox is a kind of module that allows you to prevent programs from randomly changing content for other packages or programs.
Every file, every process, every socket is labeled and we ensure that nothing can go wrong there.
When it's set up correctly.
So, let's start with the intro.
Our team had a history with Essinox.
In 2022, issues filed by our support engineer that insights client is producing a lot of AVEs.
A lot of the Niles.
So, a few months later, the Essinox shipped an updated policy in RL 86 and 9.
And a few months later, the process was updated again because of complexity of behavior of insights client.
When you bring that to the specific topic of this talk, slash rules, slash new PG.
Also, in 2022, one of our engineers filed a baczilla saying that insights client tries to create rules GPG folder and put stuff in there.
So, around a year later, a year and a half, the cart was picked up by the developer.
Then, after a few days or weeks, the code review started.
One of the focus I asked was GPG, actually.
We knew that it's the one to make us with the race conditions.
When it comes up after itself, our colleagues from Convert to RL and RL upgrades let us know.
It took some months, but in February, 2024, our Q-engineer performed the maneuverability verification and the PR got merged.
Now, we have to start mentioning dates, specific days.
So, on Monday, 12, the big insights client outage started and engineering wouldn't know for another week that there are some problems.
Actually, how does that happen?
We are talking about RL, that's enterprise distribution.
Doesn't it really cycle, segating, or component tests?
So, here, this where we have to make a small B-tour.
The client in core, while insights client repository itself is RPMShell around the insights core.
The client is a minimal program that ships the configuration, file, system D, services, and ensures the core is up to date.
The core is GPG site archive that contains collectors and powersers that later are the insights used to display recommendations or ensure compliance.
So, what was the problem there?
The patch was that we shouldn't touch the directories in the road.
It's ruled, it's weird, when this admin's look at the content, they can see some keys they didn't put in there.
It's just, it's not how you should treat the system.
So, the task was to upstart GPG away from the rest of the business logic.
And instead of substituting directly with ono additional setup, we create a temporary directory, use it as the home for GPG to perform all actions.
And so, when we have verified the payloads, we finished and cleaned up the temporary directory.
This is the protocol for it.
It was like 200 lines of Python plus unit tests.
As I said, first we create the home, then we import the keys that are known to be signing the payloads,
we verify the file, and we delete the home used by GPG.
That's it.
Now comes the sling of spart.
By default, the GPGT, the label for GPG process, writes to GPG directory, which has label GPG secret team.
Python stem file, use a team p by default, and because it's created run by inside client, the Python code,
it's the directory that's created, it's labeled as its size client TMPT, temporary directory of inside client.
Well, and GPGT is an allowed to write to inside client TMPT, which is what this then now looks shows.
The source content context is GPG, target is inside client TMPT.
You can see we are trying to do something with directory and we get denied.
So as a result, what happens?
The denial results in OS error being carried out.
We just are not allowed to touch that directory or files.
The 100 error is propagated from the core into the client wrapper.
The wrapper exists with nonzero status code as well.
Again, it's nonstarter behavior.
And the system disservice that runs these transitions to failed state.
Well, was it bad? For story reasons, this service configured as restart equals no.
And it's been there since 2018 or seven.
And it's because was seven was the first error version that shipped with system D.
Before it was six views grown, which runs every time no matter the previous executions.
And this behavior was just replicated.
It's been known that system on network glitches may cause the service to fail.
And it was never addressed because other issues that were more visible, very paradise.
So we have some systems that were used, that were running system D service.
And now they are not, so that means the systems are not collecting the data archives.
And they are not uploading them and they are not updating the core.
So we know we have a problem.
As I said, on Monday, the core was published a production.
On Tuesday, the server-site team discovered that the component is broken due to the dependency on the inside client.
On Wednesday, we released the fix into a production.
We just reverted the commits.
On Monday, the next week, we discovered that the bug is affecting more engineering groups.
So the next day on Tuesday, 12, 12, 12, 12, the first emergency meeting was held.
Where we have reproduced the issue.
We published a support article.
So both our customers and support knew what to do there.
And we had more of them.
So who is affected?
To run out around eight, six plus and nine oh plus with a ceiling of an e-bolt.
How will inside services react when the host is supporting?
Well, you can assign a metadata to the host in inventory or in other services.
And if the host gets deleted because they are not reporting anymore,
these configurations would get deleted as well.
So we said, let's pause the duration and worry about that later.
And how many customers and system are affected?
We actually didn't know.
Because, or it is a big organization,
the end of client tooling doesn't handle the services.
So they don't really know how many systems there are.
We don't need to.
So how to get the numbers and how to even figure out who the relevant teams are.
And there was no percent on informing customers about these problems.
Would it be best way to send emails?
How many to stack?
Scared the customer?
Some are really scared about their infrastructure because
it may be running something very critical.
So for three weeks, the representative from engineering,
product managers, support, and people managers met on a daily basis
to prepare the next steps.
All the time, discussions were constructive
and everyone on the problem and no single person was blind for the problems.
That was great.
This is what we were working with.
I can tell you everything about this graph,
but it shows the deep of when the host still reporting.
This is on a scale of weeks,
but I can't really watch more about it.
I can't tell you which systems were displayed on this graph,
but we knew that there was a drop,
and then the system started slowly growing up.
As the system started,
the system decided to service again,
because they were used to it crushing.
And when I was in the cop, it wasn't in the cop.
The manual query verification didn't check for silenced alerts.
The automatic QE test also didn't check for silenced alerts.
The staging environments for insights didn't use the staging core.
Well, service engineers don't check their end-to-end pipelines
or weekends, because why would they, right?
So no one saw something most failing.
And the worst part is that we almost found it before
it went to production.
Because the same event was made also to the client wrapper.
And the same week the core patch was merged,
but they also were raised when we were trying to make a release build.
So my colleague created a buck report,
we touched it the next Monday,
and we decided to make the revert both patches before we investigated further.
But at this point that Monday the core was already in a product production.
So what we have here learned, and it's a lot.
It's a lot.
So first is that you should have some emergency strategy planned.
Nothing is too big to fail.
It's all bit built by individual people.
It's all individual components, and all of them can fail.
It is important to make informed decisions.
If you don't have the data, well, you have to guess and hope your guess was correct.
And you should be aware of what you are special in.
You have to own both the advantages and the problems that come with it.
For example, here, overhead is in software as a service first company,
and inside is an outlier.
So the whole organization should have ways,
documents that will describe step-by-step what should be done to prevent issues like this.
Responsible software development is about people,
mainly about people.
It's not or that much about code in the long run.
So developers should talk to the quality and journey of scoundrel part
to ensure they understand the problem.
Especially if they are junior engineers, they might need some guidance.
So don't skip them.
Don't because they guarantee the software works,
and if something breaks, they are the one who will also be blamed for the issue.
Because, well, the quality engineers, why didn't they catch it?
They just didn't know.
And talk to others, like extent visibility,
established context in the company with different teams,
different parts of the organization,
make friends.
It makes it all easier when you need to solve some problems.
For communication with the clients,
I think this is far better to over communicate.
Because knowing about issues that you are not affected by,
is far better than not knowing about issues that you are affected by,
and you just have to figure out on your own.
And if emails start to get a bit annoying,
you can always set up a filter to ignore that source email,
and that's fine, that's on you.
And it is hard to tell the ping customers that they are not getting the features yet.
I'm the product owner.
I face this on a daily basis, it's not nice.
But if it's much harder to tell them that they have to fix the problems,
you cost, and you didn't test.
Also, exit a ceiling of C, yes, it's hard to get into.
It's still on my to do list to go through the learning,
to understand how to write the policies.
But if the supported use case for your program,
develop and test with it, that's the only responsible way to deal with it.
That also works for other constraints systems,
like FAPES or other compliance.
And you should test further the analysis automatically
if you need to work with a ceiling of C.
If it's done by humans, something will slip through,
and it's like nothing happened.
And code reviews.
Well, everyone likes to focus on a code style or how the methods are split.
That's the easy way, how to do the review.
But performing a thorough behavior of verification,
it's much harder.
But it's much more rewarding when you find some bug that wouldn't be caught.
And the developer, when they are doing the PR review,
they should put in the QA head as well,
because the quality engineers will, by definition,
never discover all HKC's continuous source code.
There know how is somewhere else,
it's not just in knowing all the very spot the code can make take.
So we corrected a lot since then.
The service, the system, the service file,
fixed-line it in real-time beta, and the online five,
about five months ago.
And we plan to do more back ports.
I think they're all back for that should be public,
so you can look it up with new, new,
and maintainable integration tests through it,
that catches the regressions.
The ultimate signal checks are still there,
which is to bring me,
but it's in a really developed manner.
And we hope to tackle it in the next few months,
and engineering now also has access to the usage trends.
So when something gets, seems wrong,
we can look at the data we can see,
is our field just wrong,
or is it something that's happened?
That's not correct.
So how is the final fix look like?
What was basically one line change?
Instead of creating the temporary director in DMP,
we created the group EG home in something that was already present
in this SEO-news policy.
I think it's something like a var,
or something like that.
Otherwise they could say the same and it works.
And by the way, I don't tell the story,
well, I'm doing how to do the patch,
so that's on me.
And the tech lead for the real component,
fun fact is that I took over one week before this started.
So it was very harsh, but it taught me a lot.
And as I said, I'm PO for the whole area,
and I think this is something that should be shared with the public.
So it hopefully doesn't happen for others as well.
So that's it.
You can conduct me in my email,
or if you want to conduct me during first them,
just catch me on the website.
First them, just catch me on the hallways,
or ping me on matrix.
That's it.
Thank you very much.
Question time.
Are there any?
Please.
I have the question.
So I was always wondering,
how did the development of the community
with the edge of the looks,
and whether the black people,
like for example,
or like how does it work?
Right.
How do developers communicate with us in an extreme?
Well, for us,
we have a monthly sync in our calendars.
So we put our agenda there,
our questions.
Then we visit them,
and we get our answers from them.
But if there's something urgent,
we ping them on Slack,
and we have our context,
we are friends.
So when there's something vertical,
we can always talk to them directly,
even outside of the sketch of the agenda.
Yes, please.
I really like this in the mix,
and I use it for my own home server.
Once at work,
I'm putting it very difficult to convince my eyes
to implement it,
especially when most applications,
I work in telecoms,
so most charging billing applications.
They don't really like us in the mix.
So did you find a way to automate
some of these policy creations,
other than audit to allow it?
How to convince others to use a slain of
and to how to create the policies?
I don't have any clear answer on that.
We are solving that with some of other other packages.
I would need to allow,
give you something,
but it's not enough.
So for that,
we created some gyro cards that are shared
between our team and a slain of steam,
and they need to help us write the policies
and to ensure they are actually correct,
and we didn't miss anything.
So it's hard,
but it has its value as you said.
Yeah.
If my main,
I read the Slinging statement at the red head,
and we can maybe have a chat at the red
because if you want to confirm
containers, for example,
you are going to generate a certain policy from scratch.
That's one of the options to put your services
to do with containers,
and we have some kind of base.
We can call things like,
you are calling them about that,
but let's be about to write them in the wrong way.
So, to repeat on microphone,
look at service over there.
We'll help you with anything you will.
Wait, what is this all of?
Yes.
So in the title of the presentation,
you talked about,
he said,
it says, story about technical debt.
I'm not sure what the spark was technical debt.
So it was on several bits.
One was the restart to know,
declaration in the in the systemy service.
It should have been fixed.
It should have been there since the beginning,
but I wasn't in the team yet.
So I don't know what happened back then.
The other part was that our integration,
the suite for the two was very old,
and we were about to create a new one from scratch.
It was man thanable.
And we didn't have it yet,
but we shipped big change,
but we didn't know how,
how threatening it could be.
So that's one of the other things that we fixed.
Now we have new tests to it
that works in on older algorithms.
We need to test.
It's written in readable Python.
It has abstractions that make sense.
And,
yeah,
there is a lot of tech that we can talk about
the later we want.
Or you can just read the source code,
only to help you go find the list.
If there's nothing else,
I think you again.