Okay. Hello everyone. Welcome to my talk. That's that the flakes. Actually it doesn't
have anything to do with laser guns if you think. It's just like trying to eliminate those
pesky flakes that are ever present. My name is Daniel Hiller. I'm going to
give you a short overview about flakes today to show you how we are trying to detect it
before merge. This system doesn't work as we wanted. Then I'm going to talk about the
approach that I've been looking at and that we're trying to implement and then I'm going
to talk about the plans that we have for that. I hope I have a bit questions and answers
time but I'll try to rush as fast as I can. Hopefully I'm not too fast. Hopefully everything
works as expected. As I said, my name is Daniel Hiller. I'm a software engineer and the
OP-shift virtualization team at Red Hat. My main concern is Cuebert, the ISIS system and
automation general. So let's start right in. I think a couple of people might have been at my
talk last year, hopefully. Okay, so I need to give you a bit of the idea what it is.
This is about what you see here is like a couple of test runs for a set of test lanes. Please
pay attention to this number. This is the Git commit ID. So what you're seeing here is the
couple of test runs all on the same commit. Who can tell me what is wrong with that? Yeah, exactly.
Because this one failed and this one passed. So on the same commit, nothing has changed inside
the code base. There has been something that has been failing the tests somehow. And that's exactly
what we're talking about. That's what we're trying to find. These are flaky tests. The flake is a test
that without any code change will either fail or pass in successive runs. By the way, you don't
need to make photos. I think you can download the slides right away. There is like you do
recording, including which you can take a photo. So just a bit of statistics. There is a survey
of flaky tests that I talked about last year. 90% of the developers claim to deal with flaky tests
of those like 23% found that there was a serious problem. And they were like the 15% of developers
that were thinking it was a frequently encountered problem. We're dealing with a daily. So I
think it should be important that you care for your flaky tests. And not say this is fine and move on.
Okay, so just in a nutshell, flaky tests cause not only problems for the individual
contributors, but also inside the community. For individual contributors, you see that
we have prolonged feedback cycles for them. So they might rerun test lanes again and again because
they have like failing tests which they obviously are caused by flakes. So they are starting to
not trust the test anymore. For the project community itself, this is like slowing down everyone
because like those rerun test lanes are consuming resources that could be used better in a better
place. Also, it reverses acceleration effects. For example, like if you're doing like an octopus
merchant trying to test everything together, one flaky test can blow up the whole system.
Also, yeah, as I said, there are waste things that I resources in general.
We're not Google, right? Okay, what we have currently at our
Cooper project is we have a lane that actually runs on each
commit which you do on each PR. We are running a set of change tests
five times in random order because that gives us like an 88% chance to catch the flaky test.
But the thing is flakes don't, it's similar to Heisenberg, right? So you can test 10,000 times
and it might show up in the 10,000 one-stime. So that's a problem.
At our place, how we implemented it, we have the problem that we have like the majority of tests
are in the range of 10 seconds to two minutes. So we are talking about E to E test here which
take especially long because you have the full system which are needing to spin up and then
you need to run your system then. So like it has only an 88% chance. So like there is a 12% chance
that you might miss them. We also like currently just use a shotgun approach of finding the test.
The thing is we are only looking at the test files. We are not doing like an AST traversal
which might be a point for approval on itself. And yeah, normally our rerun test lanes on average
take around one hour which is faster than the majority of the remaining lanes but yeah, it's still
like one hour which you need to wait. And yeah, for if you are having touch too many files,
obviously you have to rerun all of them and there might be hundreds of tests which might
take longer than a couple of hours which at some point needs to get kept.
So I was researching and trying to find something about that and a stumble of the
Kenya approach. This was referenced in a paper from 2022 and they were stating that they were
able to reduce the time costs and monetary costs of rerunning at an 80% average of 88% which sounds
very promising. And they do that by leveraging AI. In this case, it's like a random first
which they use to give you a prediction of whether a test is flaky or not. So they have upper and
lower boundaries which you see here. And what happens when a test is in the lower threshold,
we just say it's stable, it's okay. And the upper threshold, we say it's definitely flaky.
And in the middle of that, this is where we need to rerun. So we are trying to cap the number of tests
into a smaller set which we just have to look at and rerun because the only way to find a
whether a test is really flaky or not is to run it as often as you can. So if you're a bit familiar
with how like a random force works. So you have in general you have a set of features which
is all properties. So you have a vector of properties which are mostly the time just float values.
And they give you numbers about how the tests are constructed. So what you see here in
the upper part for example is like runtime things. So these need to get looked at at runtime.
You for example like have I just only give you two examples like for example recon max memory.
But also you have the other features which are constructed from static,
static code analysis. For example like looking at the AST depths or a
stoichiometric complexity, if you know what that means, stoichiometric complexity is just
just as a side of the number of execution paths inside a method simply. So what we want to do
we want to implement that. We want to have that as a replacement for our existing thing,
just rerunning things we want to cap down. So the thought was what problems do we have.
The thing is that the canier is implemented, but it's in Python. And we are talking about
Cupid which is fully implemented and goes. So also the code base is also implemented and
go so we can't use that. So we need to reimplement that. Then also we have the question where
do we store the runtime data? Which might not be entirely necessary, but since you want to optimize
a bit, you should probably think about that. The other thing is like when do we capture that?
Do we have to run the tests always before so that we have the data inside the future vector
or do we fetch it from somewhere else? And the last thing is like the data signs at GoLank.
Python has well known and established frameworks and GoState actually we didn't know.
That's the point. So our plan at the moment is this should be the implementation
of what we want to do. Like I said, this is the fetching of the prediction.
So what we want to see is we want to have the exact set of change tests which we for per each
test we run the prediction to put it then into the random forest model to get the prediction
of how flaky it is and then have a set of tests that we want to rerun. And as a side effect
we get the positive and the negative predicted labels. So the parts that I must admit I wanted to
have a prototype and show you a bit but it wasn't I didn't make it. I'm sorry for that but yeah
I'll give you a point to a to a pull request that I'm working at at the moment. So largely
this is for components. The tests that extraction, the feature extraction, the model prediction
and the model generation and obviously also the model hosting which we're also doing and go.
Then we have the test lane which currently is like a system what we are using we're using
Kubernetes prow which most of the time we we have just like batch scripts that are running our
tests as a base execution. Obviously we're using go gink or framework but that's just a
and obviously like the model deployment on the cluster which we which we need to access at
some point from all the lanes. That's so far the plan for v2 then we also have like the improvement
that we want to plan on that one which you will be like a prow external plugin which we can
use aside from the test lane. It can run on the side and give you additional information about
the features that actually have been seen on the test cases. For example like the
psychromatic complexity is something you might consider as a room for improvement on your tests
and that's a valuable information that you can give the user spec and that's what we're going to do.
Two minutes. Okay. Yeah so as I said sorry wrong direction a bit confused. Yeah that's
what I already said. Yeah further improvements as I said like the components of the future
vector provide in such a little advice to the contributor as I said like psychromatic complexity
and but the but the other thing is that we can obviously increase the number of reruns we are doing
at the moment we are just running five times but if we are having a largely reduced set we can do more
and we can so be more exact and finding the flakes. That's all I have. I have a couple of links
here in for cubic resources there is the link to my initial pull request. I have held another
presentation about the flakes which you also can find here. There is the canier link for the
paper and for the implementation of people of you are using python you have something that you
can already use which is great and hopefully you can use something if you're using golang soon.
So yeah that's it for me I've probably a bit of time of questions hopefully. No sorry.