Okay, so my name is Lucas Tavarek. I'm a technical leader at Intel and today's talk is about AI-based failure
aggregation. So in our CI pipelines we often see hundreds of test failures every day and most
of them are very similar but not exactly the same. Because of this minor differences, our
engineers had to expect time to manually analyze these failures, repetitively like over and
over again. And in today I will show you how we reduce the overall noise in the CI system with
text embeddings. Let's start with a quick and high level overview of the original workflow that we
used. So we had on the left-hand side regular CI CD pipelines that like build components and
run tests. Then we have a monitoring system that automatically gets the test results from the CI CD
pipeline, performs some initial analysis and then reports the test failures to an engineer.
The engineer have to make a decision whether to update existing bug report with a new
instance of a given failure or treat it as a completely new bug, create a new bug report and if
needed also start a regression isolation that we will try to find the faulty commit and
reverberate. Now let's talk about the scale which is quite important in this solution.
So on the x-axis you have a number of tests on y-axis the percentage of failures. So from the
scale perspective if you have a low level level of failures then you are in a happy place you
often don't need some complicated system or process to handle these failures.
If you have a relatively low number of tests and a high percentage of failures this is still
manageable with some semi automatic way you can handle all of the failures. However in our case
we had like hundreds of thousands of test basis executed each day and hundreds of test failures
that needed to be analyzed every day by engineers. So this did not scale and we needed to find something
better. So this is a overview of the desired workflow that we try to implement as you can see
most of the elements of the diagram are exactly the same with one major change. The engineer
or the human in the loop was removed from the like critical path. So we tried to create a system that
will automatically get the test results and by itself make a decision whether to update the bug or
create a new one and start the regression isolation. The only interaction with the engineer was to
create to send a some other report for verification. So in short our goal was to create a fully
automated agent and the main problem that we faced was to how to determine if a failure is a new issue
or an already reported one. So here are a few potential solutions that we considered.
So the first one the most basic one is to simply have no aggregation at all so each test failure
is a new bug report but we simply like move the issue from the test reporting side to a
bug management side. So we have a lot of duplicates and also a risk of a waste of resources as the
we unnecessary start the regression isolation process. Then we could go like once the further
and try to aggregate the failures per test piece. So if we have exactly the same test piece
that is failing over multiple bills then we simply can try to treat them as a one bug report.
However, we still can have duplicates because the same bug can affect multiple test cases and we
still have like noise in the system. Another issue of this approach is so called nested
regressions. So we have exactly the same test case that on one build is failing due to some
numerical errors and on another build started to fail due to like segmentation fault. With this
knife approach we will not detect this and we will treat still these two issues as a one bug.
Next we could also try to do a direct log comparison. So simply get that error messages from
test failures, try to normalize them. So like remove time stamps, memory addresses or anything like
that that changes from build to build but didn't change the error signature. However, this
text operations are highly like complex and rule based so we will need to like maintain this
over and over again. Another option is a new machine learning model. However, with this
basic approach, dozens scale and it's really time consuming and error prone mostly due to the
need to prepare a dataset. We can't go with like a few hundreds of examples. We need a few
thousands and we need to label this data which is quite expensive. Of course there's another way.
Now let me introduce three main concepts that are essential to our solution.
First one is text embedding so what is even a text embedding. It's simply a numerical vector
in a multidimensional space that represents a meaning of a sentence. So if we convert the
like sentences into a vectors, then vectors that are close to each other should represent a
similar meaning. So the like worth in a sentence can be different but the meaning will be treated as
the same. Then we have a vector similarity search. So if we have these vectors, then we need a way
to decide how similar they are. To do this, we use the standard way which is a cosin similarity
metric which is simply a cosin of an angle between two vectors. So if cosin similarity is
near one then we treat such vectors as highly related and if the cosin similarity is near zero
then we treat them as unrelated. Finally the buy and coder architecture. Here we have an example
of a training stage of such architecture and the model. So we have input sentence A that we pass to the
and coder model. In most cases this is some like AI model like Berter and things like that.
That we create an embedding and then we repeat this process with another sentence that we want to
compare with. We go over the same encoder module, generate another embedding and then based on the
cosin similarity we decide whether these embeddings are similar or not. With value of cosin similarity
we can then compare them with the reference data in our like validation data set and update the
whites of the encoder or in other terms send a positive feedback to the encoder if the embedding
was good and like we get expected value because I similarity or send a negative feedback
if the like embeddings are far off of the reference value. That's the like theory part. Now let's go to
the implementation. So first we use the sentence transformers package that is available on the
pie pie it is written in Python. As you can see with one line of coder we can load the already
pre-trained model that is available online. Then we can create some example sentences that we will
work on and finally we can simply call the model.ncode with input sentences and in such
case we will get three embeddings because we had like three input sentences each embedding with
384 dimensions which is related to the model that we used at the beginning. Then we have some
utility functions from the sentence transformers package in that case we have like similarity
that we can call with the embeddings and we will get the cosin similarity between these vectors
in like tensor object. If you are not using the Python ecosystem then I really recommend
the hugging faces text embedding inference project which has some pre-built Docker images
with like CPU support with GPU supports and it can also be easily deployed on the Kubernetes
clusters. In such case we can easily scale the solution so the server can serve given model
and accept multiple requests at the same time. In this case we have web interface so we simply
send a web request the server with an input sentence that we want to embed and as an output
we will get the embedding vector. Then the question is which model should I use? So here we
comes the M tab leaderboard that is available online. Here are some like examples from the
levered their board and a few essential parameters that you need to take care of.
So the first one is memory usage so as you can see the first model uses around like 44 gigabytes
of memory so to run is you need a really powerful GPU or even multiple GPUs. On the other hand
the last example is the one from the sentence transformers example and it uses only like 100
megabytes of memory and you can easily run it on CPU. Then we have embedding dimensions
so more dimensions interrupt the interior leads to a better accuracy. However the it's a cost
of storage that is required to store these vectors. Finally we have max tokens parameter which represents
the length of a text that we can embed in a single go when we call the model. So if we have like
large, last text input but the model can't handle it then it will in most cases try and get it
to the like number of tokens that it supports. We have text embeddings we have a way to
compare them then we need a way to store them. In such case we use the PG vector extension to post
the SQL. With this extension we have a new column type for vectors so of course we can insert the
vectors to the database alongside any other standard data that is available in postgres and then we
have few dedicated operators that allows us to query the database to retrieve vectors that are
related to each other. Here we have like overall overview of the workflow that we prepared so this
is we have a Python orchestrator that automatically gets the log from Jenkins that runs the tests
then we have we are extracting failures from the error signatures from the logs from the
Jenkins. We pass it to the hugging faces text embedding inference instance to get the embeddings.
We've started embedding we are going to post the SQL to get the information whether we already
have seen similar error in the database. If that's the case then we create automatically create
black report and also automatically start the git bisect and git forever workflow to
revire to the faulty comment from the system. Of course if we have an embedding that already exists
in the database then we only update the back reporting the database. As always there are
there is a room for improvements so here are a few recommendations from our side first is to analyze
more logs so there are two cases the first one is simply the case when you have a really long error
messages that you want to embed. As I said previously the default option is to
truncate this log file and by default most of the services will truncate it from the beginning.
From our experience in most cases the error messages and the like real error signature
is available at the end of the log so the like basic option is to simply truncate from the
bottom of the log file and then you will get like better output. Another option if you have resources
is to simply use the bigger model with more max tokens parameter that will handle your example.
Then there is an option of multiple log files so for example you have error messages from your
testing framework and then you have some like log files from let's say web server that is tested
and you would like to aggregate the failures based on this to like input streams.
So the basic option is to simply get these two logs, merge them into one file and the embed
everything as a one vector. Another option if the logs are too big then you can try to use an
LLM or any other option that we will try to select more important log. So if for example you have
some segmentations faults then we maybe we don't need to like analyze some basic log files and
go straight to like some system core files or anything like that. Another option is to embed only
error signatures so if you have like few thousands of lines in the log file and the error
signatures are only like taking I don't know like 20 lines or anything like that then also with a
help of LLM or any other such solution you could try to extract the error signature
and then embed only this signature instead of a whole log file. Then we have fine tuning.
So by default the models that are pre-trained are like pre-trained on like a lot of big data sets
that are like generic and generalling in all. So we found out that there are some
issues with domain specific patterns. So for example we had a test that compared to vectors
and reported the percentage of incorrect values and for example if we had like in the same test case
2% is 2% of difference and then we had another run on another build that resulted in 80% of
difference by default it was treated as a the same error message and it was aggregated because
like LLM didn't know that the such issues should be treated as a separate ones. In such
case we can create a small data set that will like optimize the behavior of the encoder module
that will force him to separate this domain specific patterns. Finally over the talk I
presented you the failure aggregation part there are which focuses on linking the failures that
look the same so we have the same error message. There is also an option for failure correlation
and by that I mean to link failures that look different however have the same root cause.
So if we have some unit integration tests that we know from the like statistical data
that will in most cases fail together with some end-to-end test then we can automatically try to
also correlate these failures and present it as one in a one view for further processing.
So to sum up with text embedding for failure aggregation you can improve efficiency of your
process by minimizing noise and I really encourage you to give it a try as the entry barrier
is really really low all of the packages are open source there are free trade models available online
and you can start running them even on a CPU. So the key takeaway is that text embeddings
is a low effort way to turn CI noise into signal. Thank you.
Now I'm open for questions.
We didn't benchmark that but like the main issue is the scale so we've like hundreds of failures
or like thousands of failures here we didn't want to like go like with this option
having the the infrastructure that we already have at the end-to-end.
Yeah so the question is about the false positives and negative so we measured the
Rico as I remember so we wanted to have like no false positives. We wanted to have like multiple
back reports instead of one that correlated or aggregated the wrong issue with itself and on
hours like testing data set we had like a 90% so it was really really like good from the get we didn't
had to like optimize it too much.
So the question is about the stability of the
tests and how they are aggregated so in such case we are not aggregating over the like single
build run we are aggregating across the like multiple builds so if you have sporadic failures
that are like occurring once a week or something like that this solution will automatically
detect them yeah because we already have them in in a database.
Yeah so the question is about the get-by-sext and whether we try to automatically detect
whether we should update the test or fix the solution. So now we have haven't done it there are
there were some work where we get the comments that could introduce the regression from multiple
components and then try to use an LLM to ask him a question which from these comments is most
highly related to the fail that we see in the system yeah so there were some work but it was not
part of this solution.