Okay. Thanks. So, my name is Uli Havna and I'm trying to give you a talk about a union
unified equality feedback across CI and C.D. pipelines. What I would like to show is how
important it is to have quality, monitoring, some kind unified across the build systems
or the CI systems. So, first a couple of topics about me. I started as a Jenkins developer.
I'm almost active since for 20 years. I built in Jenkins the quality monitoring tools,
the warnings plug in and the coverage plug in. And I used in the beginning Jenkins primary for
continuous integration with industrial pipelines. So, what I have shown is how
the quality of industrial projects works. Now, I'm switched the topics I moved to education
and now I'm at the teaching at the university software engineering and software programming.
And there I realized that we have the same topics there. We want to see how students
perform while they are programming. So, we need quality monitoring there as well. So, we have
the same topics and it's just a little bit different interpretation of the results.
So, my talk will show you a little bit about modern CI pipelines and how they build quality
reports. Quality feedback and we've seen already that you are using a different CI systems
to show you results. Quality feedback is a little bit fragmented in the meantime. So, we have
a lot of tools that produce different data. So, we had a talk about fortune, a couple of minutes ago
they produce test results from fortune. And when you are using Maven, you produce test results
in Maven or you produce coverage results or you produce static analysis results from spotbacks
or check style or things like that. So, we have a lot of reports, a lot of data.
And yeah, the problem is the data that is shown is totally different if you are using
GitHub or if you are using GitHub or if you are using Jenkins. So, there is no consistency
about everything. So, I think what we need is something that is unified for all these building
tools. And all I mentioned that I started with Jenkins and yeah, Jenkins exposed this.
Yeah, the need for a quality, a shared quality model because in Jenkins we have a lot
of plugins that show some quality information. We have a plugin for a unique testing.
We have several plugins that show coverage information. And we have a warnings plugin that shows
warnings. But everything is showing some kind of quality data. And some plugins even show
it differently from build to build. So, for me, it makes sense that we have a missing piece.
We need a quality model that is shared across all those plugins in Jenkins. And we need
one that is shared across different CI systems that is the same in GitHub, the same in GitHub
and the same in Jenkins. And this quality model should gather all these quality information.
And finally, it should enable aggregation. It should provide trends. It should provide feedback
for industry people or for students. And it should provide some quality guides when we say
when we fail the build because of some quality problems. So, I took the code that I programmed
in Jenkins with a warnings plugin, the coverage plugin and extracted the common code base.
And use Jenkins currently as a reference implementation. And I noticed the interesting thing is we can use the same
quality model in the different systems. So, it does not make sense to have different libraries for GitHub.
So, we use one quality model that can be used in all of these continuous integration service.
So, finally, we should have GitHub and GitHub produce the same output on the same build.
It should not matter if we use continuous integration on Jenkins or on GitLab. It should produce the same output with the same data.
So, we centralize the semantics. So, what is a quality guide? What building tools do you have? What quality tools do you have?
And use a unique model that can be used across all these systems.
Maybe this will be a little bit more clear if I show you the architecture of the whole structure.
So, I try to provide a small architecture image where you can see how the plug-ins or the model behaves.
So, basically, we have a different quality dimensions, I say. We have static analysis results.
These are cumulative. So, we have 10 warnings, 100 warnings, something like that.
Then we have code and mutation coverage results. These results are more relative.
And then we have test results and test results are more binary. So, fade or not fade.
And then we have software metrics like complexity and things like that, they are even differently.
So, how are we dealing? This is dimensions. We put the dimensions here. You see the far arrows here into separate quality models.
That means we have one specialized quality model here per dimension.
And we have specialized models as sorry specialized parsers that transform the output of the build into these quality models.
So, we have parsers for instance, for check style or for service or for spot bugs that feed into the static analysis data.
Or we have parsers for jacoco or for cobertura that put the input from cobertura coverage results and feed them into our quality model.
We have tests, a unit results, XML files, or we have software metrics from PMW, a PMD, sorry, and they feed into these specialized quality models.
The parsers are the only thing that needs to provide it when we want to use different new systems.
For instance, I'm not sure if we have fortrend. If someone wants to report fortrend unit tests, we need a parser that takes the output of fortrend and feeds it here into our testing model.
And then you can see fortrend results in a unified perspective.
And this is where contributors are required or come into. So, most of the parsers are written by contributors.
So, I started with five parsers that I used in my projects, but now we have 150 parsers in the static analysis model where everybody who wants to show warnings about cobertura and whatever can show them in my models.
And then we have the unification, we have specialized parsers and the quality model, and then we can start with a unified approach. We have a unified quality evaluation at the end that processes the results.
And another key, you need to mention this, or I need to mention this, the quality evaluation is totally decoupled from the build.
So, the project build is still with your project tools that you are used.
So, when you are using, for instance, Chava, you are using Maven and the Chava compiler, when you are using C, you can use GCC or something like that, and you are using Make, it doesn't really matter.
We decoupled the build and what is important for us is only the results that our process. So, each build produces a lot of reports.
So, for instance, when you are using Chava, you have a console log where you see the warnings of the Chava compiler, you see XML files from checkstyle, you see XML files from spotbacks from cobertura, all tools produce some kind of reports here.
And these reports are from your build system, and if it is not supported here, well, actually we need some contributions, then it will be supported as well.
And here we start with our unified quality feedback. That means we take the reports, parse them, transform them into our model, and then we are starting our unified quality evaluation.
That means we don't evaluate the builds, we evaluate the outputs of your builds.
Let's see how does it come now to that works for a multiple CI systems. So, we have this unified quality evaluation, which is of course independent of a CI system.
And then, what do we produce? We produce a lot of different things. So, we produce, for instance, scores, how good is your change, how many failures do you have in the test, what does the coverage, what are the static analysis results.
So, we show you trends, how does it work out for the next, for the last 10 releases, for instance, or we show you a quality gates.
And this is one of the important things. We have quality gates that enforce, that you have zero warnings, for instance, or that your code coverage is 50% or something like that.
That is something important. This is also generalized. This is available for all CI systems.
Then, we have these modelings, and what we need is some feedback for the users. So, we also have a consistent feedback with same context, same semantics for all CI systems.
That means we produce feedback and say, okay, the code coverage is not good enough, or there are too many warnings, et cetera.
And this feedback, we need to present in the CI system. And here we need some customization, because Jenkins has a user interface, where you have HTML and JavaScript, where you can produce a result in GitHub and GitHub, you only have pull requests and mark down. So, that needs to be changed here.
But when we have this common thing, we can kind of have a short look at the quality feedback. The quality feedback is implemented. We are a quality gates, and this quality gates are for us, a main control mechanism for your project.
We have different kind of quality gates, and they are the same in Jenkins and GitHub or GitHub. So, we have quality gates that are absolute, which makes sense for your maybe greenfield project. So, you have 70% of global code coverage in your code.
But yeah, sometimes it does not make sense to measure the global coverage, because you have a legacy project, like in Jenkins, we have a lot of code that is not tested, but the new code should be good tested.
So, we also have a relative thresholds for our quality gates that means we can check which is the new code and what is the code coverage, only of the change code, and how many warnings have been introduced in the new code.
So, this is something which is also very important for users of our tooling that we use relative quality gates.
And finally, this is something we are using very often in Jenkins. We also kind of have a data based thresholds that means when you have your main branch, you can see how it behaves from release to release.
And the idea here from these quality gates is that you say, maybe, yeah, we want to improve even better, but we want not to say, okay, it needs to be 90%, but we just say it should be better as the previous release. So, these are the three types of quality gates you can select to choose.
So, the goal for the quality gates is not to punish, it's just to improve the results.
So, once we define the quality gates, the next question is how do we represent some, so in Jenkins we have the user interface, but in GitHub, in GitHub, we don't have a user interface.
So, what we do is we provide some structured, a pull request feedback, that means the scores, quality gates, and trends are shown in the pull request.
We have marked down for this, and maybe the best thing is I show you an example, this is, for instance, quality feedback in GitHub, where you see the results of your build, you see you have some tests here,
and code coverage of new code, you see code coverage of existing code, you configure what you want to see here. So, you can see everything, what you have configured, the test static analysis for code coverage.
What you also can see, you see the quality status, that means for each quality that you define, you see the status, okay, the overall test rate is okay, it must be 100%, that's line coverage of new code is 90% et cetera.
So, you have different fuse, and all are shown in the pull request. And if you are an IT tech, and want to dig a little bit deeper, you even can have a look at the diff, and each diff will provide some different visualizations, where we have some annotations in GitHub, where you see, okay, here, a mutation has been survived, or we have a warning or something like that.
So, this is something we use to show the quality results. And another point is, I already mentioned that I'm working in university and I'm using quality monitoring for students as well, and so we noticed that when we are doing this for students, we can use the same thing.
We need the same quality model, we have the same quality evaluation, the only thing what's changed is the interpretation. For students, we want some different interpretation.
I try to summarize this in a table. So, in industry pipelines, we have quality gates, and these quality gates decide if something goes into a release or not.
For student projects for assignments that students have in my courses, they get points for code coverage for warnings, and finally they get a great and automatic great, where they can see how are they doing from learning, or how they get some feedback.
And I have hundreds of students and it's very simple to get them a first feedback before I go make manual review. And in projects, in real projects, you need a pull request reviews and a quality gate is something that helps to simplify your reviews.
So, the lessons learned, yeah, CI systems, they differ less than we expected. So, the quality models are the interesting part, but the CI systems, they come and go, and it doesn't really matter for my tools.
If there is coming a new star, then we will report the user interface, but the model will stay the same. So, the scoring, we are using a little bit more complex, not just simply failing, but yeah, the scoring to put it in a library helps to reduce the maintenance costs. So, we have a single module that helps to show in different CI systems.
And finally, yeah, the key lesson I have with this system, or with the implementation of the system, it is totally independent of the CI system, we can replace it anytime.
It is independent of the user interface, we use Markdown, which is visible in GitLab and GitHub. And yeah, finally, it's open source, that's also important to get contributions for different passes.
And yeah, finally, I think, so the CI systems built software, this is one part, and the other part is we should have quality feedback, and this is something I am providing with my tools.
So, yeah, try it them in your projects, I hope, yeah, it will help to improve the quality of your project as well.
So, thank you for your attention, and now I'm ready for some questions.
Can you use this to test software that has machine learning models and neural networks?
Okay, the question is if I can use the software that uses a machine and learning model in it.
Yeah, I'm trying to create a software that uses machine learning neural networks, but I don't have a quality monitoring.
Yeah, the question is if it works with a software that uses a machine model, but yeah, it can if your tools produce some output that I can pass.
So, if you're tooling produces a report of the tests, a report of the coverage, a report of the warnings, then it is possible to use.
If there are no passes yet, we need to provide new passes, but that's typically a one day thing, so I get a lot of passes from different areas.
So, if it's not supported, have a look how complex your output is, and then we can integrate it.
Yeah?
The question is where does the auto-grading run?
It's part of the CI system.
So, in GitHub, it's a GitHub action that you're actually binding in.
In GitHub, I didn't find a similar thing, so it's part of a Docker container, and you call the action in the Docker container.
And it takes the output of your build, so it must have access to your build files, and then it produces a result.
Yeah, that's currently a student working on this project.
We are recording the dead, the reports for each build, and storing it in the build, and in a pull request or much request build, we search for the last build on the main branch, and then we take both result files, and both result files are compared against.
I think it's a little bit simpler because we have references in the model, but in GitHub, we need to point to the last build of the main branch, which is always not the right one.
Yeah, but this is something we're currently improving.
Okay, the question is how do we report trends in GitHub and GitHub we don't have trends yet, and because I don't see where I should show them, so in Jenkins, we have trends, because we have it in a database, but in GitHub, I don't have this database, so this is something we need to add on, or I'm not sure how we can do that in GitHub or GitHub.
Okay, another question, yeah?
Yeah, I think now we're going to get those AI agents that are creating the code and everything.
Have you thought about how this is about this type of system coming to that?
Have you thought about the type of code by AI?
I don't know, then I think those進 type of system is much more important, not to guarantee what to get out of the main branch.
Yeah, I'm not sure how this relates to the project, because as a the question is how AI generated code.
How the system fits with that?
How the AI generated code system fit with my code base, so I'm currently totally independent.
I'm taking the results of a build, and if the build is produced by code that has been generated by AI, it takes the code, what the build produces.
So I have no knowledge about the content, and I don't have the tools, the tools are provided by you.
I just take the results of the tools.
Yeah?
Yeah.
You mentioned everything is the cover from the CI CDC system.
Yeah.
You basically provide a framework.
Yeah.
For the trends, you integrate with the check in second.
Doesn't make sense to create a separate data, but it doesn't, it doesn't, it doesn't, it doesn't, it doesn't.
Everything that's moving.
The information from out there is integrated to check is then looking for the deep love and information.
Yeah, the question is if we shouldn't create a real application that has a database, and so where we can put all the data into, yeah, this would be a good idea.
Maybe this is a good master thesis for one of my students, but we have, we have it not yet.
So yeah, it would be possible.
Yeah.
That's the back, yeah.
So we have, you can have a look at these links.
We have 150 formats for the static analysis and 10 or 15 formats for code coverage.
So you can have a look if your format is supported.
And as already mentioned, most formats are a couple of hours to implement a parser.
So you just need text input provided in my Java model.
And that's really easy to provide if you at least maybe a I can write such a parser now.
Okay.
Thank you.