So I think we can start, so thanks for joining, then I mean assuming that you're not here for me, but so two years ago I came to share a little project I was working on trying to solve a problem that I was facing.
So the idea is, I'm maintaining quite a lot of git repositories.
Is it moving?
Do this this way.
So basically I'm mentioning quite a lot of git repositories.
Those git repositories contain quite a lot of informations and those information can get audited pretty quickly.
And so my problem was how can I identify those information and then to be able to do whatever was needed?
Sometimes it's just updating a version, sometimes it's updating an installation links.
So it was really about being able to identify what to monitor and what to do when we find something that is audited.
So at the beginning I was working on a Jenkins project where we have like thousands of git repositories.
So we already use scripts like Python Ruby and so on to catch those audited information.
We were already using as well to like renovate but renovate but it under but the problem was it was not flexible enough for us to be able to identify all those changes.
So we started working on a new project, which is like, I'm pretty bad at naming things, so just to see like that of this stuff.
The best way to describe this is it's a declarative dependency management.
So the idea is really to be able to explain, so to explain what we are monitoring and what we want to do.
So the core has been pretty stable since the beginning of the project five years ago.
So we have like five main resources, the source which defines where an information is coming from.
So for example, in this case we are monitoring the Java version 17, but I could be like Java, I could be a lot of different things.
We have like a plugin ecosystem, so we can really try to adapt to the different scenarios.
Conditions mean this is a condition that needs to pass before moving to the next source.
So that would be, doesn't make sense to update, I don't know, a hand chart if the image is not available, the current image is not available.
So those are things you want to check, the target is what you need to monitor.
So for example, in this case, that's a target of kind of Dockerfile, so we are monitoring Dockerfile with a specific Docker instruction.
And so if for some reason the Java version is not set to the latest version of Java defined by the source,
then update rely is designed to update that information, so we can move to the next, so we can just update information.
As Sam resource is Git repository is that we want to interact with, so it are fetching information from a Git repository or updating a Git repository.
And finally the action is that doing something one of the target is updated.
So the idea here is really to define a pipeline about how to update files.
And then run the commands from a CI, from the machine, from wherever it wants.
Yeah, we made the same. So this is an example on the Jenkins project where we tried to monitor all the Docker images that we are building,
and we have the core images for different Java version, different architectures, for example.
So this diagram was generated by of the CI, so the way it works is, you just show the different components of that specific pipeline.
And there is a cure code if you want to see what the manifest looks like because it's pretty long one.
But what we do here is we have one source where we monitor the Java version, as I show in the previous example.
We have one condition, which is check that we have a Docker image available for specific target for all the different architectures that we are monitoring.
And finally we have a lot of targets, and those targets represents different files that we are monitoring, like,
a Docker file, a packer because we are using Packer, so we monitor HGL file, we monitor.
And so yeah, we need to be able to monitor all those different files, and then open one PR, so that's someone can just validate that.
We want to use that latest version.
So as I said, the core remained the same, it's always the same, like running kitsch code, fetch information, and so on.
We have kind of built over time, kind of plugin ecosystem, where each plugin is dedicated to a specific problem.
So for example, the yellow file is about defining what kind of file you want to do, what are the key, want to monitor, and what they can value you are expecting there.
And so all those plugins work, so we work with different git providers as well.
This approach about like being very flexible about what we need to do is on one side it's like quite simple, because if one day you're not monitoring the version 17, but you want to monitor the version 21, just one variable to change, so you're not like having to go deep in some complex scripts.
The other part that I like here as well is you just have one process, so you know that you always monitoring a specific piece of information, so you group all those changes at once, so the thing know what you do.
The counter side of this is you have to write manifest, so obviously that's, I mean, the less manifest you write the better we all, we are all, but.
And more importantly, what is really challenging with this approach is that you need to know what you are monitoring, so you need to know an advance what all the five, so if someone in the team tomorrow decide to, I don't know, had a new verify.
Someone, that person also need to know that the data automation to be able to catch those updates, so on one side it bring a lot of flexibility, but on your side it also.
For the team to change the mindset about all we're thinking about, okay, I'm bringing something to the git repository, is that for good that version become outdated.
And if it's the case try to do something that could be sure that tomorrow it's still working.
Um, over time the project evolved as a need, as only developed as well, so we, we started to think about like, okay, we have to write manifest to define what or approach, what the procedural looks like, what if we could detect automatically based on some git repositories, what we need to monitor.
So for people familiar with tip and about, you know, who's familiar with tip and about a renovating room, okay, so that's that's pretty much the closest idea here is that we let up the July to analyze a git repository.
To detect like it has a map and file or go go modify it for go and then it generates the manifest and apply them straight away.
Um, the purpose was not to replace renovate or tip and about the purpose was more to be like, um, something I would work well with it.
But yeah, that was it turns out that, um, we could also.
Automate different, up to make it more easier to to make different, again, scenarios. So this one is an example of a very small manifest where we want to to make this in the hand project.
And so if you have this more manifest that tells us like monitor any hand dependencies in that git repository, um, do whatever needs to be done.
So the, at this moment we're not any more about defining advanced what we need to update, but we're trying to guess which can with a different set of challenges because each ecosystem, um, as its own set of rules.
So for example, I mean, you know, that in the Docker, the Docker ecosystem, you can use whatever version you want.
And so those the different version can mean different things to different people.
Um, so it can be rich for you to identify what should be the next version. So that can be the different, it's really come with a different set of challenges.
But it was kind of you sort of from time to time. So the biggest benefit of this one is like the same with tools like depend about is, um, you don't necessarily know need to know what you want to update.
Just let the tool detect those, but then over time you also need to be able to to limit some updates or to say like this, this dependency I don't really care.
I'm fine to update that once a month or I want to group those informations together.
Um, but on the other side, that's the same approach with all of those where it's really noisy because you quickly get like hundreds of dependencies in your project.
Um, and you need to to decide when to what to do when something change.
Um, those are the main main approach where we try to be as flexible as possible to be able to identify all those policies.
But what we noticed of the time is that we ended up copypasting the same.
Yeah, we'll manifest over many different repositories. So we were wondering what if we could just have like an update policy.
Um, and that's what we did. So we know what we have is that we have the ability to publish those policies to a registry, like Docker Hub, GHGR, the Taiyo for example.
And we can version those policies. So we just try to have like generate update policies for the different projects.
Um, and so we now have like one fine name of this, you like compose that channel and then there we can list like all the policies we want to do.
We can just change the parameters. So it became a lot easier to scale about the different repositories work.
So the way a policy, I mean, if you're curious, there is a good repository name of this.
Eli slash policies where we have like a lot of policies we use for various open source project. We are working on.
But a policy like a bit similar to him.
Um, um, um, um, um, hurting artifact, where we specify metadata. And then we can, yeah, publish that to, to our registry.
So what we like with this is we moved from, um, now we have like one repository where we maintain all the update policies. So we really have like few people looking into those policies and
You can more is your to to change those a bit policies over time because you can version
them.
So you can also have a different policies if you're on prediction environments or if
you're in a deaf repository.
The control side of this is you just have more templating in place.
So you can end up with very complex policies if you're depending on what you're trying to
do.
So the last part I want to finish here is I just want to thank all the people who contribute.
So this project started what I was working on a Jenkins project as a side project and over
time we started to see a lot more people contributing to this small project.
We also have sponsoring from time to time so thanks to perfect HQ is also sponsoring
the project which gives us the ability to have stickers and some to sponsor some of
some of some stuff.
So yeah that's that was nice to see this project evolving.
If you're curious we have a chat where we regularly interact with the different people.
I'm not sure I'm going to have that kind of time to go on this project so I'm going to
just keep this one that I get.
So yeah I'm going to stop here.
So yeah that was it took a quick update on how this project evolved over the past few years.
That's it.
Yes.
How do you deal with like tricky versions of Russian numbers if you have like timestamps
or something like that?
So the thing is when we specify your source information you specify for example Java let's
say don't have an example that I show and then we also have way to filter the version and
then you can have different roles like semantic versioning dates.
We have a few of them like that but the challenge you have here if you go with a declarative
approach and that you know what you're looking for you can respecify I'm looking for the
latest version matching a date on the line you can specify some constraints.
When you go with the auto discovery which is like like the button above it that's a tricky
part because you don't always know when the meaning of the version and sometimes you can
have version that are very similar.
So for example some projects use a version as a date and that date could be like a huge
major version in the context of semantic versioning and so you have when you have like
very old project you can become really tricky to know what should be the latest version
and so that's why the declarative approach is really useful.
So typically if you have internal dependencies and you know what you're monitoring you can
just specify I'm looking for the latest information based on some date for example and you
can also specify if you want to transform that information.
So if you're for example some project to be version where they have the project name dash
and semantic version that means as it's not really semantic versioning because they have
the project name in front of it so you have like extra things to do if you want to know
what should be the latest version but that definitely something went out.
Can you tell it to stop upgrading when you definitely know that there's a major version?
Yes.
That's something we can as well do we have plenty of example once again that that one comes
with one questions about if you know let's say you want to stop at the version 20 so you put
some rules I don't give me a version after 20 but then at some point someone need to update
the update policy to know that no you're looking for a version 21 for example and that's
where the policy is we is useful is because you have one policy that says don't go over
20 and then the next version will be don't go over 21 and so then you have the control
to decide which gets the repository is monitoring the version 20 which gives the repository
is monitoring the version 21.
Can you send that it would all automatically break the R?
Yes.
Yes exactly.
So that's the idea I can show an example do I have here?
One minute.
Okay one minute and then I finished.
For example if I go, I think you can send from, I think you can see I.
The term.
If I go for performance so there is a bunch of policies so the thing is it's not only
used to track I mean we can use it to track a lot of different kind of information so for example
here we specify the git repository is we want to interact with.
Now we have the action at the bottom.
Yes here.
So here we specify what should be the title of the PR what should be the labels of the PR.
You have like a few configuration you can do like for example you can also open.
A PR on the upstream repository so for example if you fork your repository you can automatically
open it upstream.
You have quite a lot of functionalities usually we use it all the time so if someone needs something
just add the feature depending on the needs and then starting from there.
Yeah that's what you are asking me can we filter version.
Yeah definitely you just have to specify what's the version you're looking for you know and then you can do
things like remove the V prefix and sound so it's really you have if you think if you know what you're looking
for it's very powerful to do.
It's over.
Next.
Thanks.
Thank you.