Hi, some of my name is Pierre, and I'm the current chair of the Apache Login PMC.
I wanted to talk to you a little bit about what happened four years ago.
We've looked for shell to see if we can do something better.
Right now, if we can do provide a solution faster,
I will not tell you that log for shell will never happen again, because that's false.
It's just a question one, it will happen.
So a little bit about log for shell.
Actually there are two versions which are very distinct.
One is log for j1, which was written by check key in 2001.
And while check key soon after Abandoned,
the work in Apache and started the successor for j log back.
And actually log for j2, which was the library affected, was done by another.
An entirely different team, most of these people live in the US,
which is important for the timing of the vulnerability.
I wasn't there.
When the scarp ends, I joined the log for j team.
One month later, I started contributing.
It was popular that there were a lot of small bug reports, it was easy to help.
Yeah, and while since June last year, I'm the chair, the PMC chair of Apache Login.
Okay, so some of you remember the 7th and 9th of 10th of 2021,
it depends which side of this Atlantic are you.
And work happened.
It happened that Apache is more mistake, because many Java applications
to be vulnerable to remote code execution attack.
So to give you an idea of what kind of mistake was it?
Look for j allows you to configure.
There is a pattern layout that allows you to configure how logs are printed.
And it has two kinds of placeholders.
Once are called message patterns with percent, for example, printed date,
the message, and the other are similar to Java properties to what, you know,
for j1 interpolated Java properties, but this system now is extensible
and you have many sources.
Just system properties and environment variables and so on.
So the problem was that when you have such a string,
instead of interpolating the lookups first, it interpolated the patterns
and among the patterns you have the message, that can come from user data.
So when I request arrived to the server, many developers
just what arrived in the message.
And so the message is expanded.
The message can contain lookups and the lookups are evaluated again.
And there was a lookup that was particularly dangerous,
which was GNDI, I think that up to that moment,
except all the Java developers, nobody remembered that you can use GNDI
to download stuff from an LDAP server and execute locally.
So the combination of this to fact came affected millions of applications worldwide.
So the flow, this inversion, was discovered actually in 2014,
not 21, but unfortunately, it was classified as a feature.
And we are very committed to backward compatibility.
So if somebody was using that feature, we gave him an option to opt in our app.
So how many days did it take to solve this problem?
It took from November the 24th to December the 9th.
November the 25th was Thanksgiving.
So in the United States, let's remove a couple of days of work
because everybody was with their families.
Another characteristic of this timeline is that unfortunately,
we published a patch on GitHub on November the 30th,
which made the patch available for users for 9 days.
Another problem here you can see is that the release vote,
the first release vote in Apache,
we have normal release votes are 72 hours.
But it's written everywhere 72 hours, 72 hours.
So most projects being that it must be 72 hours.
Actually, my colleagues discovered later that it doesn't have to be.
So it can be 8, but every member of the PMC must know that there will be a vote.
So this is a problem.
So you need 72 hours to tell people that you will vote for 8 hours.
Okay, so this is the timeline that we can correct.
We can reduce because it's our timeline.
Another timeline that I don't know how to improve is this one.
This is from Sonataip, the follow the downloads of vulnerable versions of Lockforge.
And the timeline ends in July, 2024 with around 10% of downloads were for vulnerable versions.
If you account that most big companies have proxies, so they don't download it every time,
well, it's a lot.
And the newest, as you will state, of Java report that I read before,
for them says that still today 49% of responders say they are affected by Lockforge.
Vulnerabilities, I must ask them precisely what does it mean,
because we didn't make any Vulnerabilities in the past three years.
Okay, so this is the summary of how the Lockforge was solved.
And let's see if we can do something better.
So after this problem, we tried to identify what are the main problems to reusing fast fixing.
One was the tests were flaky.
So there was a big chance that I read this build face.
Sige generation was not slow, but hyperslow.
It was a couple of hours.
Do you the technology that we used?
It was difficult to make a release.
So we had two people that had the time new, how to release for Jay.
We identified also a problem that we had to many features.
And they were all included.
And what, some minor documentation problems.
So there were two companies that had solved this.
One was tied leaf, that started supporting Lockforge in January 23.
And then with Christian and Vulnerability,
we got a big sovereignty grant that allowed us to work for 60 months just on a project.
So Vulnerability and Vulnerability are not involved.
So what did we achieve?
What did we still need to achieve?
So as I was saying, releasing the project was very complex because you had to run everything on your machine.
We had a CI, but the CI was not allowed to make releases and was not to put a test.
was not programmed to make releases, so all the stuff was on the release manager and there was
one candidate that could do it. Nowadays everybody can do it, so if I'm here I've
forced them and something happens and you have a PNC member can do it. So we managed to
totally automate the first part of the release up to the vote, so the release manager needs
just to do stuff that you must do. Prepared release notes decide the work to cut the release.
There are still a lot of manual things like publishing to next source then publishing
to the subversion repository publishing the website. These are not handled automatically.
There is a project inside. In Apache Apache Trust, this platform and we believe
in a couple of days, years, they will do that for us. So we will just press a button,
the votes are there, release the cracker. Okay, what are the most important parts
projects that help us with this automation? As I said before, we released on our own computers
because we need a trusted platform to make release so that nobody can mess with it.
Thanks to the reproducibility, the reproducibility will be its project,
ever thank you. We now have reproducibility which means that the CI can make the build
and we can verify it locally. So once we verify it locally, it's okay. We can release it.
Okay, depend about updates, our dependencies, GitHub actions is what we chose for CICD that
before we will use some figures. Of course, a lot of maven planks and test libraries that you never
heard hearing talks. They are removed from its bombs. They are removed from everywhere,
but without these things, we are not able to publish fast. So right now, we can publish in half an hour.
I had already scented it for another conference, so at the conference, I showed it how it
happens here. We are in the middle of solving some problems, so I cannot make a release. The second
problem was, as I said, the test. So we had only a sequential test, unit test, which were
unit 3 to unit 5 and there was a lot of flakey tests. So that was the problem. So you make
unit to make a wrist fast and you have a test failing and you need to analyze if it's
the flakey test or something Korea. So nowadays we switch to parallel tests.
We added dynamic tests, which was needed for us to have the
open SSF best practices. Mark, there are still flakey tests that's something that's
if you want to help please help us. It's always good. We had a present from STF that paid
another company to convert everything to unit 5. So now we are unit 5, there are a little bit more flakey
tests that we are working on that, but the build takes half an hour with the website.
It was a little bit more hard. A lot of discussions on how to handle features because
one of the important things is that, okay, if you are a maintainer, you are allowed to
patch your feature, right? Well, this is a big question. So no, we felt and we even
in look for J3, we did a clean up. We modularized many features. So if they are affected by vulnerability
and you don't have on your stack, it's not your problem. We removed a lot of features
that nobody was using. Even if a PMC member contributed to the feature, maybe he was not using it
right now. And now we have a strategy to accept in feature. Look for the modularized, you can publish
your own Git repo, publish the feature in a jar. If it becomes popular and you need an
Opus on Stuart, you can bring it back to your project. Okay. Then so that was what helped us improve
our internal processes. There is also the problem how to say to people that we have a vulnerability.
So right now, as bonds are very, very popular, but honestly for us, we are a library. So we don't
decide what people are putting in their applications for vulnerability handling as bonds
for our project that it doesn't do anything. What it could do and what we implemented
is you can link from an as-bomb other important stuff. So for example, we link a video
which is the list of our vulnerabilities described as we would have done it. So it doesn't come
from the NVD. It's right from a humanitarian video. That's what we're able to do up until now.
In the near future, in this month I'm working on some tools that will use these as bonds to
download the videos, to check the compatibility between the as-bomb of the project and what is
published in the dependencies and so on. But of course, the ultimate solution is what you will hear
from all that tomorrow in this room, I think, T transparency exchange API.
As soon as T is ready, we will try to publish videos, which versions are supported, which are
not through this interface. Okay, and the last thing we did is there are some things that we
cannot fix, but because depending on how users use a project, they can do it in an unsafe way.
One of the problems is, look for J allows you to have placeholders for parameters or string
concatenation. You can choose if you want string concatenation or pass the parameters separately.
If you do both together, you can have some my logging injection because if users contains the
parameter, then the parameter will be printed here and not there. There's nothing we can do it.
So the documentation is also important to tell users how to save the user for J. Okay, so now I have
a new proposed timeline when lock 5 shell will start. So on the 0, we always start to answer
to reports in 24 hours. We request the CV number as soon as we have the CV number. We request
a private G3 point, we didn't know about the possibility. So the patches will be full git, but you will
not see them. And the problem with the voting procedure, we will start not a vote, but a consensus
to shorten the vote. So all the PMC members, which are all over the world, have 72 hours to agree
to make a 24 hours vote, for example, 8 hours to decide, okay, we will release at this time
a piece vote. Then we will not try to keep the feature at all costs. So the first thing we do
is to Java lines of code removing the feature, if nobody comes up with and better solution,
that's what will be in the patch. Because in the end, after lock 4 shell, strongly limited in
G&GI was the final solution. So on day 5, maybe 6, we can already prepare the release,
we don't need a separate date for that because it's half an hour. During the vote, if it's 24 hours,
if it's 72 hours, we ask the project that depends on lock 4G to help us, very data thinking,
it's and running their tests. But even if it's short, it should be possible to automate the
process to seeing that there is a vote, you know, for J testing it, so you can say that. So you can
already say my application works with the new version and not for J can say, okay, there were
no bugs introduced. And on day 7, we have the announcement. Okay, so we reduce it a lot.
All right, and smooth up. Any questions?
Thanks for the presentation, we have three minutes for the announcement.
Any questions? We have unit tests, but the open SSF, the question was about the dynamic tests that
we added. So we had unit tests, we didn't have fuzzing tests, we introduced fuzzing
during the SDF contract. So that's the best practice that opens SSF requires if you want
the best practices, Mark.
Do you have any solution that customers don't know of the older version of the
universe? What one of the main problems of Locker J that was? Yeah, I thought a lot about that,
but yeah, maybe I didn't say it, but if you looked at the graph, many customers in the first
month, there were three reasons and there were many people that downloaded all three. So there
is a part of the industry that's conscious, security conscious. The rest, I think the solution is
CRA and fines, but I don't know. I thought a lot about that, or maybe we could support something
for 10 years, but yes, we support this version for 10 years, but as soon as we have a patch release,
you need to upgrade. Thanks for the very interesting talker. I have a question,
because I mean, the Locker J case was very popular. Yeah, many lessons learned that you
think they shared with us. I don't know where, like, if this lesson there were propagated
to other projects in the world foundation, is it? Well, actually, Locker J was one of the first
that's automated the build. I don't know if it's a direct, I direct cause effect, but now we
have this project, a patch trusted releases, so we have one person that works full time on this.
Well, we try to spread the world for conferences.