I'm just like, I had no idea what's really being doing something like a keynote talk.
I'm feeling like I like to talk about clarity, licensing, and metadata for mixed packages
and way to problem.
I maintain a bunch of tools, some of you may know about them in the space, something called
scan code.
I double those so a bit in standards, like something called package URL, which is used
to identify packages in S-bombs, and standards, we need more of them, and the co-founder of
SPDX, and I'm also a core contributor to CycleDX.
So I'll try to avoid making the room too late starting late, so I'll try to finish
early.
The problem is that mixed package has metadata in particular licenses, is a bit of a mess.
It's not your fault, it's not all fault entirely, a bit, but it's also because it's hard,
because upstream is also sometimes pretty brain damaged.
If you look at, say, something like, look at this license, this page, which, and you
squint, it says MIT, but it says, you know, permission is not granted, right?
If there's some junkity, but I have a whole collection of these, I call these fubo licenses,
and it's just a small slice of this.
It's easy for us to be full by that, because, you know, it says MIT, and in some case even
GitHub says MIT, and it looks legit.
So that's the problem, so it's not entirely on our fault, it's a problem of the tools,
it's a problem of upstream or so, which is sometimes pretty brain damaged.
In the end, it's useful, each time you have a license issue, you're putting a bit of
a friction in using packages, and license of somewhat important, open source exists because
of packages being under an open source license, right?
You remove the open source license, there's no open source anymore.
So, so useful for some things like emerging regulations like CRA, and, and really,
any kind of responsible reuse, whether it's in the corporates, in an organization, in
an open source project, you want to know about the license.
So what can we do?
Does it take a fix the problem?
So we've started a small project called Nix Clarity, which is supported by a program funded
by European Union called Fegiversity, and the goal is to help fix the mess.
Hopefully also help upstream, because the great thing about Nix is that a lot of the code
is pristine upstream, and ideally, we don't want to fix it for Nix.
We want to fix it for everyone, so in a thousand years from now, open source license clarity
will no longer be a problem.
So the plan is, you know, if we can use package URL to help standardize the case where
we have Vendor code, which may not have been externalized, could be paste, automate the detection
with open source tools, eventually deploy at or from Nix, that's not up to me to decide,
but ideally, like whenever you publish a package on Nix packages, then you get feedback.
If the foundation of community wants to block on funky, weird, missing license, proprietary
license, more power to you as a group, that's not for me to decide.
So, quick work about Rails, I have any of you heard about package URLs?
Good, so I need to talk a bit about it.
It's a small strip it's standard to identify your package in a Nix bomb or elsewhere.
It's useful and you need to know about it because it's been merged into the CV schema for
vulnerabilities in data October, eventually you can go from a scan of a codebase to straight
look up in the vulnerability database without much friction.
So that's that's really useful for that.
Somebody has to tell me if I'm going late because I can go on that for ours, it's been
recently on the 6th of December standardize as an ECMAS standard, it's only choice to
be an ISO standard, which is really interesting for a small string like that.
But you know, it's important for companies to have also this standard to ensure these are
not moving and usable.
We are working by the way specifically supporting package URLs for Nix packages, which
has specific challenges because Nix and because we have their revisions and a lot of
hack has to deal with.
It's used a bit everywhere, most if not all the tools that scan for origin, license and
vulnerabilities do use package URL, so the standardization can basically after adoption.
Database, CVs, using that GitHub, GitHub, all these companies, most open source foundations.
So it's hopefully useful, it's not perfect, I like to say it's less bad than other approaches
that have existed before, so it's just a small set for it.
So now what are we going to do for Nix?
I don't if I vendor it copied code is one thing, so we don't want to create and validate
all the licenses.
You may not know how Nix licenses are actually tracked in Nix itself, but some saying
you search like that, I had a page up and we said that, there's a big, oh, there you go,
it's at the top.
You have a big Nix file that at least each and every possible license that could be
used in the package.
It doesn't scale because you can have something like the Linux kernel which may not be
strictly available as a new Nix package, but you may have a package that has a combination
of license, license, exceptions, these kind of things.
In the current mode, you would have to expand this file forever to list all the possible
combinations.
So that's one problem.
So we want to validate all the license, but there's probably going to be work to do
specifically to support license expressions in Nix language.
And I don't know how to do that, I need to write, how to write Python, I can be dangerous
in C and a bit of C++, I don't know Nix to do something there.
The problem also at scale is it's very large we're talking about tens and tens of
terabytes of code.
We want to clarify the license, so using scan code and match code to do things, scan code
that takes licenses.
It's basically a stupid dump, deep, between many license texts and license mentions
and the code, except it's not stupid when you need to do that billions and billions of
times, so there's a few tricks to make it fast.
It always requires new licenses, especially with AI, there's a lot of very bad innovation
going on, everybody wants to involve great new licenses, which are almost upon source, but
not exactly.
You've seen the case where you just had not every week we find new license, typically from
AI-related projects, which are really problematic.
So being able to detect the license, one thing, the other thing is being able to detect
vendor it's copied code, that's where match code comes in, it's basically database of
harshes.
Again, very simple, in essence, can be a bit tricky if you're trying to find efficiently
matches to code that was copied and it's been modified.
There's a few tricks for that also, and the problem here, we're talking about generative
AI, I have a tool and I can prove that
and scientifically, about 20% of the time when you can easily point LNs to actually
speed, very bad in copies of the source code that was used to trend them.
So AI-company used collectively or source code to trend them model.
This is eventually an index that can speed and memorize everything, whenever you have
that, it's great, copyright infringement machine, it's fast and of course there's no
attribution, no notice that comes with it.
We're going to have a lot of bugs to fix, again, nix is big, nix packages big, we have
some not-secret, top and source, but some machine learning tools to spot incorrect license
detection, so to fix the fix, help fix the bugs, and we're going to need also a way
to avoid being a drain on package maintenance at nix and eventually upstream.
I don't know any solution that due to it one at a time with humans in the loop, I hate
bugs, it's not the way, which means we cannot do the fix of problems upstream without
involving massive the community.
So we talked about the license expression for nix, right now the solution of license
that nix in a scale, if you really want to have a credit license at the scale of the
current nix package table, it's going to have to be 100 times bigger, which is going to
be a challenge even for poor request merging and this kind of thing.
It's SPDX not SPX sorry, we need a way to manage and deal with SPDX expressions, which
are combinations, where you just keep the individual licenses and you can say, oh this is
GPL and BSD, as opposed to say, here's the GSD, here's the GPL, today the approach in nix is
to store GPL, BSD, GPL and BSD, GPL or BSD, GPL and MIT and all this combination, again, cannot
scale.
I have Python code for that, we need help to bring that to fruition to nix.
We need also to find a way when you have package competing to packages, to make sure we
are correct origins, it's going a bit beyond license, if, like you have say, in a rust
crate, a vendor copier of a PNSSL, and you only know about the top-level rust package,
and you forgot that you have vulnerable copier of a PNSSL, that's a problem.
This may go undetected when you actually build nix packages, because you may not
devander systematically the PNSSL copier, the PNSSL copier, the PNSSL copier may have
been patched for what you know, and that's a problem, so it's really interesting and
important, not only for license, but also for security, eventually it's also an issue
for upstream, right?
We want to do that on a slice as big as possible of the nix packages, and brings
and repeat, and again, the goal is not for us to own that, but to empowers the package
maintenance, to have this information, and, on the nix community, and the nix, so
it's conditioned to own this stuff, if possible.
And in the end, we use that for our system.
We have running project with a star in the space, with the maintenance of logforj.
You cannot make a presentation on security issues, and open source without talking about
logforj and logforcial, and what we're doing is working together to fix the problem at
naven, to find hidden copies of logforj.
Doing something on rest also, debyan is in need of love, and suffers a lot of these
licensing issues, when a package is uploaded in a...
They lose the, they lose track, and the metadata usually drift, so if we can do good for nix
for debyan, that would be awesome.
That's it.
We're, as I said, public benefit on profit, foundation-based Brussels, recently formed,
we've been blessed to receive support from the German government, large US corporations,
and a lot from the European Union through the NGI program in particular with NLNet.
We, we leave up and survive as a charity, and I just want to say that we're trying to help,
so if you have questions, and that's pretty much it.
Any question?
Yes, go ahead.
So, the question is, how do you, how do you handle the less common licenses?
So, the detection scan code, as I said, is stupid.
You, you take all the variations, non-variations of MIT, which are bona fide MIT,
the original correct license text, that's agreed upon, because there's no real
original version for MIT, and all the known bad variations, and you're trying to match that exactly,
using a D, when I said using a D, eventually we use a modified version of D,
which works on big vectors, but it doesn't matter. If there's variations like one more difference,
you'll say, there's not exact match. As a matter of fact, we're also indexed, these we have
fubo license, so we detect them eventually as proprietary licenses, but it's, it's really a string matching
at the world level, and the goal is to have as big a database of example of licenses,
about at the moment, about 40,000, and the more we have of this example of bad and good licenses,
the faster we are detecting, because we build automatons, and we use an algorithm for search
called alcharsic, which is used typically for detecting viruses. And in the most, we were very
efficient fast, exact detection, exacting the exact sequence of words, ignoring formatting,
and all that kind of stuff. So it's really string matching on a large scale. The one thing to understand
is that say, given an index like Google, queries, ignoring the new AI mode, was limited to 32
words in a query, index is petabytes. In our case, a codebase or large codebase could be a
copolygabytes, but the codebase is the query. The index is 10, 20, 30 megabytes, the codebase is
the query. It's interesting and less common problems. How do you deal with gigabytes size queries
on the small index? I'm going to see, hopefully people can hear me with this microphone,
I apologize. No, that's okay, that's what the sign to tell me, it's over.
Philip, I'm going to thank you very much. We can take one more question.
I apologize for not introducing you ahead of the talk, we're still figuring out the Microsoft
situation a bit. So we're going to see if the mic can ask you questions that way,
we'll work, so it's because don't have to repeat the questions. Are there any further questions?
Please raise your hand if you have a question.
Somebody can think of a question, right? Can be interesting. Okay, so go ahead then.
No, I didn't hear the question, sorry. Sorry, I did lose my phone coming. Can you hear me?
Is it working? Yes. I'll shout, okay. I was wondering, is there one specific place for working
on this project? Because there's a lot of links on the board. Yes, that's a good point.
So there's one specific place, which is a board of issues we track on a robot code organization.
I'll make sure I'll put the link on the first-hand talk page. Thank you for that. I forgot about it.
All right, thank you very much. Thank you.
Thank you.