You can reach me on my email on MasterLun and I'm here on the floor but what I'm present is a
an effort of a multi-year and several people. So for example, I'm 21 and you do
a big Stefano team with you and so on. So who knows Geeks? Who knows software
heritage? Okay, so for the people who don't know software heritage, there is a
talk in open-day room a couple of years ago so you can go watch this and you
will have all the details about software heritage and you also have in the open
we share their room representation couple of years ago about Geeks and you will have more
details about that. So today for sure I present software and Geeks. So what's the
problem? The problem is software is dual. As human we with source code but the machine
run binary. So we need something that transforms the source code to binary. So
okay we have the yellow world program so everybody can read it and and so on and this program
is transformed by something like a compiler or interpreter or whatever to be
binary. And the machine runs a binary but you cannot read the binary as your human.
You cannot know what is open. I mean all you are kind of alien. In general we read
we audit and we verify the source code. So we need two things. We need one thing to
keep care about the source code and this is software heritage and we need something that
care about the transformation and this is Geeks and if we have the both the both
Geeks and software heritage. So we take care about the source code and about the
transformation. We can have a long term transparency to be able to know exactly
what happens from the source code to the binary. So this is the main message.
So a typical scenario if we are in academic is you have Alice for example
she publish a results in 2022. The source code is publically available and so on and
the software is identified as zero dot nine okay nice couple of years later
for example in terms 2025 break need to to give a look to this paper and the
publication and so on. And now it's really easy to install the version 1.2
but it's really more complicated to install version 0.9. And even if you
success to install this old version 0.9 the results are not the same.
The results are different and the question is why and the answer is because
break doesn't control all the source of variation of the computational
environment and to control the source of variation you need to
insert this question what is the source code what are the tools that you need to
build this source code what are the tools you need to run the binaries that you build
and this answer for each tool of the dependency. So the question I'm not new
but answering this question means control the source of variation. So other
we insert this question because it's not new so we have all the package manager
the IPT and so on and so on and this is where gigs come I don't know sorry
what does the package manager the package manager in fact the control
graph this graph all nodes are our source code and all nodes all nodes describe
build time tools how you compile it and so on and also you have the dependency
so for example the first package I think you don't ever to read it but it's our
money and this package depends on other package and so on so that's why you have
the graph and in fact if you want to install if at least say I need the
package our money in fact the graph is almost 500 nodes so it's really big you
cannot describe that by end and the package manager does a job for you so that's where
gigs come in gigs manage this graph so it's a software developed since two
more than ten years now so the communities start to be to be nice and you can
use it as a package manager or complete a system and it's based on the same
principle as as nix so it's a functional software deployment model and the main
topic I mean the main point is it's transparent and you can verify all the graph so
if really a perfect tool for software deployment in reproducible research workflow
especially in in open with search context so okay if we are back to the to the
example at least say okay I use these tools at this version as a version zero
dot nine and we need to fix a graph and this graph if we fix this graph we can
we deploy it later but to fix this graph saying zero dot nine is not in us
because where do you describe the dependency is the version of the dependency
the compilation option of the dependency and so on and so on and in gigs
in fact we describe a state so one state for example eb three four f f one
this state this revision fix all the collection of the package all the option
everything and if you know this state this revision you can really
deploy the exact same graph when Alice says I use gigs at this revision
black no everything and can deploy the exact same environment with the same
dependency the same option everything is the same so we have a nice command line
where where you say gigs time machine you specify the revision and you jump
in this revision to install the tool that you want so if Alice says I use these tools
at this specific revision black later or whenever when whenever where so on another
machine another point in time black is able to run the exact same software
environment with this command line so this is really cool but there is assumptions
there is two assumptions the first one which is the second in the slide is all the
bills must be deterministic so this is not always the case and thanks to the
flexible bills effort we have been the progress are incredible so we are
it's almost we are going doing really good job in this area and the other point
is that all the software need to be publicly available so it means that we need
almost 500 source code to be able to deploy later so this is the issue in fact
so if for example now more or less you give a look to the source code in 2022
so you say okay now I give a look to the the location of the source code
that I register that I know that the source code was on this server two or three
of three years ago and if you do that you see that almost 4% are meeting
so in in 2000 in 2022 there is source code at some place and three years later
this source code is up here so this is a link rod and so okay is an issue
because if if in this three dot 6% we have something that is missing from the original
URL you cannot build it so you are done but it's worse than that because this package
can have dependency I mean package can depend on these packages so for example
OpenGDK is a package that have a lot of package that depend on OpenGDK so if you lose
OpenGDK you don't lose only OpenGDK you lose 184 dependency which depend on OpenGDK
so it's more than the three dot the three dot 6% and if you go try this order order
and order it becomes worse and worse so what we are doing currently in science
so for example I mean 2000 in 19 it's not so all but you see you have almost 10% that it just lost
so science currently is building on sand and that's why we need software heritage
because we have this problem the link rod project are created postdoc or is a
IP and so on and and the source code are just disappearing from internet or was you have
kind of big company that just closed the service for example Gitu use Google code
bit bucket and so on so when you have a website that is up here where you go you go to
internet archive for example but where you go if you have a link repository that is up here
what do you go you go to software heritage so software heritage collect preserve
and share all the publicly available source code and currently to my knowledge is the
the largest publicly available archive of source code so they they and just a lot a lot
a lot of stuff cool so you ask me cool but you just said to be to fail so why
software heritage cannot fail I mean it's the same story why and I don't know maybe they
me that but the guarantee is that it's it's a it's a good project and the strong
project is because it's international and non-profit so for example you have UNESCO
you have a lot of company that that shares the vision that put money that sponsor and
so it's widely supported cool so what is good with software heritage with software heritage
you have this identifier the identifier you don't identify for example with the label
but we identify with the content so it's content address it like a digest a checksum
and when you know that you know exactly how to refer to the object that you want so for example
this is a file but you also can use the same system for snapshot really is revisions
directory content and so on and so on so here you have the address where you know exactly
the the the revisions the the file how he was when he was archive and this is an
image and standard so you have a specification and so on you can go in S
W ID dot org and you have information so okay the point is we have a content
address server so this is the first I mean this is a point with software heritage we have a long
term content address server so now back to geeks weeks for for the origin so to fetch
a source code we have a different a table geet mercry and so on and so on and so on and
we have a location and we ensure that the content is exactly what we expect to have
the the quick-up to the f-e-h so source code in geeks is also essentially content
addressed so if if the the the the the location is no longer available or there is
something wrong in fact what what what what you can do so black can work around
if a copy is available elsewhere so for example elsewhere if it is a new URL you
can use geeks download and and or or you can use a content address server so the geeks
project or the next project are these server for these files or also the software
uh heritage initiative so this is the the architecture so in geeks we we we we save
software in software heritage in two ways the first one is we list all the the
source code we have in geeks all the packages we have in geeks and we feed software
is that and a user can also say okay I have this package on my own and I want to
save it to to software heritage so you send a request with the this command geeks
length dash c occurred okay and when the upstream is up here in fact the user
sent a query to software heritage and you have the the the the content back and
you use with the vault and the user of the data can build and so on so what the
point with this these archive uh uh uh part the the the part is in fact the
tarball is a really complicated problem because for example if in geeks for example
with the software harmony we have this arch 1-D if 1-D f7 but if you
decompress and recompress with one level of compression or you decompress and
recompress with another level of compression we have different arch so this is
work all the content address system so in fact you need a way to to extract
some information of the level of compression and and so on and so other other
is a well known it's in case you need a standard problem you know we have you
know uh uh four 14 standard and asian we have 15 so this is what fix this archive
this archive creates this bridge between all the different standard and extract the
meta information of that that you need to recreate later the exact same tower
so in fact what what what what what we do we have a tower the tower we disassembled
the tower we extract the meta data for example the compression level the the
times 10 of the five and so on and this is going to the desired archive database
and the content itself the pure content is archived in in software
heritage then when we need it we ask the to software heritage give me the
content we ask to this archive give me the meta data and we will build the exact same
compressed tower ball okay so this is what I'm just said so does it work yes
more or less or it's improving so over the year you see that we are improving
the situation so the unetermined is we don't know so maybe it's it's stored in
and we are able to to to to to use it or maybe not it just we are not able to
determine if it works or not yet so we are working to fix that but the
the interesting part is everything is is decreasing we are more or less improving
the situation so the the ID is to say okay I have a source code from 2019 and I I
I I want to spell is the exact same version and with this command line gives
time machine dash dash commit blah blah blah you can recreate the exact same
version of the of the computational environment whatever whatever what happens to
GitHub GitHub whatever blah blah blah and this I mean we are improving the
situation and we are fixing all mean yeah what I said but what is not working
currently so conclusion conclusion sit and reference source code using software
we take identifier and yes use geeks and yeah thank you
questions yeah questions
yeah
So, so the, we put the question. The question is, when the, we want to
rupture do something the other way as an impact on the rapid reproducibility
artifact. So, yes. And, the, I mean, the answer is, I don't know. So, my
opinion is, do we want to ru, to rebuild everything? This is the first
question. And, do we want to pay the cost with, for example, energy, see a carbon
emission and so on? Do we want to rebuild everything? So, the first question. And, if
we don't want to rebuild everything, what could we have to, to be able to
re-re-re-ify that everything is correct. In fact. And, this is where gigs, I mean,
provide really good answer, because you can trace all, all the dependency,
everything and check everything. So, my answer is, for, for the binary reproducibility, I
don't know. And, this is something to do. And, on others, on other end, I think that we also
need to improve the verification part by, be able to audit without, rebuild everything.
Does the answer, or the question? Yeah. Yeah, because it's a huge amount of data,
where it is stored and, as you encountered, which are next on libraries, because it's a
storage problem. So, so, software heritage, the main storage is in France.
There is a mirror in Italy, and probably some other mirror will pop-ups soon all
around the world. So, this is for the, for the answer. And, the other one is, what about the
relationship with the National Library, also is for? So, software heritage is, is, is, is, is there
the agreement with UNESCO? For example, so they try to take care about all this,
I mean, national library and so on, archivist.
All these people doing this kind of job to try to keep care
about the long-term artefacts that human knowledge,
the human knowledge, that's right,
to be in touch with everybody.
So I don't know exactly for these countries,
these countries as a country, but for example,
in France, there is connections.
Yeah, Nikola?
Just one comment.
Software heritage has a blue tomorrow and yes.
Or if you have more questions about software heritage,
come and see us tomorrow.
Excellent.
What's the goal of this question?
Yeah.
Thanks for your talk.
I have a question.
If for some of these packaging weeks, as a package,
if you make the link between the original source of the package,
you'll be at the bottom of each item.
Is it made of a matching package or do you have to specify this item
if there's a field of packaging?
No.
So.
Ah, yeah.
So how do we connect the gift package definition
with the location in the gift package and software heritage?
So this is what I keep a bit because I always a bit late,
but in geeks, you have the checksum to say,
okay, you have the checksum.
And in fact, we send this checksum to software heritage
as a request and software heritage,
say, okay, we know this checksum and this checksum
of拿 and so on could match the hour identifier.
There is another question.
I was wondering because, as I understood,
the sub name, that a bit special.
You're all the way to the software heritage?
Yeah.
Good. Good question.
So the question is, is,
okay, your story is really nice,
but what about these archives in the picture?
Is it in software heritage or not and so on?
So the answer is not yet,
but there is a work in progress to try to have mechanism
to be able to rescue this database.
Yeah.
This is a work in progress.
And today's the goal we discuss with the software heritage team
to see what can be done in this direction.
Next, thank you.
Thank you.