Okay, hello everyone, my name is Hugo, and I will finish the
San Tox, so my colleagues talk about how we started date on this, how we started
date on tapes, but actually also how we distribute the data, because
Seren is not the only place where we started data. And this presentation is about
a project is an open-source project called Rousseau, it's a Python
Britain software, and before entering into this project in detail,
I would like to say a little bit the scene. This is probably the most beautiful
slide that you're going to see in my whole presentation. But
Serenna, as my colleague mentioned, we have all these different open-source
systems that we develop, or we contribute upstream, like for example
theft. So we have focused in last on the last one, which is Rousseau
on this presentation. This is a plot that I present a few weeks ago, just to
show you a little bit the scale of the data we have to manage. So in the last two years,
we actually have to start more data than in the last seven
years of the lab. And as you can see, this curve is getting more aggressive, and in the next five
years, it's going to be more and more aggressive. Which means that on a constant
budget, we have to be very creative about how to tame that data delusion.
Okay, so who knows about WCG? Okay, one person, two, three person
for person. Okay, so it's not very known because it's okay, it was developed
to a hundred-delete program, but actually this is the largest
computing grid in the world. Serenna sits in the middle,
so this is like an onion. We're all the data produced by the LHC machines.
They are stored at the cell, which is what we call the TR0.
And then we have to distribute the data around 170 different data centers across 42 countries.
And this infrastructure is used by more than 42,000 people to actually do analysis of this data.
We have physicists around the world, not only at cell.
So how would we distribute this data? How we can access the data that is originally stored
at Seren from other places. So if you run a storage system,
sometimes in your career, you will get a user asking you, okay,
my analysis at Seren, it's working very fast. I'm very close to the data.
Can you actually make your storage system faster? Because I'm connecting
from the US, and the performance is not great. I got this request very often,
which means in our storage group, this is what actually means the user.
Okay, how can you optimize a local POSIX workflow? The physicist will write some in script
that is using basically some system calls on a link, which is from the US,
to Geneva, or the transatlantic fibers, over why they are network,
you see a few protocols. And the question, okay, how can we actually optimize that part?
And the answer is like, we can make some tricks, but we are bound by the speed of light,
and we are not really going over it. So inside Seren, and this is just an example,
or real example, actually of the latency is not we have. So inside the data center,
we have 0.25 milliseconds between two machines. So if you have another five system,
this latency, let's say for the analysis use cases that we have,
is barely noticeable. But once you go to the US,
or to other continents, where you have much bigger latency,
this is for example a thing I did this morning. It's around 125 milliseconds.
So this means for every operation that you have to do over the network,
the operation is 400 times slower, which is not nice.
So what do we do actually? How can we optimize this?
The answer is like, we cannot do it. So we have actually to copy the data from one place to another.
So the data is analyzed locally on the remote system.
This is an activity example. I did this morning, okay.
I just straight a stupid program to come some lines. And just, no,
this is stupid program. It has around 125.
It's that system calls and 400 reads. To just count how many times
the word your appears in the text, you actually was a Russian text.
So there is not many, zero. Just to show you that, you know,
the physical analysis and much more complicated than that. So there are a lot of more
system calls over the local kernel mount or the field mount.
And when you have this white area network links and the performance
rapidly decreases. So yeah, the answer is like,
when you have to really have local IO performance,
you actually need to ship the data to some place and then you do local operations on
the metal on top of a local file system or you have a fast network
interconnected with inside your data center.
So this means that you actually need to have a way. You know, a mechanism
to move data from server to over this 170 different data centers
that are distributed in the globe. So how we do it?
You know, this is the typical, when you have a student coming to my office
when I arrive for the first day of job and how can you transfer this data?
Okay, let's see please some data.
Okay, you have a few megabytes, not a problem. When you have to transfer like
billions of files containing around one petabyte of data,
you can still leave it at the right, but there are many things that you will go
around, right? And this is a few of the things that we have to deal on a daily
ways. Okay, we have network failures. Sometimes the network is not reliable.
The server has some access. So at the end, you may basically fail
some transfers. Sometimes we have a lot of different countries that
we have to transfer the data. Sometimes you have high-speed restrictions between
so, for example, you may not transfer data to a Chinese computing center
from the US, so you have to pass through Europe, etc.
So there are a lot of logic involved to how to basically see the data from one place to another.
And also the typical question, you know, how we transfer. Are we confident
that we transfer all the data and that we have not corrupted any of this data, right?
So for the end data, we cannot just use STP or whatever tool.
We have to heavily get robust and system that have some estate,
but understand the data that we are transferring. And to do some analysis
of these data to see if the data has been corrupted.
So this is what I'm going to talk about. This project is called Rousseau,
and actually solves many of these problems. So Rousseau is an open source project
that started in the Atlas experiment at 7, so it has over 10 years of
operational experience. And the Atlas experiment decided to open up for the community
a few years ago, and now it's used in many scientific communities that I will
mention later. So Rousseau's free open source is from GitHub, has
a permissive license, Apache V2, and it provides actually a scientific data
management platform. So you have data that you need to distribute to many of
the different sites. You register this data in Rousseau. You don't need to run
a Rousseau null with your computer center. This is a nice part about Rousseau.
It's a middleware that runs in some place, and then it just has a catalogue of the data,
where the data is sitting, has a pointer to all these data centers, which means that
location are where any city originals. Because imagine you have 170
data centers in different countries, you cannot just say, okay, you all use
these same technologies. So we, you know, the life is great. So every country
will have their own technologies that they would like to use, etc.
So you need to have something that actually under this
heterogeneous technologies. So what are the
functionalities that Rousseau provides? I will not enter into all of them,
but the view of them is that it's horizontal, it's scalable.
The name space is a SQL database that you can buy. You can use publish,
SQLite, or Oracle. It transfers between different
facilities. You can be these facilities types or HPC cloud
systems as well. And actually provides different
authentication mechanism. So the user can authenticate with
open ID tokens, or with X iPhone 9, or like TLS client certificates.
This is a high level overview of the architecture. So in the
top, we have the clients. We have CLI clients, Python SDKs,
REST API, and web interface. So you can actually use
these interfaces to access the system. And then there are different
components. The most important ones are the demos.
Demons are like independent processes, these are Python
features, multi-threaded, that what they do is like
they connect to the database. They obtain some job, they do some processing,
and then they store it back in the database, so other demos
can work on that. And at the end, we have different
transfer protocols, like S3, or like pure HTTP with web
dub, and some other ones that probably do
know about, like logos, which is more orientated to
scientific data. Yeah, and that's basically it.
Yeah, but this is the most important thing about RUSIO, is that
is a system that is declarative. You don't tell RUSIO how
you want to transfer the data. You basically express how
you want to see the data in different data centers with
the declarative language. So for example, you can say
I want three copies of this data in three continents, and at
least one on the tape system. You can say with a rule, I want one
copy of this file anywhere, but on a fast disk. And you can
create a center, about, which let's say, what are the capabilities of each
data center. This is what we call the replication rules.
So when there is a need to copy the data to some place, just create a
replication rule. This one simple command in the CLI, and then there are
thousands, if not millions of transfers that are created on the back of the
system, to transfer this data, and to verify that the data is actually
correct, and not corrupt, etc. All you can do it automatically. This is
one of the reasons you can say all the data that comes from this
computer center, or has this metadata, because you can attach
metadata to every file or catalog. You can create
transfers. So for example, you have data workflow that
you are writing into the system. You can trigger
replication rule automatically through the
subscription, so then the data is copied to other systems.
Yeah, this is just the concept in RUSIO. It's what we call
the RUSIO name space. The RUSIO name space is consisting of files
and data sets and containers, containers, we will disappear,
by quality meter, I can tell you more about this. But yeah,
files and data sets, this is our digital abstraction about a file, another
directory, and a normal file system. And this is how we
explain basically all the data that is cataloging RUSIO, so we have a scope,
and then we have a name. And this is unique. You cannot change it,
but once you basically mint it, it's like, yeah,
you cannot replace it. Let's say, it's persistent.
What I mentioned before also is that in RUSIO you can explain
different queries with metadata, so when you register the data, you can
actually add likeization payload, describing which metadata
you want, this is arbitrary, and then you can create RUSIO based on that metadata.
So it's quite flexible on that regard, and this interface is actually even
enhanced, being enhanced right now with different projects.
On the operation side, what is in the day to day of operating this
service? The objective actually to minimize the
operations of human people, because at the end, this is one of the
most of the costly factors in every organization, which is to optimize
the human course, and not having a person that has to do it
but the system that does it automatically for you, is really great.
And moreover, it also means that when you have a system like that,
you have to understand all the protocols, and you know, if
Amazon decides to change the SDK and then breaks in
compatibility with, you know, on premise,
is three clusters, but you have to follow up this. But in this case,
it's the RUSIO server that will handle this
upgrades for you. And the last thing is like,
you know, how the transfers, they actually move from one place to another,
and this is a great topic. But actually, RUSIO does not
proxy the data from one place to another. We actually
expanded this community for WCC, created an
expansion of web-dub, is what we call the third-party
copy mechanism, and this is a peer-to-peer copy.
If you have a client, this is an HTTP
copy command to server A, and there is a
header, which is called source, about five
data. And then the server A, it will use an HTTP
guide to get the data, while the client gets some
performance metrics out of the server A. And this is how we
work on, because you don't want to have one single
system, or even if it is distributed, and you can
originally scale to proxy all the data through one single
planes for 170 computer centers. So this is peer-to-peer,
and this is just simply like extension of web-dub.
And that's it. Just to mention that this project is
used in many physics, scientific organizations, and
some of our physics ones. And we are looking for what
actually, if there is other interest in other
communities, so you are looking, or you have the same
needs for transferring big data sets across different
countries, or different institutions. This could be
an ice project to contribute to. Yeah, so that's
it from my side. Thank you.
I'm fast. Any questions?
Yeah? You said, once you created, or you
cannot change the network? Yeah. So what happens if you
do a typo? Yeah. So basically, the question is,
what happens if we change an identifier? So in
Russia, there is a content that when you create an
identifier, you keep it open, until you are very sure
that this is the state that you want, and then you
have like a double commit phase. You can say, okay,
I close now this DID, and then if you
broadly mistype it, then what you do, actually,
the metrothinatlas is a good answer.
The metroth was with us in the same team.
So you can grab him as you want a specific
like, you know, from operations perspective, if it
happens in real life, what are the tricks that you
can do? Okay. Any other questions? Yeah?
Yeah. Yeah. So the question is, can we add metadata?
So the files are set several? Yes, absolutely.
You can use a key value pairs when you subscribe
to the data, and you can also put like a JSON file,
and then basically you can issue queries on this metadata.
Okay. One more question?
Yeah. So basically, when you tell the truth that
sorry, the question is, if you want to download the data,
if you want to download the data, if you want to download the data,
if you want to download the data, if it proxy through
Rousseau, or it's basically obtained the
rally from the storage systems, and the answer is,
you can get it directly from the storage systems.
So as Rousseau, where did the data sit in, and then
Rousseau will do the magic to find the best copy
available to you. Actually, it has some logic,
using basically JYP to find the closest computer
center close to you, so it gives you the closest one,
and then you download the rally from the site.
And you don't have to deal with authentication.
The rally, this is handled behind your back.
Yeah.
You mentioned that the data placement are based on Rousseau,
or the Rousseau, right?
Have these rules that you find by the creation of the data
about data consumers, and how they,
like the dynamic report properly by a business in the US,
and try to access some data that is not in Rousseau,
whether the data can do a bit change or,
like, some part of the data is in Rousseau,
whether it's not for the next instance.
Yeah. So the question is,
Hohan does let's say the creation of the Rousseau,
and who defines the Rousseau.
So depends on the community.
This will be my generic answer, because every community has
different needs.
But for example, what I see on a daily basis,
is that there are some systems and administrators
that runs the Rousseau, let's say, instance of this project,
and if the users they get some quota,
so they say they can transfer up to 50 terabytes across,
let's say, all the computer centers,
but if the request is more than 50 terabytes,
it's an approval from someone else.
And Rousseau has built in this approval workflow.
So you can define the threshold, so the rule can be approved.
Okay, so that's it, thanks.