Here we give them a big round of applause and a round of applause and a round of applause and a round of applause.
All right, I think it's working.
All right, so I'm here to talk about the use storage.
So before we talk about a certain, I mean, before we talk about the project, let's just talk about a certain.
So it is one of the world's largest physics lab. We largely do particle collisions here.
And I forgot about it. So let me do everything again.
Hi, hi everyone. So I'm here to talk about from particle collisions to physics results.
It's about use project search, which is a large storage project we use it's own.
So before we begin talking about a certain, one of the world's largest physics labs.
We largely do studies on the fundamental nature of matter. It's also a large international collaboration.
It's like a composed of 24 member states and 10 associate members and a lot of other states around the world.
Our mission is largely to gain understanding about the fundamental nature of matter and then how our universe began.
And how we do it is via particle collisions. So we largely collide particles itself.
And we have a huge accelerator complex where we collide protons on your light speeds.
And the largest program at largest research project we have is the LSE program, which is the large hydrogen collider.
It's 27 kilometer circular ring, which is 100 meters underground.
And you have superconducting magnets that try to steer particles to a neon light speeds.
And particles eventually collide at neon light speeds.
If you talk about the three pillars of the LSE program, it's composed of the three things.
So one is the accelerators themselves, which try to accelerate the particles.
But what is more interesting is when they collide, we have very large 3D cameras of sorts called the directors where these collisions happen.
And finally to understand what actually happened, we have to analyze this data in this world compute.
And so talking about the directors themselves, they are mostly like 3D cameras.
And what actually happens in the LSE ring is you would actually send bunches of protons, like 100 billion protons each.
25 nanoseconds apart. So it gives rise to like almost 40 megahertz of collisions, which happen every second.
And you set processing this data to come down to what actually is more relevant for you.
And ultimately, of course, the particles do not remain, it's only the data that remains. And that's the challenge.
And putting this into a much more translatable perspective of the data flow, this is how it looks like.
So from a very large experiment, one of the bigger experiments is settlers.
We have a workflow of where you have this 40 megahertz of collision events.
This initially passes to custom hardware, more like assets and you know, custom hardware that tries to actually filter out this collision data.
So you cut down the data from 40 megahertz to almost 100 kilohertz of data.
It'll still a lot of data. So 100 and 60 kilohertz per second, new storage system can sustain over, I mean, 3 years of LSE run.
And then, this is further cut down with MoCPU and GPU event firms. You come down to a more acceptable data rate of 10 to 40 kilohertz per second.
And this flows into the large scale disk storage system, we have it's own call EOS.
It's also all the data, raw data is also archived with the tapes system called CT, which would be the next stock in the series.
We largely analyze this data in batch firms using HTC Condo.
And finally, the data is also not just analyze itself, but it's also across the collaborating institutes all over the world.
And this is the WLCG and we use a data management middleware called GRUSCO, which should be also talk about two stocks later from Google.
Talking about EOS itself, currently, as of this year, we are almost storing an extra weight of hosting an extra weight of physics data.
And this is the evolution of storage needs over time. I mean, the actual storage we had.
So from 2010 when the project just about barely started, it was around two petabytes to almost an extra weight of data now.
We are also, I mean, the data is also across, I mean 8 billion files and, you know, we are close to 70,000 hard.
This is the point of time. The largest instance we have is 180 petabytes in size.
So this is another large experiment we have it's own. And this is the largest storage instance we have for that.
And here, we are actually having data rates much higher than the usual experiment.
So we have incoming data rates going up to almost 250 gigabytes per second.
And then the scene workflow, what I explained earlier, happens within the data actually is also moved across the globe as well as, you know,
active tape and various other things. So, so why did we need to build another shared storage files system again?
So this project was started in 2010 and the main goal back then was, you know, suitability for physics analysis,
because physics analysis is actually like a large streaming workflow. So it seems equal into something like streaming like 100,
and the next six movies where people are just seeking around.
Another important aspect is of course, the remote accessibility because the data is not analyzed just locally at the certain data center,
but analyze all across the globe. So you need to have something that actually can speak wide area networks as well.
Then of course, the standard things that come across with, you know, a large lab of 16,000 users, you would have the resource sharing with
quotas, accels and securities. And you mainly want a quality of service because you have not only the machines writing data at a very fixed rate,
but you have users analyzing this data and you need both of these to work together in the same system.
And one of the largest constraints we have is the cost, even though the data is always constantly increasing our budget is more or less fixed,
which also brings us to the other point of heterogeneous hardware support, because every time you start buying this in a data center,
you would see that the market has evolved, the latest and the greatest sizes would be the most expensive,
but then if you go for a more cheaper hardware, it may not be the size you previously installed.
So it's very important that the storage system actually speaks multiple, you know, disk sizes.
So I mean, the another aspect is because of this aspect, your data centers, physical space does not always need to expand, which is actually a big premium,
but you just by larger disks, but then you cannot always retire older disks.
So this is the use project, it's started in 2010.
And if you see the storage department at some, we actually host a two major, I mean,
storage systems, one is the use, which we developed at some, the other is set, which is largely used for the IT applications
and many other applications that we'll done on the storage infrastructure.
Otherwise, users largely the file store that supports the physics analysis needs of the lab,
but it also supports many of the user home folders and this kind of use cases with certain box,
which is like Dropbox, like Sync and Share platform, that also gives you access to many other applications like Office and
I mean, also even analysis workflows.
It's also the disk sentence for the taper table system, which we talked about in the next talk.
And as far as the protocols we expose, of course, the main protocol we expose for the data transfer,
is something called root, X root, which is common protocol that many of the high energy physics clients speak.
But also not only that, we expose a few centuries because that's largely intuitive for end users who do analysis.
So that's also common interface that's heavily accessed.
So any general purpose client can mount uses a few file system, which makes it usable everywhere.
And as far as the non-technical and also mobile clients and everything else,
we have a common interface exposed via certain box, which accesses a web UI and also has mobile android devices and many of the other things.
It also exposes a Windows interface so that Windows clients can also mount this.
In addition, we also have HTTP and JRPC interfaces.
So that the data upload can also happen via HTTP, which is largely useful for the transfer across the globe.
And largely, if you see how the architecture of deployment itself is, we largely split the,
I mean, installation into multiple instances, each serving, it's own experiment or something like this.
But one experiments work flow does not affect the other.
And if we talk about certain itself, elixine is not the only experiment we do.
We also have a lot of small and medium experiments, which are also hosted in the years.
In addition, it's all the users' storage for documentation and everything like this, which happens in sandbox.
All of this happens in the use storage.
So we just split it across into more logical smaller deployments so that they can run more easily.
And talking about the peak, the data rates over the last last year, it was in the order of 150 gigabytes per second for one of the experiments.
During the higher one, and then it goes in the order of 20, 30 gigabytes per second.
This is only the data traffic, but then it's not only the data traffic that the storage system has to handle.
It's also the user analysis workloads, which always go in the order of 200 gigabytes per second.
And if you see the networks are going in and out, this is what we are talking about.
So when we have a large streaming analysis workloads running, we almost hit a peak of almost a database per second.
And our incoming data rates also hit almost a database per second.
But when you're talking about incoming data rates, it's not purely the right, because every writer's always mirrored at least,
or also our erasure code, which means like you'll have an item notification of at least 100% or like 20, 30% depending on what erasure code coding configuration you run.
And talking about the architecture of the storage system itself.
So you'll actually have a metadata server called MGM, which handles the authentication of clients, and then clients directly talk to the storage servers, which are called FSTs.
The metadata itself was actually persisted until this persistence table called the co-op DB, which we developed it soon, which actually shows the persistent metadata.
And the idea is that your metadata kind of scale up kind of architecture, because it's usually very small.
Even if you're talking about billions of files, you're just talking about a database of metadata.
So it's just not something that actually exceeds an NVM or SSV.
And the data story, of course, you have to talk about multiple files storage servers.
So I mean, you're talking about exhibits of data servers.
These demons are built on top of this framework called X-RootD, which I'll talk about later.
And this is largely how, I mean, it's similar to any shared files system wherein you actually talk to a metadata server,
and then it really helps you to direct storage servers, where you actually write the actual data.
And this is a simple example of the data workflow.
You're client would actually contact the MGM and ask for a given file way to write.
And it will tell you which files to be written.
It's all persisted into this persistent metadata store.
And finally, once you've closed the write, this is actually, you know, the system is ensuring that the,
when you issue a close command, it's both replicator and also everything like the checksums and everything is calculated.
Any write would always involve multiple storage servers depending on the file layout.
So it will be for replicator, it will be to replication.
If it's a ratio coding, it's at least that number of, you know, K plus M ratio.
And talking about the demons we run a useful, one is the metadata demon.
It's quite complex. I'm not explaining the main things, but the only thing is that it basically translates your metadata path like,
into actual file IDs that is actually stored in the disk.
And it also does all the maintenance activities of bringing and managing many other things.
And talking with the persistent data store itself, it's actually a very simple thing that actually talks,
read is on top of, I mean, it kind of speaks latest protocol on top of ROXDB.
So, every metadata transaction is essentially a rough transaction that eventually is persisted on ROXDB.
And how we actually host the co-op DB themselves are like they are usually collected with MGM nodes,
having an NVMe storage.
And this is usually how you scale these numbers, like, you know, you'd store about 200 megabytes for every 100 millions of files.
And talking about logical concepts, you have, of course, nodes.
And then, you know, you have a concept of a group, which is a aggregation of filesystems.
And then you have a logical group of filesystems called the space.
Coming down to the actual code implementation, it's actually implemented as a plugin for the 6D framework.
And then, which provides a POSICS like API and also, I mean, other methods of authentication and these kind of things.
The X-roody project project that was started on Stanford, linear accelerator.
And this is mainly a C++ framework that provides POSICS like namespace.
And also, many other clustering functions.
And it also gives you authentication mechanisms.
And, you know, mechanisms like the party copy, where you can actually do a sync of one file server to another.
I'll not go into the details of how, you know, you cluster and these kind of things, but then I've left it in the slides for, you know, data reference.
Talking about US itself, when US is authenticated, it's usually via Cobra's.
Or in Excel, it's largely Cobra's that we authenticated US as well.
Otherwise, you also have X-Final, final times certificates.
Then, you have a token system where you can actually give us token of a given scope for a particular file and data access.
And finally, you have also integration with over the these kind of things so that you can target into your authentication system.
Access control is implemented similar to NFS V4 accles.
These are mostly attributes in the metadata server.
And it's also not restricted to users alone.
You have also group concepts of groups and egroup sets on, where you can actually deny your grant access or particular permissions and files.
You have a quota system at the group user or egroup level.
And as far as an administrator is concerned, you also have an access interface where you can actually ban users groups or things like these.
And this is quite essential when you are actually having lots of people running very heavy analysis workloads because these workloads can actually sometimes kill the entire project.
So we have to make sure that these things, you know, at the rear case when you need them, they are available for you.
Talking about the one of the most fundamental features, one is the file layout.
This is a policy, especially how you want to replicate a file.
Default does at least like to replica, but you can usually for the large experiments is usually a ratio coded in, you know, two or three paraditians.
And you can, I mean, go up to two device stripes.
And these are not consequential just at the file level, but you can also do it at the directory level or a space level or something like this.
Or you can even have policies at a user level.
And you also have a system that can actually convert from one file layout to another.
So when you, you can actually write files in a replica mode and then later convert them to a ratio coding, with the system.
And we support multiple checksums, of course, the usual MD5 chart and these kind of things.
And all the files checksums are guaranteed to be verified, I mean, before you issue a close command.
And also a ratio coding, of course, always, I mean, computes block checksums because that's very essential if you wanted to do.
If you wanted to easy to rebuild.
One of the other fundamental features is of course, the converter interface which actually tries to converge files from one interface to another.
And you also have this interface which can actually do a soft deletes in this kind of thing, so that you can actually expire files.
One very important feature we have is the ishipping policies.
This is very essential because you will have data from the machine that always comes from the electricity program, which is always critical.
So these rights service have to go through, but a user doing analysis can actually wait.
So you can actually say priority for rights which is configured for every experiment.
And you also actually can cap bandwidth for experiments.
So you can actually cap set of policies like this user does not get more than 250 megabytes of data streams.
And that the hood on XFS filesystems, we also, I mean, configured the Linux scheduling priorities for these.
Not only for the data streams, for the metadata also we implement a quality of service.
And here you can actually throttle operations like file open or you know operations like listing or these kind of things.
You also configure limits and the amount of threads that the user is pointing.
And also that you will probably need a server responding.
You can actually configure these things to actually have a hard limit so that when the user actually comes with the operation like this, then they are actually installed.
So that they can no longer do the operation.
They have to wait for the, I mean, for the given time before they can actually do.
And this is actually because you don't want one user who is running a batch job with 5000 compute notes.
Taking down if we are the resources in the server of other users who just want to check something very simple on the,
home source directory so something like this.
And if we talk about applications that are built on top of us, so one of the main applications we run as soon as something called serverenbox,
which, you know, provides this interface similar to top box and also who is a lot of applications on top.
At, as of current speaking, we are talking with 31 key user accounts at certain box.
And there are many other applications like, I mean, and as a server workloads like Apache Spark and these kind of things.
And then largely over analysis workloads of Python notebooks and these kind of things also happen at top of this.
And talking about the current and future development roadmap, one of the important things we have in,
so what is the SMR feature, where we are trying to actually write data into SMR disks.
Currently we have added basic support that, but then we are actually waiting for a larger test pool to understand where we can actually do it at scale.
Our understanding is that HTML hard to support should just work out of the box because in theory HTML is very similar to your normal conventional magnetic recording.
Talking about loop cost flash, we are also evaluating QLC flash and these kind of technologies.
We have a research project that is in the pipeline which should be having a position open very soon at turn.
So, you do look in our jobs portal.
One of the other things we are working on is the metadata server scalability and in terms of trying to make it more performant.
One thing we are also working on is the SMR gate wave support because that is one of the most prominent protocols nowadays for any of these file system like interfaces now.
Another thing we are working on is the different quality of service storage tiers.
So, you can actually write to a flash storage and then this actually eventually gets lifecycle into a slow storage and then tap and these kind of things.
And I see, I mean, exploration of where we have fast network interfaces for our, I mean, though, through the farms.
And this almost links me to the end of the talks as far as concluding the March group.
So, the US was developed at some primarily as a software to support the large scale physics use cases, but it also supports use cases beyond physics.
Largely, we saw it's own box which is actually used for user filesystems not related to physics alone.
So, every analysis is a more or less involved COS.
And I mean, it's also used even data acquisition systems which actually have very high transaction rates.
It's particularly appreciated by the users because you have the access to unified namespace where you actually access the same file on your physics analysis system,
so you can actually use your own box to write your PDF files or something like this.
With the next run of a let's see a few years ago, we are also trying to improve the system so that we can actually support the data rates for the next run,
which is expected to have 10 times more the bandwidth requirements.
The software is GPLV3 license, so we would love contributions, all the links would be in the next page.
And largely about operations itself, we don't actually actively run them.
We are running your system as a basic support, but it still seems like we still run at the 4th 9th of availability even with this.
We largely do not have large operational incidents running them.
So, I mean, it seems to work pretty well for us, and it's not only used at 7, but also few of the other high energy physics sites like Fermi Lab.
And European Commission's GRC is using yours as an analytics platform.
And finally coming to the project links themselves, you have the documentation and a bit here.
We host server packages for Alma and all the EL variants like Red Hat.
Client packages, we do have open to another things.
You also have Docker images and these kind of things if you want to just run these as container images.
And two months on the line, we also host this use workshop at this thing called Techweek Storage, which I think will be covered in the next two slides.
That one moves with the end of the presentation.
And I'm open for question.
Let's lift.
Yes.
So far not yet.
So the question was, for the yesterday's support, have we considered RGB standalone?
We haven't evaluated yet, but I did see that the RGB now supports post-6 file systems.
Is this merged in RGB already?
Okay. So the answer was like, I mean, there is a feature developed in RGB to expose post-6 file systems.
Once this is stable, we will probably evaluate that as a referendum.
More questions? Yes.
So the question was, when we are talking about limits that we can apply to users for metadata operations and these kind of things, is this feature built into X2D or it's a feature that we developed.
And the answer is actually, it's not a X2D feature, it's something that we developed on top of EOS.
Primarily because it's something that we experience a lot when we have user activity.
And this is something that we developed over time, understanding what physicists usually do, which is trying to do a lot of not the correct way of accessing a file system.
We do lot of listing operations for checking whether the file exists, which shouldn't be done in this format, but then people do it anyway.
So people do metadata intensive workloads without them intending them.
So when you kind of read them with them, they kind of get creative and then you understand how to do it properly.
So it's something that we built.
So the question was, what kind of this we are using? Hardness.
So the answer is, we largely for all the physics stories and it's all hardness.
For very limited use cases, we have essays these, but we are mainly bound by cost, so almost everywhere is just hardness.
We don't have budget for running a large NVMe storage.
Any more questions?
Okay, hi, one more question.
So the question was, how often do we replace hardness?
Normally, I mean, you come with a three or four year warranty with the hardest, but usually we try to run them almost to the end of life cycle.
So if it's not there in one of the slides, we do have a side of all the hard-wage operations we run.
So the oldest hardest we are running is almost 10 years old at this point.
But this is not always a happy thing to run, because eventually over time you would actually have a lot of operational issues that come along with it.
But more or less we are seeing like, it's largely okay for large physics files, because I mean, as far as you constantly do all of your corrections and these kind of things they are fine.
It seems to work for us. Yeah, one more question.
So you have states that you consider as your mind, because there are these might be not working.
All right, so the follow-up question was like, if we have it in consideration that the hardest may not work, given they are out of life cycle.
And yes, we do, I mean, this is one of these important aspects we have and the other factor with which comes with it is also that you try to, I mean, cycle to different generations.
So that, you know, you had, I mean, we try to optimize our groups such a way that they don't spend with the one kind of hardest generation.
So that you try to kind of, I mean, hedge this.
You mentioned one two for parity.
Yeah, so the person was for erasure coding, what's the minimum parity we use.
We start with only two, I mean, at least base minimum of two parity for erasure coding.
For the application, we might go with two replicas in some cases, but for erasure coding, it's usually minimum two parity.
But we do support even one parity, but it is not very safe to run like this.
Okay, it seems like it's time. Okay.
Thank you.