Thank you.
Hello, so my name is Manib Zulian Leduc.
I was living in Brazil, actually,
and my wife was working in ULB, so it's nice to be back here.
I felt that I left Brussels and landed in Geneva
and started working at CERN.
Now, I'm the service manager for the tape RFF service at CERN,
which is like kind of a big responsibility.
All the data that, basically,
I'll be sure you what happened at CERN,
but for me, CERN is like that.
There's a collision and everything lands on tape,
and that's it.
The end product of the LHC is data.
If the communication is cut outside CERN,
it can only land at CERN and in the CTS service.
So this is what I show you.
And it's kind of important because,
I think it's one of the only measurements that came out of IT
and through this presentation by the director general of CERN.
So, basically, what is the name space statistic of what we have on tape?
This is the amount of data we have on tape.
CERN runs LHC, the big thing,
start in runs, and during runs we take data.
So, basically, you see over the last 15 years,
this increase, which is very exponential.
This is start of run 200,
and run 240, so due to an increase,
this is technical steps, not much is happened there,
but reversing happens and still data is created at a slower rate.
But the start of run 3 in July 2022,
we are run 320, and a 2.2 years to double that.
So you see, the fully story called data,
is just two years of future data you're going to take.
This is kind of thing we're dealing with.
And only in 2024, we took 250,
of data we had on top of tapes.
Okay, so this is what we're talking about.
In terms of what we have on later,
this is the values runs 1, 2, 3.
We expect to be at 1.3,
at the end of June, June 2026,
but it looks like it will be worse.
And see, that the future it's always green,
so we won't lack work, that's just real.
This is what it looks like in terms of monthly written data to tape.
So you see, run 1 was around 4.6 petabyte record.
This is Avian and of the year,
when they collide, I am together in the LEC.
And run 2, the record was 16 petabyte and last year,
20, 24, over 40 petabyte for four months.
Okay, so record is everyday basically.
This is the infrastructure we have to deal with that,
to deal with all those tapes.
This is one old one I'm showing you.
This is the GB, which runs here.
This is the enterprise world.
This is the linear tape open world.
We have two different technologies,
so that we share the risk and take the cheapest one.
At any given moment in time,
we don't put all X in the same basket as the strategy.
As you see, we have between the enterprise drives and the linear
to open drive, about 180 tape drives.
The tape drive is where you mount those tapes.
It's got by a robot sticking there.
And then you write or read on the tape at 400 megabytes per second.
And this is what we have to feed as a buffer in front of the tape.
So let's see a bit about the evolution.
This is historical stuff.
It's how we did it in run one.
Basically, there was just one strategy stem that makes disk and tape.
As Abhishek showed you in the previous presentation,
he also was introduced in 2010.
It was used for run two to distribute,
all the bandwidth requirements were.
Castor was kept for data transfers to tape.
But the problem here that you see,
the disk plate of the experiment is between two different storage
side.
This is still this based, this based.
So if you want to use a full disk you're paying for,
you need to split your data.
It's a bit problematic.
And as tape drives read from the previous storage tier.
That's 100 megabytes per second per file.
This complicated unit like for run three,
we would have needed 70 better of our drives to get the bandwidth.
We didn't leave little capacity for the rest.
So basically for run three,
we did this decision to replace Castor by CTA,
which has a SSD based tape buffer.
Use driven so that we don't have to deploy to develop
the protocol into different system like Castor and use.
Use the ZTOL and on the CTA side,
we just care about efficient transfer to run from tapes.
SSDs are also much better because in the passing castle,
we were multiplexing the files in memory.
So you read 10 files, but 100 megabytes per second.
When Wi-Fi is ready, you can stream it from memory to the drives.
We stand gig files with luck and faster drives.
That's getting problematic to feed that in memory.
So SSDs is the way we multiplex those files.
Native speed of SSDs at the time was 500 megabytes per second,
which is higher than the tape drive speed.
So it was nice.
And we had to switch and try this.
For the run four requirements, you saw before.
There's no way we would write two exhibits during three years
with 100 of petabyte of disk in front.
So basically, this simplification made a bit more complicated.
But you see a formula one is efficient, but it's not.
You don't start a formula and I want to start a standard clar.
It's much more complex process.
So now we have EOS for the disk side.
FTS is transferring the data to CTA.
Now it's compulsory for the user to check that the file is on tape.
Before removing it from EOS.
We have just one copy on the tape buffer.
If you lose the file on the way to tape, you write it.
There's no need for us to have redundancy on the CTA side,
so that we are more efficient, no redundancy fine.
The file is still in EOS.
Just keep it until it's on tape.
So what is CTA about?
Basically, it's basically the part that is queuing the transfers.
And basically, you write a file on EOS on the EOS cache instance in front of CTA.
It's in an event queues the file and CTA will scale you the tape mount.
And the tape drive will read the file from the buffer.
And this is this and write it on tape efficiently.
So you can copy CTA with something else as a buffer in front.
It's on its EOS plus CTA, but this is a V cache.
And they use CTA for the tape movement engine.
The architecture is this one.
So we have the schedule for the queue.
And the catalog to understand on which on this tape, for example, the 100 file is which file.
From which EOS CTA instance.
And as well when you read back, you trigger an event that you trigger a amount of V-s tape,
mount the tape, go to the 100 file and read it back, pat it in the buffer,
and scale it for transfer.
So a lot of small little steps.
What does it look like as a production infrastructure?
So basically, to drive well is traffic.
We are only 64 hyper converged servers.
So for me, an hyper converged server is a streaming machine.
16 times two-terabyte SSD, set-eye SSD machine.
Connected on 25 gigabits ethernet.
A very small blocking factor because whatever enters one machine needs to go out at the same speed.
There's a bottleneck inside.
I cannot read the same throughput to tape and data spiling up.
Basically, if you take those 64 servers, it gives 160 gigabits per second,
a full deplex throughput.
You write a 160 gigabits per second to those 64 servers,
and you get 160 gigabits per second to tape with 400 megabytes faster.
Every single view, which is an experiment, has its own set of eos and eos.
It stands for compile, and they write to the same set of tapes and writes.
So if one experiment runs, write less than the others, we can give more drives to the other one,
and we can distribute throughput more fairly.
In terms of buffets, it's more or less very conservative.
You see, we still have, we kept something like eight hours of buffer.
10 gigabits per second per per each experiment.
This is roughly what we have 60 gigabits per second of throughput to tape with 180 drives.
And that's kind of it.
Inside one of those instance, we also separate what goes to tape.
It's a different set of SSD than the things that comes from tape.
Because you see, when a file is written there, it's evicted when it's land on tape.
When the file on here, it's evicted when the user moves it out.
So if we had only one buffer, a user not moving files fast enough,
we'll fill the default space and then again it cannot archive.
And archiving is critical because we have one example to move quite regularly.
And another thing is that during the year, you saw the traffic and so on,
but if we look archive volume and stage volume, we have some data taken by it,
they take a lot of experiments right a lot to tape.
And there are some not so much data taken per year.
We have some technical stuff and stuff like that, where they read back the data.
So in this case you see, you can anticipate what we happen.
During the data taken for AVI, for proton proton, you get a bit less archive throughput.
A lot during AVI on, but then in between, they read back and they read back a lot at the end of the year
to distribute the data to other sites.
Okay, so basically, we have 10 SSD server, those like per commercial server.
And during the data taking period, they put 9 on the default space for archival.
That gives me 0 to 29 gigabytes per second of throughput to drives.
With this, I can drive 7, T, tape drives at full speed.
And I still keep one server because they may need to occasionally read back it.
There's no server on a return site for get about reading.
And when you enter the end of the year or recall season, it's more balanced.
But they still have at least 10 gigabytes per second of throughput at 12,
which can drive 34 drives for, for, for, for, right, 34 for read,
so that they can do still late archival.
Okay, it gives us a lot of flexibility.
How do you really make the dimension that? So, you see, I went back to the benchmark.
I did in 2018 when I was buying my machine and testing my, myself, about them before COVID,
and I still in production, basically.
So, basically, I did a bunch of streaming performance using DD,
simplex-wise, so you do a lot of writes, a lot of reads,
increasing the number of SSD you used, and then you see at one point,
in the notebook, you reach the button neck on the HBA, my HBA on the machine,
cannot go over seven gigabytes per second of throughput inplex.
Then you test a duplex through duplex, because what we read is what we read.
What we write is what we read, but we read the different times on different SSDs,
because we read the drives read later when the clothes happen, it's cute.
So, basically, you see, you get 3.5 gigabytes per second of writes
and read at the same time, from different sets of SSDs, that's okay,
because it will be statistically distributed between the number of SSDs we have,
since it's under an N60, where we see we have 70 reads on the N60,
in terms of stream writing, we have about 120 streams writing,
so there's not a lot of collision, and we need to tackle statistical performance.
So, basically, the HBA was good enough, in fact, because 70 gigabytes per second of internal throughput,
is more than accumulated simplex throughput of 125 gigabit per second,
port, which is around 5.5 gigabytes.
So, no internet, now we need to move to how it looks like when we're going through this.
So, basically, I take one machine, here, this is just one machine, right to the SSDs,
read from the SSDs to tape. So, one color is when SSDs, you get the 16 SSDs here,
saturating the network at 2.4 gigabytes per second, and then when the tape are mounted,
here you see, one color is when tape, when the tape are getting mounted,
we read from different SSDs in the late manner, like I was explaining.
This data in production is around 20 minutes, okay.
And you see here, the strings, one server, can drive 6, drive full speed in parallel,
to give this back the number we had.
So, that was the test, those were the test I did with one server before internal production.
Now, next step is then to scale up, but your nice network together,
so we have a separate network for tape, to drive that type server,
400 megabytes per second here, and they're a blocking factor that is bigger than this one,
because here is why we have this buffer.
And then in production, this is what we see.
Green, my KPI is efficiency, so the greener, the more efficient,
whatever is green is over 350 megabytes per second,
and those are 100, one line is one drive, okay.
You see here, another range one, that was one drive,
one type server, and the port was locked at one gigabyte,
when gigabit, as a net instead of 10, so it stayed like that for, it's really bad.
When I have an alarm on that, and you see your system strings that are not used a lot,
basically, you see, those are the generations I showed in earlier,
we have L29 here and here, 1170 here and here,
so that I capacity 50 to rob like that tape,
and this is slightly older generations that we use only when we need bandwidth.
So you see some peak here, where we use them a lot,
because it was mostly saturated, or we needed to do an intervention
on one library of more recent media.
And here, we had a peak of throughput during this period,
and we kicked them in.
The premise is that whatever we write using those older dried,
we learn on older media, we will have to read it back,
and write it to new media as that's what we call Repack,
which is a huge part of operations.
So let's go to Repack.
That's about operation, this is the kind of tools we're using.
Yep, and what is Repack?
Basically, that's what I said, you take one on mega, you read it,
and basically, when you have 50 terabyte tape,
and the previous generation was 20, you read 20 terabyte,
2.5 terabyte tape, and you write them on a single 50 terabyte tape,
so you make space in your libraries, because libraries are just slots.
Then you change the drives to newer generations,
so that you can read the new media,
and then you feed up again your library,
until you can repack again for the next generation.
The Repack is doing the same thing, it's reading from tapes,
writing to an SSD buffer, and from the SSD buffer,
whatever is cut there, it would be cut for archival
on the new generation of media.
We have this nice, complicated automatic tripack state machine.
I think that you can look at those presentations if you're interested,
but that the way we manage it automatically,
and this is what happens in production, basically.
You see, this is the set of our buffer of who it is.
It's up to 100 terabyte,
of SSDs for repack, and this instance,
when it's filled up by a few more five for archival,
so that we drain it, and we start sleeping some retrieval.
Retrieval is reading from tape, filled the buffer,
archival will lower from the buffer, so we have a top limit,
and a low limit, so that we had a remove tapes for read and write.
You see here that during three months,
we had three gigabytes per second,
reading and writing at the same time for repack tapes,
and we repacked, basically,
35 peta and 2,000 tapes over three months,
while we were still archiving 40 terabyte payments.
So, you see the repack is this sign line on top of archival here,
like the three gigabytes per second line on top of the 20,
20, 30 gigabytes per second of data taking,
and this is what it's a bit more obvious on the read side,
because repack was only thing reading,
mainly during data taking.
One thing, I'm happy about that CTA is also opening
to the other communities.
We're trying to drive protocol,
like ABFAC was saying,
a good concept of mostly HTTP based now,
where we wrote on HTTP,
tapressed API,
that CTA can use to mount this Monteq query,
where my file is,
and stuff like that.
We use it a lot.
But then, CTA is not just about physics,
because even at some,
you see, we have plenty of custom uses for various aspects.
I'm running this AFS to show,
because AFS backup's actually learned on CTA,
and GFS backup's file is backup's on there.
Use namespace backup.
CTA backup's a backup in CTA as well.
And yeah,
we need to also evolve our backup off at some,
and CTA is the fact to stand up for whatever needs to learn on tapes.
We would like to,
we're looking to adding more protocol to CTA,
like S3 Glacier.
HPC community is a trail growing,
but you see HPC is like,
it's a close community,
we're not really coming practice today, I'd say world.
And yeah,
basically what we wanted to demonstrate,
that the mineral performance and efficiency was good for R&F.
We also reached R&F during R&F,
with over 24 gigabytes per second,
for one instance.
And the next step is clear,
oriented towards reaching efficiently,
because the disk capacity,
with regard to all the data that's coming for R&F,
is shrinking.
So there will be more,
much more reads from tapes during R&F,
and we need to place files,
intelligently.
So one of my projects,
those days,
is work on archive metadata,
to give a color to files,
and put them closer together,
so that we can know how efficient we will be when we're really back.
And that's it.
Also,
if you have,
if you want,
we'll try to pinpoint source storage with tapes,
things EM here and contact us,
because we're really eager to do up until I'll side different.
So thank you.
Thanks for questioning me.
Alright, So when now I will number five,
from years to all,
So the question was, when we move one file, so let's go back to, we were here, yes, kind of.
So when we move one file from EOS to the SSD, before it moved to tape, do we stripe it on the
SSDs? In fact, we, we don't need to, because we have enough throughput on one SSD to
re-box everything, and CT manage single file only. So we take one file, the file is on
EOS, it's moved on CT, on one SSD, on one file system, and then it's straight from this
file system, and on one tape, in one F-sexo, in one position on one tape, and that's it.
Yes, what's you archive something from tape, and that's not through the file
system anymore, you only keep one copy on tape, and that's it, so as well, multiple
back-ups of files. So the question is, how many copies of a file is on tape, do we have?
So basically, at some, I mean, the pieces are like that they don't, they wouldn't
want to keep all the eggs in the same basket, especially if some or the data sounds
burn at one point, they have secondary point where they duplicate the data, so physics
data, we have only one copy at some, and the second copy is as well. So basically, it's
often distributed across the different tier ones, for example, if you take atlast,
there are main tier ones outside sound, is brew cabin national lab, so they usually
have, if not a copy, at least a good portion of the data. There are some pair of ways
slightly more complicated, especially during AVIon, when the data rates are increasing, and
we take, we took, for example, 21 gigabits per second, for system period, for CMS,
that's more complicated for them to move it at the same time to tier ones, to the processing
and so on, especially if they're running over the SLAs we agreed before. Okay, things are
the machine for 10, if you go to 21, expecting to be a bit more complicated, so.
But now, we have mostly one copy, the only, we have two copies for, for data that's
only, it's, yes.
So I would first use, use to use CTA.
Question.
That's the question.
Can CTA work with something else, and use, for example, S3.
So, in fact, CTA, if we go for the, here, I have a backup slide.
Now, I already show you, showed here, that we have, use a sound in front of CTA, use
writer or buffer, but a Daisy in Germany, they have a D-Cash disk system, and they hooked
it on the back of CTA. This was the first step for us, as you see, we can address,
one type server will read either from EOS or from D-Cash, and we have to use common
protocol for, for, for, for authentication for triggering events and for, anything like that,
like we're, we're moving to GAPC for this reason.
They use HTTP for the transfers from D-Cash to tapes, we're still using XDD, but we are thinking
about how to evolve that, and you see, this, somehow, we will have a technical student who will work
on how to deal with other protocol, like S3 in front of tapes, so that we can interface
backups in front of CTA.
So we'll see at this moment, if we are more things, more things to change, and you see,
I'll just say, the run, the current run 3 is up to mid-2026, which will lead us some
time to redo things, we have things in the pipeline, like, reading back efficiently,
we will change the scheduler, we are likely to change the APIs and stuff like that.
Very likely.
Yes?
Yes.
So the question was, when a new generation of media accounts, do you copy the whole
exhibit of data to the next generation, we will have to, because we don't have anything
in its space to keep our data, and the next trip back, we will have to do, we'll be
400p, so 400p, a 10 gigabits per second of throughput reading, plus 10 gigabits per second
of throughput writing is one and a half here, so we have to be faster.
So you see, yeah, you see, take my job out.
So the good thing with the repack is that it expects you to need a shutdown.
We have massive repack that exercise our infrastructure before the next run comes.
And you see, if we, as I said, I mean, future data is just two years of future data we take.
So if we cannot deal with historical data, forget about dealing with what is coming.
Okay, so we have to, it's a good exercise.
And yeah, we have to go for the generation of media that the next run for, we use as well as you,
your legacy of media and drive is just insane.
Plus we have to pay maintenance and spare drive.
So slower drive costs more and slower, so you have to pay more libraries if you keep all the media.
So we really have to back up to repack, so that we know and we demonstrate that we can deal with future data.
And you see, in fact, we get aside a city architecture with SSDs during LS1,
while still using Castor for repack, because we can explain it with repack.
Yes?
When the next generation of tips in ready, so HL9 or what else?
HL9 is already there, HL10 will come soon.
And we follow the cycle.
We don't know exactly, so when do you expect the next generation of tape media to be available?
So to your 10 will be the next generation, I think it comes either end of this year or next year.
But then you see, we have also a technical stop of close to three years.
Is it, should we focus on improving performance that we repack fast on the very last year of the shutdown?
Do we need to get that with the two types of drives I showed you?
Like we have enterprise, we have LTS.
The cycle is more like, there's LTS one year, there's enterprise another year.
So we see, because buying a full exhibit back of data is on the expensive side of things.
So skipping one major generation could spare a bit of money.
Can you use the compression and encryption from the tape drive?
And how do we manage the tape drive cleaning process?
Okay, so do we use encryption and compression on the tape drives and do how do we manage the cleaner process?
So basically, the data is mostly highly compressed.
We compress it on top, but basically we don't get much more.
We don't get a huge gain, because it's mostly a root fight that are already very compressed.
So if we see that we can fit 10, like 40, 10, 10, 10, 10, 10, 10, we will contact the user and say, hey, please compress before.
And regarding encryption, we use encryption only for non public, so physical, all physical data is supposed to be open and near.
So that's not critical data, we need to encrypt.
We only encrypt backup side of data and we manage the keys, basically.
You configure the drive with a key and it will encrypt on the flight.
And I think that's it.
And the cleaning that's often part of the tape of the media library, the media library, they come with their own cleaning cycles.
If a drive mounted in the entire user, they'll come to us and they will mount a cleaning cartridge every now and then.
Yes?
We have a specific procedure when a new version of ASS comes out, or if you want to test the ASL of a matrix, let's say,
to see if the performance is similar.
So do we have some procedure to test the performance of ASS now, because we don't use ASS by definition, we use CT.
For the next generation of tape drives, so basically we put them on dual-copy tape pool, because as I said, we have the physics stuff is one copy, but we have some set of tapes that have two copies.
And one of the two copies will be landing on the new media.
So if we have some issues with the drives, a free series of tape or whatever, we can lose this tape, because we know that the first copy is,
it's valid on the new media, and we can just did your tape and recreate the first copy.
So this is how we test it in production along with the rest with no risks.
I guess time's up for question.
Thank you.