Thank you, and good afternoon, everybody, so I'm Mario, and today I'd like to talk a bit
how we're working to implement an S3-fronted cold storage at serve. Just very quickly, a bit
of what me, so I'm computing engineer, sir, and I started just three months ago, and I joined
the T-Percad on the cap team. And my first task in the team was actually to review proof of concept
it was done by a summer student last year, concerning putting a tape back into an S3 endpoint,
and my ultimate goal in there will be to design an S3 interface for our tipping infrastructure.
Just briefly, what we will discuss today, so we'll go over the project goal and give some
technical context that will shed more light on why the decisions and discussions, why do what
I will talk about. Then we'll analyze the proof of concept that has been developed last year,
and then we'll go over brainstorming of the possible architectural solutions that we will be
and at the end, there will be some time for questions. First of all, what is sir,
so it's the world's biggest laboratory for particle physics, it's mostly concerns itself
with the fundamental physics study, and it's mostly famous for it's most famous for it's
LHC, it's a large hydrogen collider, which is a particle accelerator that is located beneath
the Geneva area, spanning across Switzerland and France border. Why does sir need tape in the first
place? Well, because experiments need to write a lot of data and it needs to be stored somewhere,
and sir, the tapes constitute a very efficient means of storage. In fact, just last year
last December, we had one extra bite of stored data in our tape libraries for x-marry
mandate, but that's not the only use case. We also store user data for could be for various
reasons for compliance or disaster recovery, but that's another use case, and the way that we
stored them, so we have developed internally software to manage the tape libraries, which is called
the Seren tape archive or CTA, and it's open source software, and so the idea here is to have
CTA as a tape backend to a nestry plus place here, API endpoint. Why is, because so as three
is pretty much the industry standard, there's a lot of clients support for it, and also we want to
fit into the ecosystem that is already there with the S3, so we want to avoid reinventing the
wheel of creating another protocol, so S3, what constitutes a good way to interface with it.
I should also mention that for the time being, this project will at least internally
will mostly target the user data archive use case. So to give a brief visual about it,
so imagine this being the whole service that we want to build, so it's take back up service
for users, you have backup buckets, and there must be an S3 point somewhere, but you can identify
within this system a subset, which I would call the appliance, which has its own aspirant point,
and this is the one responsible for actual physical tape storage regardless of how the system
is internally built, and this will be our main challenge to build it.
So first, technical context about certain type archive, so again it's the software that
provides physical access to the tape libraries, and the interesting thing about it is that you
don't interact directly with it. The way, so CTA needs a disk buffer in front, which is an intermediary
storage medium, which will act as a buffer between the client and tape, and this is because
the tape medium is really high latency, it's not always ready to write, so you need this intermediary
space to store that on, and for this reason you can see CTA as a tape back and for the disk buffer,
and for example the most support, I mean there are some support that flows for the files,
you can write files to the disk buffer, and it will be archive, or you can replace the file to be
recalled, and it will be written back to the disk buffer if it was not already there.
Yeah, there are more flows, but we'll mostly focus on these, there's not enough time unfortunately,
and it's free and open software, it's on a certain GitLab or on GitHub,
and just to be a bit more clear, so this is an example of the archive also, there will be some
data producers, which will dump data into the disk buffer, and the disk buffer, which knows that
is being backed by CTA, will basically queue work into the archive queue, and eventually the
tape drives will, when they are free to perform work, will pick up some of the work that needs to be done,
and will mount a new cartridge in there, and it will start writing to tape, you can see there
is 400 megabyte per second average, there is a reason for that, and this is for each of the
tape drives, and the workflow works in reverse, so one thing that I didn't mention here is that
as soon as a file is written to tape, it's actually truncated in the disk buffer, so the disk
buffer retains the metadata, but the content of the data is not on the disk buffer, or all of the
data is fully on the only on tape, and given this situation, you have the recall workflow to call the file back,
so there is a component that will ask the disk buffer, actually, for the file for restoring it from tape,
and this will happen again to a retreat queue, and whenever a drive is ready, and the conditions match,
and the conditions allow it, it will mount the cartridge for which it was requested, and will dump the
content of the file back into the disk buffer, so it will rehydrate the file, metadata was already there,
but now the content is also back there, and the client can access it, so a bit more about the disk buffer
that is a sits in front of CTA, so it's the one holding the metadata, you can see it has a file system
higher hierarchy, so you write as you want to a file system, but one thing that should be noted that
metadata leaves only here, whenever CTA reads from the disk buffer, it will only read the data,
and write exclusively that to tape, so U.S. actually, U.S. I didn't mention that, but this is the
software that we use for as a disk buffer, so U.S. stands for U.S. open storage, and it's also an open
source technology, but basically the fact that metadata is held only by it means that its persistent
is critical, so if you lose U.S. for example, you will be able to retrieve the file content
not there metadata, so you wouldn't know what the files are, and so a little difference with the
normal file system is that U.S. is aware of files being online or offline, so if the content
is there, you can consider a file online, you can retrieve it right away, but if the content is on tape,
even though the metadata is there, then the file is offline, because you need a retrieval operation
to be able to read it back, and it's explicitly designed for large and stable throughput,
this is important for tape, because actually tape has a minimum speed that it should be written
and written to and write from, and this is because of how it works, because it's a linear tape,
and so it spins and breaking it is not something that you can do immediately, if there's no data,
I cannot stop right away, so there are some constraints around speed, about this technical context,
I want to take away the main points about the minimum speed for the tapes, so that whatever solution
needs to be able, you can see this as constraints or features, so it needs to guarantee
a minimum speed stable and minimum speed, then the fact that metadata lives on the disk buffer is
also important, losing the disk buffer, whatever implementation is, it becomes critical because of that,
then there's no object affinity logic in CTAs of today, but this I mean that whenever there is a
bunch of work to do, a bunch of files to write on tape, then a CTA currently doesn't
have any logic around which files to write, it will just do some work, but for some files,
it may make sense to live on the same tape because they have a highest, a higher chance of being
restored together, so you want to mount only one tape and not more of them, also one currently
present semantic on CTA is that the file is considered safe when it's fully on tape,
and not if it's in between, so if you write on the buffer only, then it's not considered safe,
yet, and the clients are able to ask the disk buffer if the file is on tape, or not,
and also there's no modify files semantics or only delete, which may be relevant if you
intend to interact with a street bucket, some operation that a street can do may not be doable
as far as the disk is concerned, speaking of which the other half of the technical introduction that
I want to do is about a street and glacier API, just really quickly, I think a lot of people are
familiar with it, but I still may not be, so S3 is a product of AWS, which offers object storage,
and you talk with using the S3 API, it's a rest interface, HTTP, and here on the left you can see
an example called that you can do, so for example, the get object operational, let's you retrieve
the content and metadata of a file, and you would specify the most important learning methods are
bucket and key, which are relevant for the S3 service, so the bucket is a namespace, a domain of files,
you can see as it has its own file system, and the key is the path within the file system,
and so on the right you can see how it maps to an HTTP request, you can see that the bucket
and the file key appear both in the path URL, and a get object operation will essentially be
a get HTTP verb, and this is the general idea behind the S3, a rest interface, and the kind of
operation you can do is on object level, write, read, delete objects, on bucket level, among which,
so you can of course create a numeric buckets, but there's also the lifecycle configuration, which is
really important, and I previously talked touched upon it, but extensively, so you can configure the
bucket for automated movement between storage classes, which are abstractions of which kind of storage
medium will hold your data, then you have metadata and other operations, but one point that I want
that is very relevant for this talk is the Glacier API subset of S3, so you can see that
it lets you do the archival and recall of the files, just what CTA models, but using the S3 API,
so regarding the archival operation, it's actually not an imperative operation, so you can
not ask the S3 API to move a file to call storage unless you copy the whole object, what is
usually done is that you set a lifecycle policy in the bucket, so you're able to tell for example,
after one year, move the file to tape, and it gets moved to a different storage class, and as soon
as this happens, the object gets truncated, and then you get an invalid object state reply if you
try to get the content, because the file is offline, it's not there anymore, to get it on online
again, you should do a recolour operation, this is an imperative, so you request a restore object,
and the system will do in the background some work to get your file from tape back into the bucket,
and rehydrate it, and after the operation is completed, then the file will be accessible again.
Most important point, as Glacier is a user of facing API, it doesn't specify any kind of interface
to the tape infrastructure, you have no way to tell how to transfer file files to whatever
take system that you use, because it's all an internal detail, so the way that it will be
doable is the implementation specific, and different S3 implementation among the open source
available ones offer a different degree of both compatibility with the S3 and mechanism to
let you move files to tape, so with that intro, we can finally take a look at the proof of concept
that was worked on the last summer, so the architecture that was picked was to take the current stack,
so this buffer in front of CTA, and then another layer, in this case, Nuva was picked, it's also an
open source software with commercial production usage, it provides an S3 interface and a way to write
files to tape, and the whole of all of these three components for the appliance in the
schema that I showed you before, so Nuva is deployed on Kubernetes cluster, there's an operator,
so you deploy the custom resource, and you automatically will get an S3 endpoint in your cluster,
so in stores internally, the data on this, what's called NSFS, it's on namespace file system,
and which is essentially a directory, so it maps objects to file system paths,
and you can see how in fact in the POC we deployed Nuva using a local file system, so you mount
that director within the pod, and you declare that there will be your storage, and so it really
maps to files, and the way that you can write to tape is by using the take cloud interface that they
offer, so in which we can dive a bit, so how does it work in this case, so it's implemented storage
classes as AWS does, so there's the warm storage class and there's glacier, whenever you write
to glacier, be it because you do it explicitly or because a lifecycle rule does it for you,
nothing happens actually, the request gets appended to a log, called migrate log, the same is true
for retrieval, whenever you call a restore object action, then your operation is written down to the
recall log, and these are all asynchronous, so what we'll actually do the operation is whenever
a Crohn drop or an operator will call this managed NSFS script, which we'll actually do some
a log notation and managing locking, and actually delegate the most important part of the
world, so actually moving that up to tape to another script, which you write, so there's an
next interface to it, you can write the migrate and a recall script, and basically the log file will
be passed on to you, and that log file essentially contains the files under the NSFS, so the name
of the files that should be moved to tape, so just move down one by one, and that's up to you
after your implementation, so you can do it however you'd like, so we tested it out, and what
did we observe from this, so in interface is indeed very flexible because it leaves all of the
heavy load to you, it's also smart enough to not recall files, for example, if the disk buffer is
full, so because it's close to full, then you shouldn't put more stuff into the disk buffer,
and the user will have to wait, basically, so the user will do a restore operation, but we'll
have to wait for it to happen, because the infrastructure is not ready. I think that documentation
could use some improvement, in this case, for example, we found some updated documentation
on the, about the log format, and also some things learned unclear, like the boundary of failure
handling, when the responsibility, when something fails during the script execution, then some
features are missing, as far as I could understand, the storage class flight cycle does not,
is not doable, so you can just write directly to Glacier, and also we observe a failure at
around 10K, it's even thing is migration, but honestly this requires investigation, I didn't
have the time to look into it, but this could very well be to our implementation of the
migration script, because it was done in batch, and it's a QC. So a few more words about the architecture,
the duplication of the disk buffer is something that we have implemented, basically, here,
there's Nuva, and there's US, each has their metadata copy, which is not ideal, it could
introduce its synchronization problems, and also duplication of the provision space is also not ideal.
I call this the one more layer approach, which made troubleshooting really painful, because there's a chain
of heterogeneous technologies at work, one after the other in sequence, also the file content
migration being initiated by Nuva, Nuva is not ideal, because again I said tape is a, so it doesn't
fit the CTA model, because CTA basically will, is a long running process that will wait for the
drive to be ready to do work. So if you run a script and you delegate the data movement to
this script, it doesn't fit the CTA model, because CTA expect the object to be fully there for it,
so if you want to link Nuva with CTA, you need to buy a disk buffer in the middle to hold the file,
or a long running implementation, but let's not go there, and the last point which may be
very specific to certain, but there's an abundance of, of sex, sex expertise, because there's a lot of
production users, users, usage, pardon, of Seth, and so we also wanted to take a look at
its implementation of a rather skateway as well before going on. So with that premise, let's go over a few
solutions, but first, one last premise, there's one important feature of Seth that I should
mention, which is a, so a zika project, also called, software abstraction layer. So Seth
has internally split the way that it entered the S3 API from the way, from the storage driver,
so you could theoretically implement a driver to write somewhere else other than rados, for example,
a positify system, and you can also write filters. One example of which is the wasp lifting,
also mentioned during the previous, the previous talk, and the other important feature is the
cloud transition and cloud restore feature, in which you can declare a storage class, a cold storage
class, which actually maps to another bucket, instead of an actual storage, I mean cold storage
medium. So in this case, you can chain buckets and make them colder and colder with this feature.
And I mentioned this because, again, in the schema of the mental model that I worked with,
I distinguished the appliance, which has the actual tape, I mean the responsibility of moving data
to tape from anything else that comes before it, which may very well be another bucket, thanks to
this feature. And there are a few benefits around it. There is isolation and control. Like,
for example, you isolate your appliance from, you can control the bandwidth between the buckets.
So because the appliance needs to output a stable and a stable throughput to the drives, for example,
and other advantages. So that said, our concern here, what I will show you from
in the solution space is about the appliance. So I grew them into the solution families because
there's a few ways how you could do it is. These are just examples, but first approach, one more layer,
what we did with the proof of concept, we saw the pros and cons of this. So, I mean, it's already
working system. So US and CTA are one after the other, and US can already output the throughput that
CTA needs. So you have a layer on top, it's easy to do. But again, you need to, I mean, there's
the metadata duplication, you need to maintain your own storage drive, so you need to maintain
the interface between the two, and it's painful to debug because it's a heterogeneous.
So let's take a look perhaps what I call a solution, one look five, one more thin layer.
Another way to do it would have a thin, uh, a three protocol translator. So since, uh,
a structured, uh, basically reproduced the, um, semantics of US, uh,
based on your port objects, red object, restore objects. So that those are all, uh, things that
US does. So, uh, and in fact, one colleague of mine has, uh, created a project called US
a three, which is a module for diversity gateway, which does this exact translation. So this is
another way to do it. Uh, you save yourself the metadata duplication because there's a direct
translation. So US has a, is the only owner of the metadata. Uh, but you have to care about, uh,
for emulation of a STIP, API for example, it's on you. Uh, another way you could do it is,
uh, what I call client driven emulated glacier in which, uh, your reverse the relationships. So there's
CTA. So, uh, so as three being as far as technology, the question comes, uh, naturally, why not use
that as a this buffer? Maybe CTA could be a client for it. I mean, the elephant in the room on this
question is the fact that the S3, uh, API, a glacier in particular does not provide you any mechanism
to work, to emulate, uh, migration to cold storage. So you cannot really, uh, ask the project,
the, uh, the file to go flying, for example, like it's done usually in the back of, uh, of an
S3 service. But you could emulate it. One, uh, where you could do it is, uh, for example, CTA can
write, or object metadata, uh, so you could mark an object as a flying, for example, through a tag.
And the lower scripts, uh, whenever it finds the tag as offline, it will reply in expected way.
It will emulate glacier by saying, you know, invalid, invalid object state, these objects
are flying. And then CTA is free to work with the objects. That's one other way to do it.
Another way would, would be out of band data control. So we said that S3, for example, doesn't provide
you a way, um, the S3 API, doesn't provide you a way to, uh, basically move the object to cold storage
as a client. But perhaps there are internal ways to do this, like out of band ways, uh, one way
could be to access liberado subjects directly to truncate and re-hydrate objects.
Another way could be to implement either non-standard S3 API or some tooling, for example,
through, uh, some other medium. These are just theoretical, uh, ideas. Well, uh, none of this is
implemented, uh, but this could be other ways in which this could be done. And of course,
pros and cons are very much implementation dependent. For example, the risk, if you access
liberado's object, the risk of exposing internal details, which are not guaranteed. I mean,
there are, in the documentation, but, uh, I think, uh, the right to change them. I mean,
there, there's nothing set in stone there. They could change in a new, uh, release, perhaps.
And yeah, in the end, there are a few solutions, which I won't spend, uh, much
words about. I included them for completeness. Uh, one of them is CTA, implement the S3 API.
I think there are enough S3 API providers, which are open source and, uh, in addition to being,
uh, reinventing the wheel, uh, it also introduces a new functional scope for CTA, which is this
buffers. CTA is a part of a full solution. It doesn't necessarily need to be the full, uh,
appliance here, but it can be the, uh, the back end of an appliance. And so it would introduce, uh,
there's a scope creep here. And same, if you wanted to theoretically create, um, like, bring the
disk buffer to CTA, but then, uh, create this standard interface to integrate with whatever, whatever
solution. Uh, uh, but here, again, new functional scope for CTA, new problems with, uh, are not
considering that at the moment. So the next step for us would be to test out, uh, some of these solutions
and see, uh, what works best. Uh, but I'm also interested if, uh, any of you has implemented this already
to hear about your thoughts, your pains and, uh, uh, your joys with your approach. Uh, so I'm
available, uh, for the rest of, uh, foster, uh, so if you want to talk, I'd be happy to, uh, just
quickly acknowledging some of my colleagues, uh, Michael Vladimir and Niels for providing some, I mean,
technical feedback and, uh, graphs as well as the Seth website and mermaid for the graphics.
And that would be all.
And, uh, I, uh, so it is an alternative to the tape rest, uh, API, uh, interfaces, uh, and
several questions. There is something, some checks, uh, on the checks, some of the files,
when there is a retrieval, uh, from the retrieval, the data and the checks, I'm saying,
on the metadata, uh, of the else. So, uh, so, uh, first question, uh, tape rest API is in
enough. Uh, sure. So the question, the first question is, uh, uh, uh, this is an alternative
to the tape rest API. So this is about Seth, no?
Oh, yeah, about, yeah. Is there a tape rest API?
It pressed the API, you know, the API protocol, the API used by WSIG for, uh, data, uh, for, uh,
for a drive, uh, data from data, uh, varsity gate for human, um, there is a, uh, WSIG standard
of the tape rest API to recall find from a, uh, okay. Okay. No, this is more, mostly about,
so that's, uh, basically, so S3 doesn't provide an interface, will Seth implement this interface.
So I don't know, I, I, I, I, I'm not part of the Seth process, uh, uh, uh, uh, project, but that
will be an interesting, uh, thing. So, uh, I, actually, I wasn't aware of this, uh, so thank you for that.
And the second question was about the checks summing, checking, when never a cloud object is
retrieved, right? So the object cannot change, uh, in that case, is what you mean.
So if the object changes on the second bucket, uh, then the first bucket will return an error.
Is this what you mean? Or if there is an error, when, uh, recalling the child from tape to
this cup, there is some checks about the test that they recall is done correctly, and all that
I, uh, read correctly from tape for, or assume that the reading is always successful from
tape to this. Ah, no, I, as far as I remember, so I'm not a CTA developer, but I think that CTA does it,
on its own.
Thank you.
All what is it said on tape, when you work on people, as we already did.
So CTA is the part, both, uh, sorry, um, so repeat the question. So does, uh,
um, there's only one copy of the file on tape, or there are more than one.
So CTA allows you to configure that. You can configure CTA storage classes, which, uh,
define how many copies of the data do, would you like?
Okay, so, uh, question is, if you were to retrieve a tape from a library, would you be able to take a
look at tape and figure out which files are lost?
Would you be able to work out from EOS, which files are on this type?
Ah, from EOS, which file are on this tape?
So this information is held, actually yes. So, um, basically, the, uh, so the, the, the way that it
works is a, but you want to need CTA. This is the thing. So EOS by itself doesn't, because the way that
CTA and the US, uh, basically agree on where a file is. So how can you track a file?
Basically, um, so on EOS, there is the concept of file path, right? And on CTA, there is no such concept.
Uh, so how does it happen? So CTA, whenever there is a file, we'll also try to figure out if,
if that file is already on tape. And how does it do that? It looks at, um, extended attributes of the object.
So, uh, if it's not there, uh, if there's no extended attribute with the archive ID,
then, uh, it will write, create a new one for it, and identify or internally to CTA.
It will write it as extended attributes. And from them on, that file on EOS would be trackable on tape.
But only through CTA. So CTA tracks where the archives ID are on which tape and where.
And EOS only cares about knowing the file path and the archive ID. So this is, uh, the, the chain.
Okay, so our time's up. If you have a question, please meet me later.
Thank you.