Hi everyone, first of all I want to thank first of all for having me, also the HPC
Deaf Room for having me and thanks to all of you for being here.
Today I want to talk partly, I want to present my talk which is partly cloudy with the
chance of sorrow, a virtualized approach to zastos from ECMWS, field database, I'm to be
a screamer, I'm a research software engineer at ECMWF and I'm over former cloud, back
end engineer working for ECMF for, on this day exactly two years and I'm also part of the
one-wheel-easy project to collaborate with colleagues from JSC, DKZ, MPI, Jarex, KT
and University of Cologne. At ECMWF I'm working mainly on the mass ecosystem,
mass is an abbreviation for a meteorological archive and retrieval systems storing all of our
focus data, so just a huge archive if you want to say it like this and I'm mainly working
on the field database which is abbreviated FDB, a caching layer on top of the actual archive
and I'm also working on interfaces to this FDB mainly the Python interface and the Zara interface
which we want to have a look at today. What is ECMWF? It's an intergovernmental organization
established in 1975, consists out of 23 member states and 12 cooperating states and we just recently
crossed the 500 head count number for employees. We have 24 seven operational services doing
operation in numerical weather prediction and having four forecasts today supporting national
weather services with coupled models and also business stone stream. On top of that we are research
institution and doing experiments to continue to improve our models, also doing refocus and climate
analysis on the right-hand side you can see the three sides and I'm stationed in one.
So, sorry, why are we actually talking about this out of the day? Well recently and recently
we saw a huge explosion in scientific data. We had ECMWF currently produced 360 terabytes
of forecast data a day and there is a projection for that to cross the petabyte threshold
in 2027. In the right-hand side you can see the image, what the individual lines are does
really matter currently but the blue one is basically the overall archive and you can see in 2025
we crossed the exoBite scale threshold for the entire archive. On top of that we're seeing
increasing pool of target users coming from all different domains for example social science and
geologists not only meteorological or climate users and of top of and as well we're seeing
more and more users rushing in from the Python side doing for example machine learning or AI.
Our normal and HPC use cases are mainly driven by the focus right so we're doing
numerically weather prediction workflows, writing and reading to parallel file systems and of course
also analysis of forecast in weather and climate and on also so if we take a look at what the
users actually are keen on to do with the data we've seen a shift in the recent years.
There are many hybrid workflows or combining HPC with actual cloud workflows or pipelines
and people are in general really interested in having cemented a semantic data storage in the cloud.
Also doing interactive analysis, explorative work or an electronic on large data set in general
we won't say people are really keen on having analysis ready cloud optimist data sets.
On the right hand side of broader example for such a use case this is the error explorer you can explore
planetological data and in this case the user was asking for a time series data of precipitation
in millimeter per month of process and in the HPC use cases we normally write the entire state of
the atmosphere for each individual simulation step of the model to our file systems and you can see
that the use case in the case of the request of the user is completely orthogonal to that
access pattern because he or she is requesting a time series right.
So the error question which initially arises is can we actually give flexible access to the data
and do that so that even non-domain people have a nice time.
So the goal of this talk is actually to show how bizarre because something which is quite
family or familiar and quite popular in the Python ecosystem is fitting to this HPC use case
and also how this fits together with the already implemented open source solution we have in place.
And we want to bridge the gap between the classic HPC approaches and more modern cloud-based solutions
right or if you want to take this diagram how do we actually fill in this question right there.
So to be able to do this we need to talk about the individual pieces of the chain right so
starting on the right hand side we're talking about in the FDB. The FDB is a domain specific
type forms storage object store it's strange act at print action has no explicit synchronization
does all this synchronization on a file system level and there's no MPI involved.
And general you can think of it as a key object store where the key is some meteorological
meta data and the values are just binary data can be grip files or ODB files.
And an example for such a key in this object store is written in the light left hand side
down below the slide and you can see this are just different meteorological variables
and values describing the actual data safe there. On the right hand side you can see the operational
setup we have different numeric method numerically weather prediction models running
which are outputting the data in the FDB and the FDB is then distributing this data over
different pack ends could be doused, saffronos or a master file system. And the models are
outputting the mass meta data and the data itself and this is typically done in a grip file format
so we need to talk about grip a bit. What is grip grip is an abbreviation for general
regularly distributed information in binary form this is a short one right and is in general
just during gridded fields really and the field is a scalar vector field could be temperature
or pressure and the left hand hand there's a picture depicting the normal grip field so this
is really for in that example two meter temperature on the surface of the of the globe but it can
also be like as I said pressure on different height levels. Grable standard doused by the WMO for
remedial logical data exchange it's a binary format compact so has certain capabilities in compression
and yeah it's efficient for large number of weather prediction data sets. It's optimized for
archival workflows or in general operational workflows it's really good at sequential access
it has a very minimal overhead for all the things we're doing in our operation pipelines
and it's self describing meaning that what I said on the last slide as well the meta data and the
actual values are stored in the same file right next to each other on the disk well there are
also some downsides for example it's a rather complex format you have some tables on top
describing the meta data and in case you want to read the grip file you need to have those
tables to be able to restore the meta data completely and also because it was initially designed
in 1985 this obviously predates cloud native use cases. The lower part of this slide you can see
that is grip file structure on disk there are different sections for the meta data and section
7 is typically where the data is stored all the other sections are certain subjects and sections
of the meta data so finally we also need to talk about Zato make the last piece of this chain what
is Zara actually it's two things really it's a library and it's a format but you could also say
it's an API and behavioral contract but if you're not in the chunk compress an end dimensional
data format to the user it looks somewhat like a non-play array although they introduced
chunking on top chunking is just a fancy way of saying instead of querying the entire array
all the time on querying subsections of the array and those subsections are then returned to
the user. The key features of Zara is that it's storage diagnostics so it doesn't really
meta or appear to the user where the actual source is located could be on a local file system
could be in S3 bucket it has a great language interoperability there are many different
language programming languages implementing features for the or implementing clients for the
Zara library chunking I already mentioned in general it's a hierarchical format mapping groups
or arrays and chunks to the corresponding file system objects right so groups are mapped to
directories arrays mapped to set of files and each chunk is one file on the file system.
All right so to answer the question how do we actually bridge the gap between grip and
Zara I want to talk about the local use case first so where the FDB is running locally and also
the user is doing only local requests so we came up with a solution for
here bridging this gap as follows the user first of all inserts this meta data we've seen earlier
and this can be arbitrary metadata really whatever data the user interested in and it's
handing that to this FDB layer and this knows exactly how to query sort of meta data about this
read about this request the user just query it and is asking the FDB for this meta data
the FDB is then returning this information and is returning a virtual view we call it to the
user this is just a Zara store but the Zara store doesn't contain any data yet so everything in
there is lazy there can be groups in arrays within the store really like arbitrary structure but
it's nothing is loaded it's just like a virtual description of what could be in there if we
condense this image now to this we now have our Zara store the user can interact with but we also
need to be able to query something from this so as soon as the user tries to access in array
I mentioned earlier already that then certain chunks get loaded right so what the user is doing
it's he or she is accessing the array and then the chunks underneath get get query from the Zara
group and array and because the Zara group array knows exactly which stores it's part of
the implementation of ours is then able to map this inquiry to a mouse request and sending this to the
FDB the FDB is then returning those bytes of the Zara for I'll back to the array and the chunk
gets actually delivered to the user this is the the high level explanation of what is going on
this two-step approach between first defining the actual view and then querying the data
from the actual database to give you an overview how this looks here the code is more or less readable
if not it's not a big deal because what I just wanted to show with that piece of code is
how concisely can define such a virtual store or such a virtual view there's actually in that
example a little bit more going on so there are two parts which we define at each part is a
you could be a request to some meteorological data and those are virtual arrays they are not really
existing in the Zara as I just described they are just the mapping which has to be done when
accessing the files from the FDB right and those are in that case concatenated together and the
initial request are four-dimensional but they are mapped with the access definitions to an actual
two-dimensional array in the Zara word so this is also shown on the diagram on the lower part of
the slide so there are two lazy views that they are concatenated together and they end up in Zara
array and the dimensions you can see right below this SARS-Zine the first dimension of this
actual mapped array is the data I'm dimension so we're the data and the time of the initial
method data is mapped together to one dimension and then the second dimension is the parameter
level list I mentioned where the same thing happened with the parameter the level list and the third
dimension although I just said that it's in two-dimensional array the third dimension is implicit
and it stores all the values of the group files we query from the FDB okay now we want to talk about
the remote case actually right because we want to bridge the gap between the HPC and the remote
user and Zara has a lot of capabilities in the regard already so if you have a user you probably
are aware that there is an HTTP store and you can just hand it a certain URL to a data file
somewhere located on a server and it's it's just able to open it and the users able to access the
content this was what is depicted on the left hand side with the aqua data set in our case because
we had to implement this two-step approach for first virtually defining the view and then
querying the data we needed to a slightly beefy implementation of the client and the server all right
so the client and the server has shown in this picture take care of this two-step approach
and they are communicating over a compressed HTTPS but once we're on the server side on the HPC
everything happens exactly like I just described for the local case so how would that look
like the first that would be the client triggers the creation of a virtual view the server then
creates this virtual view of the FTP data as described earlier and is returning the URL to the user
and or to the client and the client then is able to open the virtual view via an FLS spec store and
send the chunk request back to the server once the server gets the request it's growing the chunk
data from the FTP because it knows how to map the the request and sends the bytes once it has
it from the FTP back to the client and to the client it looks like the actual data was stored locally
and this is this is the entire the entire process so to summarize and what have we seen I think
it's the easiest ways to have a look at the diagram basically we were able to extract data from
the FTP directly while by our creating virtual views which can be concatenated as is not necessary
this is an optional step right but can be done by the user and then it's handed to the
user and really the the user only grows the data once it's sitting the certain chunk that's
everything I have so far thanks a lot for your attention and I want to thank the BMFT for sponsoring
the project and also my colleagues who collaborated on the topic with me and yeah I'm looking forward
to your questions yeah I don't know climate reasons and I mean so I'm wondering how
designing the place with net CDF files it's a way so if our scientists have our
connection of net CDF files do they have to convert them to a lot of data storage or is this
you more as a cache like the data is used on the fly and then the end so the question was
there are certain files which may be HDF net CDF and so on and whether they need to be converted to
ZAR right to be usable and so there is a certain library you can use which is called virtual
ZAR we have nothing to do with this this is part of the ZAR ecosystem right and this is basically
able to open net CDF HDF I think to some extent grip files which are not stored in the FTP just
on a random file system as well and display them as a ZAR very close to this virtual approach
we implemented but like a bit more general to with other file systems as well right so that would be
the option you have there without so there's no need for conversing this but you could use this library
okay so I actually brought as well the link collection oh yes sorry so the question was
whether the API is already available right so the local use cases already living in the FTP repository
here the client server thing I showed is under development but it's going to hit this also rather
thanks yeah
yeah that's a very good question actually and so the view is so the question was whether the
there's a state saved on the server right which persists the view and this is actually done
so currently we are just holding the viewer certain user created it hasht in memory and this is
reused once another user queries the same view right there are some some open questions that
regard for example if you have a lot of like overlap between re-quest what do you do but this is more
less open yeah so the question was whether the virtual is our state there's some whether ours
has the state I'm not entirely sure how virtual is work some of the hood right I'm not
affiliated with the project or anything um I think the two step approach is hidden in the virtual
is our stuff because there has to be a map in this mapping as well I'm not entirely sure
alright if there are no further questions thanks a lot