Hi everyone, good afternoon. My name is Pratikshar and I am a Geomatic
Engineer from the Open Year Team. Today I am joined by Michael Lee Mell, who is one of
the developers or the face behind Open Year. And the main topic that we are here
trying to cover within Open Year is about the reusability concept that we support for
the algorithms that I developed using Open Year. But before that I want to quickly
have a recap because last year as well I got the opportunity to be here. That was the first
time when I was here in fourth time, as well as it was our first time presenting CDSC in
first step. And Open Year is one of these open source components that is supported in the
CDSC which is Copernicus data space ecosystem. Quickly giving a recap for those who are
still not aware or who completely have no idea what a CDSC is. So CDSC is a user centric platform
where we provide satellite data sets all at one single point with the focus on the Copernicus
data sets. Copernicus is a European missions for capturing satellite data sets that would be used
for several applications related to earth observation. And Open Year is one of the tool that is
provided also within this ecosystem for not just accessing the data sets that in the ecosystem,
but also process it in cloud environment. So that was what we covered last year, but this year
we thought why not let's go a bit more into technicalities of what is Open Year and how we say
that okay Open Year solves this this problem. But how does it does behind in behind the scene
or how does it suppose the concept of fair that we always try to present with Open Year.
To start with a classical example of can you guess where it could be?
Yeah there was voice Brussels to be more specific I think the dark image that you see below there
that somewhere next to us if I'm correct. And it is actually a beautiful snowy image that we had
in the second week of January this year. So this is the Brussels itself.
But I wanted to show this picture saying that it's a pretty picture to look at yes a snowy
picture of Brussels but along with that it also gives us insights on looking about this information
on what are the landscapes what could be the additional information we can gather from this
kind of data sets and these are a couple of other pretty images I would say but these talk
much more than just visualization. In the right you can see a image that describes about the
land surface temperature and to the left you can see couple of ships that are detected using
one of these Sentinel data sets. So these are some examples of information that we can gather from
this satellite data sets along with that we can also study several earth observation earth
related phenomenon disasters and also perform several applications related to this task.
So here you can see hurricane followed by wildfire and there are gas flames so these are all
detected with satellite data sets. How able they are often challenges when working with this
kind of data set that they are terrified of data set that are acquired every day and in order to
work with them to make a useful information out of them it could be quite troublesome to download
all these data set in your own environment and to process it furthermore the data set that I
showed you they were like Sentinel 1 Sentinel 2 these are just named to couple of sensors that
capture the data set and they can provide you data in different formats. So the algorithm that you
will be developing for for example net cityf could be very much different to the other format.
So you will every time have to think about how do I change my working method in order to work with
this kind of data sets as well as where are they stored. With cityf now it has been simplified
that we try to bring all the data set at one location so that you can directly access the data set
from in there. But to address data formats that was still quite a challenge I would say.
That is what open you try to answer but in addition to that I would like to show what where
the traditional method that we as a remote sensing engineers or sorry remote sensing data
engineers try to do with the data sets previously. So for example I wanted to identify what was
the change detection in this location so that time I used to download the data set from the previous
year for how much data set was available. I just downloaded it in my location and I want to be
specific with Brussels so I just do additional processing and all those things and then I again
do this same step for the next year and I try to find the difference between those two data set
by finally focusing on developing my algorithm which is often tricky if I want to do it again in
the same calculation in a different location. So that didn't support the concept of reusability
I would say or reproducibly of what I wanted to do in a difference scenario. Apart from that it was also
a challenge I think someone else also addressed this challenge earlier in this room that I prepared
the data set but it is very tricky on how do I share it with the other or within my team as well
how do I easily share it. Do I download it? Copy it in a pen drive and shape please can you
look over this data set in your own environment so that was another tricky that tricky question
that open you tried to answer because following this traditional method it was often slow
it was not just hard way expensive but also computationally expensive in doing the same thing as I
mentioned earlier reproducing of the output was quite tricky part in the whole algorithm development
and more over scaling up of the algorithm. I managed to do the same work for the Brussels reason
how do I take it to a continental label or to a global label. So that was a question that always
lied on the table when working with the more sensing data set and as I have been always matching
okay open you is the one that tried to solve it me as a remote sensing data engineer I always
face the problem that I mentioned earlier and open you solved it but how to answer that question
how ML will be helping me out because he is one of the developers of open you who has been
intensely working with improving the algorithm developing various features and processes that could
be used by the remote sensing community so ML is too. Thank you Platicia to explain the challenges
and also showing the nice images of Copernicus so hello my name is Emil Sonofeld I'm a developer
for an open you one of the developers and open you is actually an open standard that's useful
to work on a geospatial data so the eostains for a herbs observation and the open
instance because it's open source and also works a lot with freely available data which is always
nice open you is an open source API that provides standardized access to earth observation data
and simplifies the deployment of processing workflows of scalable cost efficient services
so what does it mean in practice we have a few examples where open you are learning and I'll
cover the the most recent example it's here on cdz and cdz there is a data center very large
with multiple petabytes of earth observation data on it and often we want to do competitions
on this data to compute road comfort to compute how much mice is growing everywhere but this
calculations they are sold or large and work on so much data that is quite difficult to write
efficient algorithms on them so what's open you'll help is to take a very large task good
into into pieces process them in parallel on different computers in the same data center where
the data is also residing and then once everything is being calculated the results are stitched
back together in a very large stack at a look for example and put on the same data center which is
very yeah the the latest way possible to store your results in a very efficient manner
so we have a few examples where open you is being used so here at at a left we have a calculation
don't on a whole africa which is a very large calculation continental scale and this one
took only 25 hours because of the shortcuts we can take by doing the calculations in the same data
we have an under a larger example here if we have growth cereal which is calculated on the whole
road on a to begin with it's calculated on a 10 meter resolution and here it's then down samples
to 0.004 degrees with still very large and it gives an insight about how much cereal is growing
everywhere around the world it can be very interesting for food security among many other things
those calculations are very very heavy sometimes takes thousands of euros just in electricity to
to calculate and can be done very efficient with open you the actual annual median composites
of Sentinel 1 Sentinel 1 is a radar satellite of Copernicus and sometimes it's handy to have already
a pre-made median of a whole time series maybe like maybe this is a medium of a year
the annual meaning and then it's easy to continue working on this data so this is a
calculation that open you does and the result of it is also very easily usable in a stack at a
low so either a little bit of points which is covered also very important about OpenIO is
it's an open standard and it helps for the code reusability we'll cover it later about a U.D. piece
the user defined processes it's a way to easily share processes that you already made like algorithms
you package them in a way to make it easily usable for for other users to be working with
also what is very important is reusability because if you follow the open source and the open
AOS standard the way you need to program makes it easier to make sure that someone else can
rerun it to because if someone is running calculations on their own laptop they might be using
Docker and good build files and everything but if you run on your own laptop there is often an issue
about reproducibility it might not work on someone else's laptop and so one of the advantages you have
about running on a data center is it's not your own laptop so if it could work there someone else can
also run it on the same data center and we also have a good machine learning support of the
classifications need a machine learning and we're quite smooth with that so OpenIO is this open
standard I just talked about the example in a CDAZ the Copernicus data space ecosystem
we are running that on a very large data center but before we were running there we also had OpenIO
running in a interscope which is in Belgium it uses more Belgium related data and also
we cache also some European satellite imagery in the Terrascope for the scope of Belgium and
Europe but not for the whole world and so it's very easy for Belgium scientists to do calculations
about agriculture and what's in the Terrascope and then we also have an OpenIO platform which is
kind of a way to distribute tasks to the relevant spaces there might be a way that you connect
through OpenIO to the OpenIO platform and then you can load two data sets for example Sentinel one
and Sentinel one median data set and it might be delegating it's two different beckons
seamlessly so that you don't even need to realize where your data is is living but it will do it
effectively for you and a few other examples where we can have a rapid of OpenIO to access
different locations Sentinel hope we are moving away from that for the moment but there is many other
connectors so yeah how does it work so this was more like a corporate view of the OpenIO
but more or less a technical how do you work with it we often use a concept of data cubes
data cubes are an dimensional arrays and they often have dimensions like the
X dimension of why dimension of very important also the bands like red green blue but also
infrared and ultra are many different kinds of bands and there is also the time dimension of them
and so it makes a lot of data and it's a lot of different dimensions sometimes you don't have a
time dimension sometimes you do sometimes you have a band dimension sometimes it's single band
so it's different many different kinds what we provide is an abstraction layer that the data
stored on disk and the data you work with we will load it as a nice yes possible
the data on disk might be in a stack catalog it might use open search it might be in tips in net CDF
GPG 2000 it might be in any kind of format or users don't need to worry about they just get
a data cube full of pixels and how do we work with those data cubes here at the right we see an
example of a processing pipeline or a micolitor process graph of OpenIO where it will be loading
Sentinel 2 data it will be loading an external stack catalog and then it will work on those dimensions
call some processes on it take the averages and then at the end it can save the results and here it's
chosen to save the results as a CSV but those process graphs they can get really complex if someone
wants to use machine learning people says the data are clean things open remove clouds
and so on and we have a very nice tutorial it's written in very small that you
will collect it's a plenty of pictures like this to show all the different kinds of operations
but a little bit more about the core how does the code look like we often use Python to interact
with with OpenIO but you can also use R or Julia but in the examples we're going to use Python
so first of you need to connect to the to the backend so here there is an openIO data space
Copernicus it's openIO running on the Copernicus server somewhere and so you connect with
then you can load to the data you see how we're going to work with Sentinel 1 the rate of
satellite we use a spatial extent a temporal extent and the bands we specify what we want to load
then we do some computations with it for example here you just want to have the first timestamp
of the data and with this discord rest it's a very simple loading operation
but we can also do way more complex things on the data we can also run some Python code on it
to process it to do machine learning and then openIO go
terrorizes Python automatically so the Python script will get us inputs and x array
and then as outputs you have to also return x array but this little snippet of code will be run
at a hundred computers at the same time giving a very efficient processing
and then when you're done with specifying all your processing then you're going to actually
execute the task and then the calculations will start and he'd also specify the output file this
is an NC file so it's a netcbf file but you can specify to do this on CSV or whatever is a
relevant so well that was more of the code examples a more of a technical stack how does it look like
so we've openIO openIO can load different stack catalogs amongst other kind of data those
stack catalogs often are stored on as three buckets as three buckets were like kind of FTP servers
but then like the modern more performance version they often use cloud optimized geotips
but might also be using many other kind of formats just as an example openIO will then also
meet a few standard processes that you are running on the data you can run some Python on the data
and then openIO will cut it into pieces run it on spark to have it working distributed and this
spark is often running on Kubernetes which will distribute all the compute making new servers
that will be running your commands and retrying if something fails if the stack catalog was offline
for a minute then spark will retry the task till it's working so it gives some stability in there
so voila so we want to use it to only experience implementation and then the deployment of maintenance
the scaling they should all be not aware of it if there is a server going down we will make sure
to retry or use another server to work on if there is a stack catalog changing their way they strip
shorter data the way we will load a stack catalog we will adapt it fixed on bugs that the user can
still use any stack catalogs they want which is which we often need to do there is a lot of
a lot of collections in this world a lot of satellites that provide information there are new
getting launched Sentinel three has some new ones Sentinel 2 Sentinel 1 they're the keep on deploying
and we keep on supporting them all for a seamless experience and then we have an example
with even more code also connecting to the same backend if you can play this awesome
so here there is a GP that notebook and this GP that notebook is connecting also to
to the backend it loads a bunch of connections you can see a little piece of Python that will be
running on the data the job has been running half an hour you can see the larger process graph
and then there is a get results this is all the data sources being used to do this processing
so it's already quite a lot and then at the end we have the calculation so here you see the heat
yeah heat waves in Europe and it's also very nice that we could compute this in like half an
hour for a continental scale always a nice now you also have some visualization tools baked in
if you have your data cube out of your results and you want to show it we have some nice
usability features that work pretty good with the Jupyter notebooks
and see if the clicker works on this one all right so a moment ago we were
talking about making the processes more real you're really usable sometimes someone writes a
very nice algorithm to detect if there are potatoes growing somewhere to detect if there are
for us to be burned or not and those algorithms they take some time to make and it's nice to share
them afterwards so there is a place for that there is the algorithm Plaza and that is a bunch of calculations
to calculate for example a cropped serve somewhere or yeah the NDVI a very big classic in
earth observation and all those algorithms are also usable for other users and so what they can do
is they can specify the spatial temporal extent and do a quick calculation on the area they want to
if they want to see like how are the forest doing next to my house you can take the NDVI for example
and calculate it next to your house because not everything is already calculated rolled white
because it's quite heavy you can do some calculation on demand and we have the same also
oh maybe a nice example the real serial temporary crop extent is calculated everywhere that is
yeah seeing you're being produced it's a very high resolution and it's pretty nice you can
run it also on Brussels and then you see that there is a little bit of agriculture still happening
in Brussels for example you can see it with those mobs we also have a place where we can upload
algorithms that are more approaching based so this is apex it's also a catalog of algorithms
and those algorithms they are more like that we developed and we're like proud of it and we want to
share them with a road but also some other projects when they had done a lot of effort in their
algorithms and they can put it on here you also have the advantage that we will be monitoring them
and if something breaks if a stack catalog goes offline if our code changes some
subtilities about how processes are being run we have a bunch of tests running on them all the time
and if something goes wrong something changes we will catch it and we will fix it for you
or notify you like hey you're relying on an external stack catalog and you kind of put it offline
it will not be working anymore anything we can do try to we try to make this work as smooth as possible
so we're working a lot on my open deal by Fito we toys a Flamengo Institute for research
but there are many people also developing on open deal we have yeah contributors from all around the road
but also some people they implemented that own backend for the open deal so we have for example
open craft they they like to use our a little bit more and then they have an our backend
so that's awesome we love it when other people also implement the same standard and then
recently a development sheet made a backend that's also working with a tea Tyler I don't know if
it's correctly but it's very performance they also recently showed a demo where you could
change your process graph and the calculations you want to do and in real time it would update
the output of it in your browser window so they also use a good thing as technologies like
our web assembly I'll hit our web assembly I'm not saying they calculate only on a zoom level in your
browser if you're a zoomed out quite a lot they won't be calculating everything up pixel they just
calculate the zoomed out pixel we're very nice implementation but it's still in a in progress
another piece of connectivity with open source cookies very known platform we have a cookies
plegin and here in you can see and explore some of the layers that are available from open
you so you can zoom around in the cookies look around and then it will download the layer where
you're on the location you are looking at so nice way to explore you can also do this online
but if you have like complicated things in cookies with georgisans and some geotips that you're
visualizing it's nice to also be able to visualize a layer in the same setting
about the open source part we are available on hitup you can deploy your own open
your back and somewhere everything is available we have like 68 contributors which is nice
yeah I invite you to take a look see if there is something you can work with
and yeah we also have a forum so if you have any questions we are kind of obliged to
answer them as quick as we can and we love having questions because it's it shows the usage
and it shows what the users are working on so you can search for the forums I didn't put
the URL here but if you search an open your forum you won't get on it quite easily
and it's fairly active so far open your is built by community what will you build with open
your folder and questions are there any constraints on how much data we can use I mean you said
this is a lot of resources to do you care or do you just knock yourself out yeah there are
so limits so when you when you start processing you get 1000 credits and when you're computing
with them you start using them and you get 1000 credits a month and for personal use you don't
touch this this limit quite fast but if you want to process whole Europe you will get to this
limit and then it's a it's a paid service if you have a project and you need to process the whole
world and it will be very heavy you can contact the user network of resources the European space
agency has a lot of money they want to invest in such projects and you can apply for them and then
you can do your whole project with it so there are some limits and need to so you
listed two implementations of the standard like yours and the one by deficit are there any other
implementations of open you are there or for implementations yeah there are plenty of them
open here sometimes rocks against other earth observation platforms that already exist and then they
provide kind of a wrapper so that you can speak to their backends in the open you away so then it's
yeah so the game the aim is that one standard could fit them all so yeah there was one for Sentinel
for Earth and for Google Earth's engine also one we did have access yeah at the moment we
operationally support a company case data space ecosystem and terrace scope and an open you
plan with Google Earth Engine as I mentioned we can access the data sets that's there but we cannot
do any processing on them due to their licensing reason yeah but they is like if you have your own data
set and you deploy it as stack then you can directly use in any of the backends so that's possible
but it's also might at dusk a tool that works with x-ray on distributed weight also has an
opening implementation and first time I still open you I made me think of geodpp from the
jrc I don't know if this is something you're heard of and I was wondering whether there were
sort of converging and trying to adopt at least the open you expect for their own services is there
work being done there I'm not sure I think that is yeah indeed indeed there is some similar
API for Earth observation processing but yeah open you and that one is not related this one is completely
different and completely open source and as a part of the cda's reconciliation but also as
you promotion jrc one is slightly different than this one but in the concept why it's somewhat similar
you the examples you showed were mostly like the typical raster examples sent now
and there are also capabilities to work with vector geometries in the platform yeah it's also possible
to load a vector data but often the vector data is used for for masking for example or if you
want to calculate the NDVI on different fields and you want just one value per field you can use
the raster data we also have some tooling to convert raster images to vector images so yeah we do have
a little factor support also hi thanks for the presentation my question is does the Python API
allows for client applications to exert some sort of control in terms of the scaling the
computational usage behind the API if not does it provide feedback in terms of what was used
at each execution thank you very good question the technical question is if you run
Python code in open you is that any way to specify how to be scaling we do have some tooling for
it because if we want to run a user code it might be that user code is very heavy or very light
we cannot read the code and know so we provide some tooling for the user to specify how much
RAM will be used in the processing and then it can allocate computers with an of RAM to
make sure it works it's a bit more expensive but if you need it and then also you can the user can
specify in how small of a tiles it will cut the data and then smaller tiles allow you for doing
more compute per tile if you have an expensive machine learning model you might want to work
on tiles of 128 on 128 if you're doing nothing very heavy of processing you might work with tiles
that are 512 on 512.
Let me have a question
絕絕絕絲.