Hello everyone and welcome. My name is Bart and I'm here together with Igor. We're both
data science software engineers from all the other. And today we will be talking about
the journey of building and the architecture as well of Bobus Dev 4.0 Alpha. So I'll
also the lessons we learned along the way and what currently available in the alpha release
and what to be expected in the final release. But before we can go into that what even is
open-step. So open-step stands for open short term energy forecasting and with short term
energy forecasting we really mean forecasting up to hours or days at. So for example here you
have a time series or load of a point in this case in the grid. So historical measurements
with the red dotted line being right now and then open-step allows you to predict the future
basically based on these measurements but also all their predictors like weather forecast energy
prices and profiles and this way open-step will generate well actually multiple forecasts. So
not just a single forecast but also with an uncertainty bandwidth because predicting the future
is basically impossible so you need to have some kind of uncertainty bandwidth to cover your risk.
And a bit more in detail what is OPSF so it's not just a model that allows you to produce these
forecast. Actually OPSF is a model agnostic and specifically it's a Python package so it's a
framework that allows you to create these equality energy forecast because creating energy forecast
is more than just having a model putting in data and getting a forecast. It actually contains
many machine learning steps. For example start with data pre-processing such as input validation,
feature engineering, they need to train models and if you want a great forecast you need to use
that trained model. If evaluating forecast is also just as important as making them because
you want to know if they are good so you need to evaluate them on measurements of the past.
And finally sometimes you also need something like post-processing and OPSF provides pipelines
or workflows to allow you to easily do all of these steps without all the complexity that's
needed to do it. And I also need to tell a bit why do you need OPSF and you probably heard a lot
of this today already including these pictures. I think these are even outdated because these are
from half a year ago. I think the new ones are even more wet that what is basically shows is that
our grid is full and so OPSF can be used for many things but I'm here now going to explain one of
the use cases we have at all the other so congestion management. So as you can see here the grid is full
so what it means is that customers cannot be added to the grid and while we are of course
reinforcing the grid that takes time and it cannot really solve this waiting list we currently have.
So schools and factories they cannot be connected to the grid right now which has quite a societal
impact as well but something we can do is actually we can connect them but that results in overload
but we can do something about it because we have OPSF so we can forecast them the load of
on these points in the grid that now if these extra customers connected and that means that sometimes
you will have overload so here on the left you'll see load over time and OPSF can then predict in this
case two days ahead there will be a moment of a few hours where the load will exceed the limit
of the equipment that's at that point. What we then can do because we know it in advance we can
call a customer and ask them to basically reduce their energy consumption or generation
against some kind of conversation so in the end we prevent this peak moment from happening because
the customer reduces their energy activity basically and so this is the congestion management process
and so you really need these accurate forecasts of these peak moments so that's where we use OPSF
but actually over the years more and more use cases are host. For example transport forecast
grid loss forecasts with different things as a district eating which has nothing to do with
electricity at all so that's one of the use cases from the community so not not even
all the other and so as more and more use cases came to be for the short time energy forecasting
we were at some point where we wanted to further improve the flexibility of the current OPSF
to really allow these more these newer use cases to be well even better supported than they're
currently are so this brought us to the redesign of OPSF we were talking about today
so we had some goals for this so we really want to keep focusing on it having to be a
machine learning library so for creating forecast and for the evaluation while also providing these
presets for common use cases like I mentioned and then OPSF also targets different target audiences
so for example the researchers and experimentation phase so those are mostly working with in
Jupiter notebooks with data sets of mostly focusing on metrics and and floating functionality
so that's already well supported within the current OPSF that we've been OPSF before we want to
improve this by also making the back testing framework more realistic second use user group for
OPSF would be small skill deployment so let's say our single data science team
and you need to create forecast for your company so from end to end so from retrieving measurements
and forecast a rather forecast that do storing the forecast made by OPSF so by providing
reference implementations or example implementations we want to support those use cases
which is already one of the strong points of the current OPSF version so we want to keep doing that
with something that's really new with OPSF before one of the main goals of this new redesign
is to target the enterprise integration so in larger companies like we currently now are as
as early on there we have quite a complex software landscape so many custom APIs but also there
can be policies on what OPSF where you can or cannot use and by having OPSF flexible or even
more flexible and modular the goal is to allow OPSF to be really easy integratable in search systems
so that are basically the goals of this major redesign and so that brings quite some challenges as
well so you can imagine so to have this flexibility and these different use cases to support
them you really need an opinionate of design so not from a single use case such as congestion management
you but you really need to be really an opinionate in that something else we found is that
data availability constraints is something that also happens quite a bit so for example measurements
can come in delayed or rather forecast that is used in the forecasting process they can have
delays so I think native support within OPSF for this is really huge when for complexity
but also for back testing situations where you can then take into account these data delays
so that's also one of the key points both the redesign that we're working on
flexibility kind of makes sense if you want to be more flexible and target also enterprise integration
and all of this of course we've out compromises on performance so not just the model forecasting
quality but also the performance of running OPSF itself in case you need to scale up and when
multiple forecast a day and all of this of course you do not want to reinvent the wheel because
well this is quite a huge redesign otherwise you will be there forever and they're very
great projects out there that we can look into so that's also where we started so with design
and as this OPSF is really a community project we did this together with the community
that many meetings planning sessions brainstorms on what are the current pain points strong
back and we improved what do we need for a proper redesign and in the end we had like this
detailed design proposal that we shared with the community had many feedback runs on and for
for that we used slack and confidence there's a great great places to
well get feedback from the community as well just like the community meetings
and in the end well quite a nice design but part of it was also looking into
existing projects right because we do not want to reinvent the wheel so that's something
I will pass over to Egor thank you so yeah when designing the project we wanted to not only
look at the community but also at what is out there and investigate existing forecasting
libraries and how they fare on load forecasting and yeah or use cases and to kind of invent
advice what's our different points but what do they do well what what do they miss what kind of gap
can open stuff fill in this space so we get it best practices things that we shoot copy and we
showed not reinvent ourselves but also gather things that yeah all every needs to do well
to be an addition for the energy forecasting space and taking all of this into consideration this
is the architecture we have come up with it contains of multiple modules which are mostly self
contained and yeah are really useful to keep the frame up modular because some teams may do
forecasting but some teams may be focused on evaluation or actually only consuming the forecasts
that's bring this me to the first core component of our library which is core which contains all the
data types interfaces and everything that is needed for what comes after this way everyone has
a defined set of interfaces they can use to extend open stuff similarly yeah we multiply
exceptions and testing utilities for the next libraries and then this is all used by the libraries such
as open stuff models which is really the core of open stuff it has the forecasting models but also
the data processing pipelines and the main specific transformations we need for energy forecasting
similarly we decided to to focus really on expandability and have a way to explain the actual
forecast you're getting and yeah presets so people can start using it quickly and audio modules
include major learning which is based on a much stronger way of forecasting using modern models
and assembling it and then we have open stuff beam which stands for a back testing evaluation
analysis and metrics and this is a yeah very important library for us it's then from an internal
project and to basically answer a question are the changes that I'm making in the model are
they are significant and do the Gaussian improvements are regression that's a way have it there and
we have made it a core part of open stuff and finally a foundation of models which we want to
talk too much about because it's still a broken progress but the main idea is to have pre-trained
models that you can use for forecasting your energy data so once we got our design ready and some
many show drafts on how we want to do this we started with a public back look in our
open stuff get a project we have created a bunch of stories and tasks that we wanted to do to get
open stuff for ready we defined a few milestones and and this is in this way yeah people can follow
us as how we build the library and yeah see where we are at and what is planned similarly we have
focused on a few more tasks that are mostly self-contained a small to market them as good for
stations and pre-refinement to get new contributor started these tasks currently limited in context
and yeah really nice for people wanting to get into the space and to on top of that we are
host co-coding sessions every eight weeks where we and our community members come together and
work together on tasks it could be designed but it could be also the tasks that such as good
first issues and things that you want to implement it or brainstorming. Support the first iteration
so once we got our design back look in the natural next thing is to start developing but
as part of this development process we wanted to get a really good setup on to make it easy for
users to contribute but also to quickly iterate on for building a better open stuff. A few tools that
we use with that is for type safety which really helps us with refactors and changing takes we get
immediate feedback on what you're doing similarly documentation tests. Similarly the commutation tests
which well we're building we're also waiting the commutation and some plans change some
that's way by incorporating it as part of our unit tests we are also keeping the computation up to date
which is up a good feedback for us and then you fear of SIGCD which are really useful
development tools we have nowadays really fast and really support our Python development
finally what I wanted to talk about a lot what I wanted to talk about is the benchmarking which is
really a big part of our regression testing strategy. Our benchmarking as I said earlier
is part of open stuff beam which is a package that's used for evaluating forecasts but to
validate the actual forecast we also need data and that's why we have open source Leander 2021
energy forecasting benchmark which includes 50 plus different energy signals ranging from
solar parks, wind parks, transformers, MP feeders and much more. It gives us a good sample on
different use cases for energy forecasting and allows us to test our models really thoroughly
and this is yeah really out of the box Renable you can clone the repo and do the test yourself
what we also did is we started using open stuff really early in our development we in our team
also really focused on improving our forecast for congestion management and by using really early
version of open stuff we were able to get early feedback see what we're doing is the writing
and refine but also find bricks early. So where are we at now we have achieved feature parity with
open stuff E3 which was the major milestone for us and which has become our alpha release which contains
the full forecasting module and evaluation framework. Overall in this case means that
we still reserve some rights to make some non-trivial changes but we are very confident
in the in its current capabilities and in fact we are already using it at the Leander
in production. In our team we make more than 10 for more than 10,000 different locations across
the grid and it's yeah steadily growing. So what are the lessons learnt or yeah from this
development until now one thing is make it really safe to experiment yeah by creating the regression
testing it makes really easy for our researchers to try different ideas see what works see what
doesn't and get immediate feedback because yeah energy forecasting complex and sometimes we need to
get a lot of data to get started but also also you may also need to wait a few days for the data
or the forecast to come in by creating our benchmark for simulating it we are festric in this process
in fact there's been also a yeah an experiment of ours to see if you could improve this process
yeah prepare for the contributors we have seen quite some good contributions for community
into open stuff and open stuff for and yeah it's really helps people to get started early and
look to look at some issues that are really good ways to get into the forecasting design for
my ability from the one we found that really useful because our library do quite a lot of things
because forecasting is the very info of process and this way it allowed us to focus and structure
our code in a way that yeah could be the early test it and yeah is something as achievable
and finally real world testing we need to know what are you building for and actually use it for
it to see if it works or next steps for our library we are currently in alpha release and we are
yeah steadily building towards a stable release it's currently a bit it's working and it
but it's rough around the edges for the documentation so our goal is to get really the
well documented website started with good getting their started guides for both researches
but also for people who want to run it in production and work with it which brings
it to the next point development deployment guides and deployment examples yeah and my
lobes is quite complicated thing and if you can get some examples for users to deploy it helps
of a lot and finally the new things made to learning module is steady on its way and yeah we are
clicked really work on it and for days you know models is also yeah on its way you bring me to the
final slides yeah thanks Igor so if you before you go into the questions because this was
our presentation that if you want to know more about it over staff we have quite some
quite some links to get to start it or have been more information so we have like a slack for
communication to the teams for to the community we have an energy project page to get to
pray it's also the hooking phase data set that that Igor mentioned so if you want to actually
start using OBSF but you don't have data but still want to see what happens we have this this
public benchmark data set we've already screwed within OBSF to run it so that you can really
experiment for yourself and see what it does and finally I've to mention we we have these nice
we have a really nice community so we have these for weekly community meetings as well
including coding sessions I think like every eight weeks or so where we sit together with the
community and think about what our features we want to add and then do it together so if you
find that interesting it's all open to join so here are the links to getting contact with us
and with that thank you for listening and let's go to the questions
yes
I'm not a user of open steps so it makes very much a big question but does it involve some
read topology or do you use a need the times three on nodes that may be aware of
yes so the question was do those OBSF use grid topology is that required
OBSF currently works is it's really point point based so let's say you have
you have for example medium voltage routing onto forecast each of those points
the ideas that you have measurements on each of those points and then forecast them but if you
want to include grid topology as well you can also use the power grid model in combination
I think there's also a paper published already about this about this use case where you can
use both the topology in combination with OBSF to even further improve forecast quality so you can use both basically
yes
sorry can you repeat
yeah do we have other DSOs that are involved within OBSF so we have RTE and RTE international so they're
not DSOs there but like DSOs themselves not yet so it's OBSF really as the DSO but we also have
for example Siegel in Sweden we were really involved as well but they are mostly like a
consultancy helping local DSOs or even for district eating but as for the DSO party where
that's only under the only one's currently is a question is what accuracy do we achieve with the
forecasting that really depends on what you're forecasting so it's a tough question to answer
because it also depends on what your use case is because if your use case is congestion management
forecasting then you really only interest it in those peak moments so what then how which of
forecasting quality is during nighttime doesn't really matter so then you want to use different
metrics so then like RME or those types of accuracy metrics make less sense so it depends on your
own your use case and I would say it's pretty good but maybe it's best to see for yourself if you
also run the benchmark with the working phase data set they can with with OBSF being you get
all these different kind of plots for different kinds of metrics so they can really get an in-depth
insight into how well these these forecasts perform but of course it also it's not just also the
models it's also like the input data that you choose so if you are input data is it's a bad quality
then your forecast will also be so it's always a different difficult question to to answer
yes how far ahead did you consider short-term we would consider well up to seven days because when
you extended 37 days as your weather forecast either will not be available in for example 15 minute
resolution which is typical but what is used within this electricity grid it also here energy forecast
or your weather forecast will be of such that quality that you cannot really foresee a solar peak
anymore because you will not know if it will be cloudy or not so that's the when we talk about short-term
energy forecasting it it is hours until days with a max of around seven days yes
yeah so the question is what makes open stuff specific or like unique compared to general
forecasting is that during this and quickly yeah okay what makes it specific
open stuff for for energy forecasting compared to all the types of forecasting so within open stuff
there's also a lot of domain knowledge included in for example within the feature engineering part
which is quite important so for example when you provide a weather forecast
open stuff will if you want it can automatically derive for example based on solar radiation
and the the temperature if you weather forecast what the typical pp panel would generate and these
type of feature engineering will you make it get the edge on forecasting because most of the
currently the models within the open stuff they are based on classical machine learning so
we're relatively simple models like decision trees linear regression so that's really what gives
it the edge of course when you add the the the the deep learning things will change but that's
for for another day thank you for your questions