Tolling and how to use them, and that might be overwhelming for users and users and people
new to that stuff, Ryan will talk about how that might be made easier for the community.
Thank you.
Yeah, there's been a lot of adfishing, it feels like doing the presentation in the morning
is the right one to do, but there's the recordings I suppose as well.
So you've heard a lot about the variety of tools today, I think there was.
Acid update, EOS, there was, they won by TNO, SN, as pipes, there was very good, there's
variety of tools that you've heard today, and maybe about five or six different tools,
and they probably constitute on the order of about 200 open-source energy system modeling tools,
being really, really specific here.
These are tools for infrastructure planning and operation, open-source, that we know about.
And it means that it's a real big problem to keep track of the tools that are available,
unless you come to every single possible conference for people to talk about these things,
and maybe you'll pick up one or two each time, or you read review papers,
because actually things often come from the academic literature, or academic community,
these things often developed in academia.
And so you could go back, you could read a review, pretty much every single year,
there will be a review about the latest developments in open-source, or just general energy
system modeling, maybe even in 2022, there was about five or six reviews that came out
in the same journal, that what are the latest developments in energy system modeling and the
tools available, and it's effectively a losing battle, because you read these, you might get
the latest developments in the tools that they know are available, they might grab about 100
maximum of the 200 of there, and they are also probably about two or three years old by the time
you read this paper, right? So they've started their review as a PhD student, probably,
they've taken some time doing the review, they actually literally, the academic review process,
the peer review process, and then you read it a few years later, and inevitably they got a lot of
things wrong. So I've been a lead developer of one of these 200 tools for about nine years,
and I can tell you, every time I read one of these, they've always got something wrong
about what my tool can do, because they are reading it from an external perspective,
and maybe that's a sign of me or my documentation, but it's true for a lot of the other tools
as well that I know about. So this is where it comes in a project that we've been undertaking at the
open-energy transition with support from breakthrough energy, grid initiative, to map all of these
tools that are available in a fashion that will stay up to date continuously. So what you see here
is a snapshot of the latest update, it's a monthly update we get, so this is the January update
in terms of a snapshot of the top sort of 15 or so tools of these 210 that we've got in the inventory.
I can show you, I hope, what that looks like in a bit more detail here, so this is,
this is it live, openmod-tracker.org, and we've got a couple of caveats at the start,
so I want to be very clear, we believe we're catching most of the open source tools that are
out there, we're probably aren't catching them all, and don't blame us for the data associated
with your tool, right? Well, I'll tell you in a minute, we get most of this data from elsewhere.
So you've got the table here, a couple of different pieces of information that might be useful
to you when you come and look at it, you've got the link to the source code, so all of these
have to have openly available, publicly available source code, you can get a link to the documentation,
and we have a link to all the 60 or so tools documentation, so this is a really good resource,
if you just want to find out how to use a tool, I think we do a better job than the
readmees of most of the repositories of telling you where their documentation is, so check that out,
and then a variety of metrics associated with the source code repositories for these tools,
so things like when was it created, when was it most recently updated, how many stars does it have?
This is ordered by stars at the moment. Number of contributors and the development distribution
score that I believe was developed in open sustained tech, and both of those really are sort of a
bus factor, it's trying to tell you, you know, if there's one or two developers and a really,
really low DDS, it's telling you that if they go out of action, they choose to leave the industry,
that entire project will probably die with them, so you want lots of contributors, you want a
high DDS, you want a lot of force, you want people interacting with it in lots of different ways,
and you want to know how often it's downloaded, and that's a difficult one because we can only
get that for verifiable downloads, things that are indexed on package indexes, usually price and
packages, is what that's possible for, and a couple of other data sets as well. You can do things
like filter, you can say, well, I only want to see ones that have documentation, maybe ones that
are at least, you know, five or six years old, I want to have a bit of pedigree, maybe you want
ones that have a certain number of stars associated with them, so you can do a variety of filtering
actions to take you through and really filter down on the tools that might be of interest to you,
for whatever project you might be doing, whether that's as a user, potential user for a tool,
or potential developers, and be joining the development community for a tool, there's a lot of possibilities
there. You can also deep dive into a few other things, so we've tried to capture all of the
users of these tools across GitHub and GitLab, if they're on those ones, and try to classify them,
to get an impression of the types of users of these tools, the people are interacting with
these repositories. We don't do that quite a job, right? So about 65% of the users on GitHub
who interacted with these tools, of the 50,000 or so that we're picking up, that's 16,000, and we
don't get about 65% of them. That's because it's just that GitHub username and no other
information, and we can't hate to do any better than that. But of those who do interact, we get a
lot of people from the academic community, there's a little small few if you go through the actual
thing you can have a look at any more detail, academic community, research organizations,
professionals, and industry, they're all interacting in various ways. We can look at which
organizations they are, NREL now, the national lab of the Rockies I think is a big one. We
open energy transition and also a big one, a variety of other ones, so you've got things like
Chinese universities or US universities. You can look at this on a two levels, that was for all tools,
I'm just going to look at one of them here, maybe very good isn't here, that would get a very good.
So this is really picking on you, but you come to mind, and we can see various organizations who
got involved, there's countries where the users get involved, so you can see the international
sort of community that might be involved with it. You can maybe look at another one on side,
we have pipes afterwards, so we can add that in, we can see across those two tools,
sort of the global reach in terms of the kinds of users are getting involved, at least the ones
we know about, about 35%. And we can look at the project development metrics, so we can say
for these projects, some information about their development cadence, right? How are they
are they waxing? Are they waning in terms of their level of development? Poor requests,
issues being opened, who are the top contributors across these? This is the top
contributions across all of the projects, who are the top organizations, and in what way do they
contribute? Do they contribute in terms of opening issues? Do they contribute in terms of reviewing
or giving feedback? And what is the type of feedback we're getting? So times closed issues,
time to merge, pull requests, all useful information for understanding how well a project is running,
and how much engagement is happening during resolution? Is there good feedback on reviews?
Can you trust the review process? Is there enough feedback on it from that perspective?
So all of this is possible from this dashboard that we've created in order to help people
understand the future of energy system modeling or where they might want to go?
So what we do is we have to kind of herd cats to achieve this. We go to a variety of upstream
inventories of these kinds of tools, and we grab them all in order to pull together all of the tools
that we bring. So we know of these, if you know of more, really happy to know about them,
we want to pull in as many as we can, and we do some triage kind of, we remove duplicates,
we filter things out, exclude some tools that we just know for sure, repositories that we know
for sure are not energy system modeling tools, but they've made their way through to these upstream
inventories, and they also have to have open and public source code repositories and public
licenses, like permissible, permissible copy left licenses usually. We tend to find that actually
our just getting stuff in academic reviews and building a manual data set as the best of all of them
in terms of unique tools that you don't find anywhere else, and some inventories don't provide
any new tools to the mix. So they have tools that they've mentioned elsewhere as well, so we don't
actually add any new information for us. We bring this together with the ecosystem's API
and I'm sure Andrew and this bit, I think it is, well, it's probably somewhere at first end this
weekend and you should really find them if you can, because ecosystems are great API to take you
to make you something with your all of our metrics, and now you can also blame them when the
information looks weird, because we aren't really, we're not doctoring that information, we take it
and we just show it as it's. We also do some scraping to try and find the dots pages in a bit more
detail, because we don't get that well from elsewhere. That gives us some stats, and then we go
directly to some DRA APIs for GitHub and GitHub, and that gives us the information about our
users and about the repositories interactions over time. This is all run in about an hour and
20 minutes in a GitHub CI. The time is mostly in this GitHub and GitHub API requests, but we run
that just automatically, it's sort of set and forget now that it should happen monthly. I say that
but the last one failed and I had to fix some bugs, but in theory it should run set and forget.
So instead of you having to wait for the next review next year on the tools that are available
that comes out of academia, you can check this dashboard instead. So why might you use it?
Like who's going to it? And we at the Open Energy Transition, we've got sort of a mission
a vision to make open source the norm in energy system planning by 2028. That means going to energy
system operators, transmission system operators, and working with them to move them away from proprietary
tools to using open source tools for their energy planning. That's our entire goal. And that means
helping them understand what are the open source tools that are available in which one might be
most appropriate to them for a given use case. And we do this by helping them with the tracker here,
to produce a long list of possible tools for them. And then we deep dive. We let them deep dive
if they want to into more information about those tools. You can't hope to look at 200 tools,
but you might be able to look at 10 or so. And that helps you create a short list. So with one client
recently, back in August, we started some things. We just had some really broad brush things.
There has to be documentation. You know, it has to be easy for them. If we hand over the tool to
the model to them, they have to have documentation. They can't rely on us. They have to rely on the
tool itself. It should have an index package. The maintainers should care enough about it being
easy to download cross platform. And usually that means having a downloadable index package and so
some monthly download stats that we can grab. When you have to compile things and you are talking
to somebody who's limited them from a technical knowledge as usually Excel on Windows,
you have to work with that, not against it. And things like permissive source code licenses,
so that if they internally have a tech team that wants to make some updates, they're not going
to be limited by that. In this case, you also included copy left. We have one column that is
manually prepared because we've got some information on the top 30 or so tools about what kind of
modeling can you do? So you can get some really high-level information about the kind of
energy model you can do and you can use that for filtering as well. I should say, so PIPSEC came up in
the earlier conversation. They do rank the highest on a lot of things production cost modeling.
When I mentioned that I'm a lead dev of the tool for the last nine years, it's not PIPSEC.
So it's always quite galling to talk to clients and be like, actually, you know, the tool that's
looking best here for you is PIPSEC. I kind of just have to say that with a straight face and
and let it lie. There are other tools that do quite well in this specific filtering for this specific
client and yes, like, they do well across these but can you choose what waiting to give them?
Because some, you know, might matter a lot more to them than others and that, it's very difficult
but waiting to forks versus DDS, let's say. So we went in and we did a much more detailed analysis
and these are the kinds of things I'd love to automate in the dashboard and if you've got ideas
of how to do it, I'd love to find out test coverage. I'd like to find a way to automate
accessing test coverage information because people publish their test coverage or store it in different
ways depending on the language that things written in and other decisions. The quality of their
documentation. I might go to a documentation, it's a single page, this is how you install a good
link. So it looks like they've got documentation but actually they don't, so that requires some manual
stuff. Where can you get community supports? Where can you get data? We've had a lot of our energy
modeling tools and the fact that you need reference data. If you can get a tool but you don't
get any data with it. I need way to access the data and link it to that tool. That tool is functionally
useless to a lot of energy system operators. So reference databases for tools, you know, other tools
that can link into it really easily seamlessly is really important, etc. etc. So we can go through
this process of shortlisting through this and we did this with them to shortlist to about
three or four and that was really good and they found it really useful to have this sort of
verifiable way to how do you shortlist tools so that they can then go back to their higher
ups and say this is why we want to go open source and this is how we're going to do it and this is
why we're choosing this tool. But we need to dig deeper because we just heard earlier about pipe
so it's stochastic optimization. If you asked about pipes and you want to stochastic optimization
six months ago the answer would have been it's not the tool for you but you wouldn't know that
from any of these metrics, right? So we need to dig deeper into this and look at other features
and we went with a proprietary tool feature set. I won't tell you which one because I don't want
to get into legal ramifications of it and looked at what are the features that proprietary open
proprietary energy system modeling tools have, themes them, don't have to remember all this now
it's all available in the GitHub repository, grouped them and then created a variety of features
so basically a feature taxonomy of energy system modeling tools but at one that isn't too technical
it needs to be really understandable to potential users who as I said aren't super technical they just
they know they want to achieve a certain task but they wouldn't necessarily know the technical
words what that might be. That brought us on to a new dashboard dashboard you will probably tell
if you go to any of these I'm not a front-end developer and actually I've seen a lot of stream
that apps today in this room so I'm guessing that most of the stream are not front-end developers
but it does the job and these are all prototypes so not in aim to be production ready
and this is very similar to actually going to be an extension of the the other page
and in which it's it's own page that you can go in you've got your tools here and then you've
got specifically what are the features they've got grouped and most importantly and this is what
you're not going to get that easily and any of these review papers verification of that feature existing
all of these little superscript are a link to a page in their documentation or their source code
their tests that proves that that particular feature exists but also help you understand how you
activate that feature in that tool if you want it and whether something's in development rather than
so this one here whether it's in development rather than actually ready. You can use it to also
filter it based on the use case you've got and I'm not going to go into this detail because clearly
I won't have enough time to do so but you could say okay I'm running an integrated resource plan
which is usually what is done in the US been particular to make high-level decisions on the
future asset capacities that you want and we've gone through and relatively with some degree of
subjectivity we've gone through and said what are the features needed by this and this is a
filter view of those features and then you can look at your tools and say okay which ones meet those
features the best. You can do that side by side specifically on use cases so you could look at all of
these and you could say okay look at it for pipes or how is pipes are doing for all of these
next which other so you look for a single tool but you look across all the use cases and finally
you could look and you could write your own use cases so you could say okay here's all of the options
give you a little of a tool tapers to what we mean by it click all of the things you're interested in
save it I'll just click a couple more save it and then you can go back
to the use case comparison and I can look at it in that context or I can look at the tools
comparison I can look at it in in that context as well so I can look at my customer use case
and look at it in that context so it's a way for you to be able to and as I can see here you
I can't scroll down so it's clearly a problem there but it's a way for you to be able to
discuss with others what are the features we need for our tool and therefore what are the tools
that are most appropriate this is a manual task right it requires the developers usually themselves
to contribute to this so that's why you only see four tools and I'm hoping to see tomorrow very
great on here right yeah so we'll be here shortly afterwards and then I'm sure that gems pie will be
here soon too and the value of adding your tool to this is the fact that then it will be
used in the decision making process by energy system operators because we will be showing them
this dashboard right we won't be filtering it for pipes so that's it will be showing it to them
vanilla so get it up there as quick as you can this is a little bit what it looks like inside the
feature set so you've got just a simple animal config we we took sort of some inspiration from
Conterforge as to feed stock maintainers so you've got maintainers of these lists they don't
have to be the developers of the tool but you've got maintainers of the list and they will be
always tagged if there's some need for them to do anything we've got version schema for all of these so
you could have different features feature lists that are tied to different version schemas
and this is where I said you've got these sort of source sources to the specific
use cases and those will be checked in review and then similarly for use cases the the only
addition is that we've got a list of the assumptions that go into that subjective use case list
so we're going we want community engagement that's why I'm here today we want to improve
UI and UX we'll probably get a front end developer to actually look at this and hopefully
don't scream we want to add new insights we want to add things like what are the academic and
ideally non academic uses for these tools we're going to rely quite a lot on ecosystems for that
and open Alex we want longer term download stats in the last 12 months pie pie and anaconda
and Julia there were 3 million downloads that weren't CI related downloads of all of these 200
tools so last year 3 million downloads across all of those tools that is not capturing all of them
because we're only capturing really Python so we're missing quite a lot to want to get more analysis
on that and we're going to rely on ecosystems as well for security analyses of dependencies
because energy system operators want to be certain about the security of these tools they're using
we're going to be doing that variety of ways over the next sort of six months or so so really
if you want to get involved get involved soon and looking at UI improvements we really want to
refine this taxonomy make sure it's useful to everyone useful to developers so they understand
what they're missing and useful to use it so they understand what they need and what they can
get from the tools that they're using just as a sort of a case in point of how we use this so
the this this sort of feature gap analysis that we've got here with the we've worked with the
pipes to to look at all of these things that are missing created a roadmap on GitHub with issues
which if all closed we'll close this gap between pipes and probably the leading proprietary tool
for energy system modeling so you can go through this process by doing this yourself you can
say well what am I missing words it's what proprietary tools can do and create a roadmap for
yourselves to be able to close that great so that is me done these are the two dashboards but also
the two GitHub repositories associated with them and as I say this was done with a lot of
ultimately financial but also inspiration support from breakthrough energy that's which is this
project thank you very much
two minutes right rapid fire
there will be tools that do low-medium voltage yes definitely but these are generic tools right
so for the most part low-medium voltage is a data problem not a tool problem really so it's more
about how the modeling is done in terms of the tools so yes there are several that can do low-medium
voltage that was the question yes such a tip yeah do you know the you open source solutions
catalogue and could there be some integration just to make sure that you're
too hospitable and it ends up you can a lot so do I know that you open source solution
catalog and could there be some synergies there I do know of it this is obviously global it's not
just use specific in terms of its use cases but yeah we would want to link to it we should
probably actually think about as an upstream inventory more than anything else so that we could
potentially get more things from it that are index on it but not in the other ones we've got
at the moment we're not thinking of them downstream what we've got to other other places
how did you feel throughout the CIA really done
pipeline has that automatically in their BigQuery database and that's the main one and a
condo doesn't but it's far fewer downloads than pipeline and Julia does it automatically
in terms of removing CI from downloads
yeah would this help return on investment building open source yeah I mean what I'm hoping it does
is stop anyone from building a new open source energy system modeling tool because that is probably
the worst result on investment yeah it's already relevant we see that but that might be just
because of where we're getting the data from that we're not capturing the latest the newest ones
we are hoping that people will use this to support the financially support tools based on choosing
tools that look like them and sure enough to be worth supporting and that have strong
organizations backing them in terms of working on them so we're hoping that people will look at
this also from a financial backing perspective as well not sure how they would work out return on
investment yeah the key takeaway is that don't start developing a new tool there
cool thank you