We start with the next presentation about the flat land framework, so it's still about
the railways, but all of us will present really interesting approach into open innovation
as well and all of us need to open source.
Thank you very much. Welcome also from my side and very happy to be here and talk about the flat
land framework initiative that originally started at SBB, but now evolved into its own association
based in Burns, Switzerland. A little bit about Swiss Railway network, these numbers are
only from SBB, but SBB is the biggest operator, so it gives you an idea about Swiss Railway
network. There are about 3,300 kilometers of track in Switzerland, there are about 11,000 train runs
per day, passenger train runs, about 1.3 million passengers transported every day, about 175,000 tons
of freight, and a punctuality of about 92.5% precise in Switzerland.
Storm has to be his open data portal, I guess they're correct.
So just to give you an idea, a railway network is quite complex, especially if you have a
dense schedule, also the demand might increase, so there might be also demand or requirements for
more even denser schedule, and so yeah, this poses a lot of challenges for the railway network.
You can also imagine if you have something like a train that is delayed or even versed,
a breakdown that takes a long time to fix, somebody needs to take care of these trains that are
already underway, on one hand to minimize delay, minimize impact, but also to come back to a nominal
state as soon as possible. So this is done by human dispatchers, they navigate or they direct
the trains, they don't drive the trains, but they help, and which routes they have to take,
they have to reschedule to redispatch. So there's already a lot of research in operations research,
traditionally from railway operators that also looks into how such tasks can be supported by
algorithms, by software, but they have a similar problem than the human operators, it is that
if we scale, if you look at bigger networks, like the whole Switzerland, this traditional
optimization algorithms, as well as human operators, they don't scale that very well. So it's hard
to optimize the whole network, especially in a short amount of time, so here we talk about some
minutes, where we have to find solutions, at least short term, so the idea was to look into
novel approaches, so not just classical optimization, but also novel approaches, and about
six years ago at SBB, the idea was let's look into machine learning and in particular
into reinforcement learning and how this might help with this problem. So just a little bit about
reinforcement learning, so basically you have reinforcement learning agent, the agent, the
picked it here and on the right is the brain in a box, you have an environment and the agent
wants to act in this environment and the agent can observe this environment. And in addition to order
to learn, you also need some measure that tells you that it's an indicator how good your action
was. So this is a basic reinforcement learning setup, and so the idea was let's see how we can do
with reinforcement learning for these kind of problems. This worked very well in the beginning,
some early experiments, but it was also clear that this is on one hand novel approach,
it's not so established, it's not the years of research in reinforcement learning in railway,
and also there are a lot of different approaches and so the question was like how can we
make use of this technology without doing everything in-house? And so the flat length frame
worked with spawn, you see it here, and the right, the nice animation, it's basically what
you get out of it, at least visually, so you see the little trains moving, and on the right you
see the basic elements, because the question was also not just how to use new technology,
but also how to bring researchers and people working in reinforcement learning into the railway
domain. And so the idea was let's create something simple that still gives enough complexity
to solve a real-world problem, or at least to look into a real-world problem, but that's also
accessible for people that don't know that much about railway. And so the flat length framework
basically is on one hand assimilated, discrete time simulation, it's on a 2D grid, so there are
some things that are not possible in flat length, but it works quite well for trains because they
usually don't fly, and you only have a couple of simple cell types, a couple of simple as
switches, straight lines, curves and so on, and basically everything in flat length can be represented
by these 10 cell types and rotations of it. Then we have a train, the agent, and the
station, the target. So the task is simple, I'm in train, I start somewhere, I need to
navigate to my train station. However, this might look simple at first, but the question is how,
what's a reward, how can I observe the state, because we have a little bit of the same problem as
the classical optimization, if you want to look at the whole grid, the whole time, so every agent
absorbs the whole grid. If we scale this to the network, the size of Switzerland,
this is again not possible. So this concept of observation and reward are very interesting and
very important for reinforcement learning, and so defining the actual observation and defining
the actual reward is quite challenging. So again, the idea was like how can we use this, how can
we use this approach for this kind of problem. So on one hand, we looked into what are possible
observations, so there's some basic observations that are provided by flat length, the global
observation, which is not so good. Local grid view is another one quite intuitive one, I just look
at some cell cells, but you might imagine if you're on a train on the tracks, the things left and
right usually don't matter so much. However, it matters where you go. You have junctions,
switches, you can go left and right, and so we have also built in this local pre-observation
that should very promising results. However, this was the beginning, we saw that this had potential,
but it was way too much to do in-house and also not just to open source, because
yeah, why should somebody care? And so the idea was, in the beginning, let's start with a challenge.
Machine learning challenge, to bring people onto this topic. So the first flat length challenge
then started, everything was open source, and we had a lot of traction also on this challenge.
So it was really, it has a community that can help and not just people in-house or not just
people that, by accident, stumbled upon the GitHub repository, but that we can actively engage
a community. And one big opportunity here for open source was really like, you can also imagine
like the flat and environment could be like basically close to source code of the environment,
and you just have an API that can interact with the environment. But what I mentioned earlier about
observations and also reward, you can imagine if you have a train run. How good is one decision?
It's hard to say, to go left or right or straight or stop. So you really need to
take into account the whole picture. So you have all the train runs. At the end, you want to be there
on time. But it's hard to say what action was actually responsible for being on time.
And so this was really one important decision for flat land to be fully open source,
because the reward shaping and observation and building is a huge part of reinforcement learning.
And this is really something that also then the community came in. And for flat and challenges,
you could basically modify most of the flatland environment. For the evaluation, there were some
things you couldn't do, but for training basically you could do most of it. You could really
adapt the environment. You can build your own wrappers to build your own observations and so on.
And what we've seen was actually a lot of engagement. On one hand, we had actually extensions,
published also, for the flatland environment. So it was not just a group that did this for themselves
and didn't share it with anyone. On one hand, the condition was that if you want to prize,
you also needed to be open source, if you wanted to prize. But there was also really a lot of people
that were just interested in the topic and happy to share. And so we had a lot of extensions,
like for example here, you have the whole network, but then some people realized only actually
the cells where I have to take a decision, they matter for our approach. So for example here,
the red cells here and the right is where you can stop. So this can have an impact, or you have
switches. That's something you have to go left or right or straight. So there you have to take
it a decision. So they made a report that only basically, or that masked basically all the
non-decision cells. But we also had people that actually built tools for analysis. So there
for whole was a Shivam was one of the, or was he driving force for this tool. So he built this whole
tool where we could visualize all the train runs. You could see the decision statement made
to concessions like here, for example. And you could analyze your whole train run and see
value model or your approach might still fail. This was also open source. Even during the
competitions are also other people could use it. And we had also a lot of people engaging actually
creating the baselines for the challenges. So what you see on the top right is the training
of the reinforcement learning models and a lot of different approaches. So also there,
people just joined in and said, like, hey, I'm interested in this. I want to work on these baselines.
Would you see an orange is the global observation that I talked about before. So you see that the
higher up, the better. So there you also have to prove or at least some proof that the global
observation doesn't work that well. But you can also see that there is a learning curve. And yeah,
and I think this is one of the big things that I came out of the framework.
I think due to these efforts, it was also possible actually for
multi-action reinforcement learning to catch up. So what you see, the top line, the blue one,
is no precincts, operations research approach. So it's still kind of a classical approach
from a multi-action path finding. However, you see here also that the multi-action reinforcement
learning solutions are close. This was not the case in the first challenge. So from left to right,
you have growing complexity of the networks, you have more trains, you have bigger environments,
and again, like the higher up, the better. So one is basically a perfect score and the more you go down
to the less. Here also I think important mention for the competition. They had a limited amount
of time to calculate the next step. So it's, you couldn't like, calculate for a whole hour
for the next step to take for the next action to take. So this was also constraint. And this
is also why you see operations research, for example, doing them. Thanks to also these flattened
challenges and being open source, we also realized others have the same problem. And so this European
research project, the rice and your projects started, AF RealNet, that looks into AI for real world
network operations. We're not alone there with the trains. There's also aviation and the
power grids. And we really realized that not only flatland, but also other initiatives like it
can also help each other. Because some approaches that work, for example, for aviation,
might also work for railway and also pop out it. And so this is really a nice story to see,
like how also open source projects can help each other. In addition, what you see here on the left
is a human. So it's not only about AI, but actually flatland is also used to facilitate human
AI interaction research. So part of AF RealNet is also this research on how can a human operate
to interact with an AI system. So conclude. I think open source can really inspire. In our
challenges, we have more than 4,000 submissions of solutions, with more than 1,000 participants.
And we have also, or they were published. And if we have, there are more than 100 papers published
that directly reference papers we wrote, but also directly made use of the flatland environment.
So from our point of view, this is really a success story. Open source made it possible without open
source. This would not have worked. And if you want to contribute, we're more than happy to go
to our GitHub. Read the docs. There's a start to keep the text about 15 minutes if you don't
have to code and paste the title to run your own environment and train your own agent. Thank you
very much.
Yes, I want you to know if you have worked on an explainability on the models for the
CPU makers, policy makers. So the question was about explainability and if we worked on this
for policy makers and so on. So the answer is we at Flatland we didn't, but there is actually
an A of RealNet, the project I talked about before. Part of the project is also explainability
for multiple reasons. On one hand, because it's in general an interesting topic for AI to make
models explainable, but also that what happened in the past also with Autosoft, I'm not just
AI, is that if human dispatch, for example, or just a user of whatever system AI assistant,
doesn't understand the solution that AI provides, especially if it's really like an interaction.
It's very hard to work with that system. So basically in AI of RealNet, there's both a research
on explainability just in general of the model for safety robustness and so on, but also to enable
really the human AI interaction. So what is the objective of the agent, most of the question,
if it's just about delay or also something else. So basically in the challenges, it's not only delay,
but there it's also kind of a very simplified problem. So you get a score at the end of a train run.
It's also, for example, about completion rate. Basically it's the time you need to get there and
the completion rate, because what you see here, well, you don't see it, but there are no actually
that locks, but you can imagine that for the AI agent, it's not just about being there on time,
but actually to avoid that locks, which might sound easy, but it's not that of an easy problem,
especially if the network is getting big. And so part of the goal for the challenges was also
to avoid that locks. But in principle, you can model any kind of question, well, that you can
represent in this environment, but basically the scoring, you can also define yourself. So this was just
what we did for the challenge to start, but it's the environment also allows for other types of scores.
It allows you to fill a lot of goals.