Thanks everyone for coming. My name is David Cornierzik and I'd like to talk to you about
the state of specifically check public transport data. I've always been a fan of trains
and for the last few and by few I mean what? Eight years I've been working on finding
and improving and then using a check public transport data ideally under some sort of
open data license. I'd like to talk first I'll give you a rough outline on what kind of public
transport we even have to talk about here. A rough history of the central timetable database
because it's fascinating and timetables I feel are sort of the core core data that you need
to do most anything with public transit. Then I'll summarize what data is available right now how
usability is. Then I'll complain about international formats so I'll complain about GTFS
because it doesn't quite work for us or everyone and then I'll give you some links to
fast projects which you can use or even contribute to. I would be happy for your contribution
certainly. So in the check public we have pretty much anything we have we even have fairies we
have trolley buses we have underground trains and the most of these are ran by private or public
agencies generally just private agencies or the city itself has a company made for this purpose
but most and I'd say like 99% by a kilometers driven of these routes are actually ordered
by the government in the broader sense so the central government region or a city. So there's
only a small minority of routes which are fully privately operated. Also complicating matters
is that on anyone route you can generally buy tickets from at least three different tariffs so
the agencies own a city specific one region specific one or one ticket which is supposed to be
a unifying tariff for all the trains and maybe other things but that's just never happened so
we need to encompass all this in our well we want to encompass all this in our public transport
data. What exists right now and exists I mean on a computer in a database. Time tables of course
exist somewhere there is an important division between regular ones and divergence because some
sources will only give you a regular time tables and when something changes you are just out of luck.
Real-time positions and delays of vehicles that's certainly interesting to look at but
also interesting for statistics to see how good the trains are running this is even important
I feel from a political perspective because you need a data to assess whether the person in charge
is actually doing things properly. Train composition that data exists and it's sometimes accessible
to normal people but we'll see later reasons for disruptions also interesting mostly from a historical
perspective to see what's the biggest problem that we should be focusing on but also just
out of curiosity you want to know why your train is five hours delayed right and trajectories
of transit lines. This all exists how much of it is centralized and how much of it is available is
another thing and like I said most routes are ordered by some kind of public institutions so
you can use that as basis for requesting all these data or most of it under a freedom of
information laws. It's not as simple as we'll see in a moment it's easier is the central time table
database of the Czech Republic every time table of every public transport line should be held in
this year under some circumstances like being there a few days earlier in case of most time
table changes it was really actually forward looking I'm not going to say unique in Europe
but very much futuristic in 1998 when this road out and countrywide the first contractor
actually decided to develop it for free maybe very forgot we weren't a communist country anymore
then they got restructured like twice and by the board records that I have found board meeting
records they decided well that's a very bad idea and it's costing us tens of millions of crowns
every year to handle the system so they wanted to cancel the contract of course the ministry
didn't want to pay now because they had like two years of service for free so another company
stepped up and decided fine we'll do it for free but we'll make a new contract the ministry didn't
care at that point so this company helps got a sole permission to use the data for anything that's
not mandated by law so selling the data became one of their primary profits and running services
based on that data this lasted about 10 years almost exactly and then says now one of the biggest
sort of internet services providers but not connectivity providers just services on the internet
provider sort of weird thing that kind of rivals Google in breadth decided they wanted to start
their own trip planner for public transit in a Czech Republic so they requested the dates are
based on a freedom of information law but of course the company didn't want to give them
their data because they built their business model on providing the service for free and then
keeping the data so this took multiple years in course very much tested the law freedom of
information law in ways that it was tested before generated lots of novel judicial opinions that are
now cited in text books so hooray and but they did win though in the meantime the ministry
probably seeing the writing on the wall at the end changed their regulation
sort of delegated regulation that's almost with force of law to make releasing that
timetable data in full mandatory and then after brief spat with the company helps they they decided
to pay for upkeep of this year and perhaps decided fine will release the data except they are
very crafty and they released well only part of the data they released data that looked very much
like what people wanted but was missing among other things two very crucial columns which meant
that stops were no longer uniquely identified so you you've had a big zip file with like 10,000
smaller zip files in there each of them covering one line in one period of time and now try building
a central database back from this sort of data when you can't join on stops
says now didn't pursue any further legal action presumably and now this is my sort of conspiracy
theory because they because they also manage a map website so the web map website that has their own
custom map data for the Czech Republic so what they decided they do is they gathered up
position information on every public transit stop in the Czech Republic and then managed to
that's the conspiracy theory sort of reverse engineer the data back into a usable form
I decided I'm not going to stand for this and I sent my own information requests to
hubs and to the ministry to get the full data back because that's what the specifications says
I should get and that's what will loss as I should get well they didn't see it as quite clear cut
after many years and two parallel lawsuits in basically the same thing just shifted by a week
or something like that and probably thousands of euros spent on lawyer fees
probably maybe more I won I've got slightly better data thank you thank you it's a fascinating story
but I gave a talk on this in Czech and spent 45 minutes on it and I had to go really fast so
so I got uniquely identified stops but perhaps being who they are they only did this for
new newly inserted timetables I am very sure that they have the data for the old ones as well but
it's pretty hard to prove and I've sort of run out of the structured ways of forcing them to do
anything so I decided screw it because in the meantime I managed to do the same thing that says
none did so I gathered lists of stops and their positions from public data and managed to
sort of reconstruct the original data from this and it works with a very high degree of precision
so that now the real issue is not the missing columns which aren't missing anymore in parts of
the data very the real issue now is that the data is just not 100% correct because
perhaps don't use this data specifically for their trip search engine and their commercial products they
also have agreements with specific agencies which use their software and so they don't have any
incentive to actually fix data issues with in the database and the firm industry which
probably should take care of this doesn't want to so short of suing the agencies for sending
wrong data I have to do something else we'll get to that later in the meantime of me fighting
everyone around me the European Commission decided to mandate the release of a bunch of
transport data and timetables in netx you probably heard of it in other talks in this room
the deadlines long gone some countries have done the thing they should have done and produce
actually good data some countries just don't care at all like Slovakia and the the check
Republic decided to do the check thing and the Ministry gave caps a contract to convert the
existing data to netx and the existing data still just as broken as the previous data
now it just it passes excess devaluedation but I hope you know that's not the end
that data shouldn't lie or it should like comply when it comes to semantics and this
I'm sorry it doesn't it's even worse than the source data it's it's basic you can't you can't
make non-broken data from broken data barring any for the inputs like says land data right so that's
sadly not really an option for us so where do we get timetables from well the other public
institutions that play a role in this system and the order most of the routes in the check
Republic so some of them are in lengthened enough to actually release these things as an open data
most of them don't do this sadly but it's a start there's other open data of course
like stop positions in GTFS of course some regions I'd say most of them like 90% of them
publish pretty good geo data databases of stop positions as well even if they don't publish the GTFS data
real-time positions are available in some regions for regional routes and for all passenger trains
sadly that's except for I think two exceptions that's never opened data
it's it's a bit of an issue this is what it looks like in Prague which is part of the Prague
integrated system which just provide very good open data so thanks Prague you can see even the
tip of a zoo is is what tram is actually running on that line right now so you can track and I
have done that when Prague got a new tram I put in the new tram in the search bar down below
and look where I could catch the new tram and then went there so railway data sort of it's own thing
most things in the check system work for railway separately and then for everything else separately
even though trolley buses are actually railway vehicles in the in the check legal system
but Prague as well as the railway manager is I'd even say oh anti-open data because they release
basically nothing and they are very hostile to any freedom of information requests
but they have very cool stuff like historical delays and reasons for delays which aren't published
in any way so you can't even strike them which is a real shame especially when the compensation
for the manager of this organization was at at some point based on a cumulative delays in
the networks so they have their cool stuff they have data for freight trains which might be in which
I'd love to have but they just don't want to give it away now we're getting to the formats
the native format for the check non- railway time tables is yet FJDF yet not yet the
delivery format it's basically just you take the printed standardized form of a time table and put it
into CSV and it works pretty well it's kind of clunky but it works let me have GTFS GTFS sounds great on
the surface but it doesn't represent it can't represent everything that we need in our check system
like it can't represent the mess of tariff zones that we have like you really can't have
one trip that goes between three regional systems and supports two additional sort of
ticket systems on top of that and GTFS is just not made for it it's made for one system in one region
managed by one agency and for a bit in travel within a group of stops it's a real problem
in for example when you have long distance buses they can't travel within one city or even within
one country sometimes and GTFS just can't encode this restriction so you get bad results
in trip plans and netx is awesome but really overcomplicated it's a standard for making standards
and then fine you say fine I'll look at EP IP but EP IP doesn't cover all the things we need
in the checker public so we're back at square one so barring any sort of international cooperation
and netx is I wouldn't say it will not be answered but it's just not ready right now it's not
something I as a programmer can pick up and get to work on now we're getting to the
cheery solutions page many people have failed many people have tried them many people have failed to
make checker wide trip planners mostly that's the holy grail a free and open source multi-model
trip planner so there's basically free surviving projects right now but I've been running for a few
years and are promising and still maintain my own year or until which produces mostly correct
data because it produces GTFS data and JDF but nobody uses that of course there's the transitors
project it's very cool an international and I like it and some people have made a check specific
data source for it but it has the same GTFS issues and then there's spoying coverage it's
very full stack it's a bachelor species of a colleague of mine and it does everything it's
sort of hamstrung by missing data and missing API sometimes because what it tries to do is
tries to do price optimization it's very cool but probably not something that you could extend
beyond the checker public without a lot of work and then as a bonus there's my own article
I'll give you duo which is a so we're not just doing timetables it's a scraper and view of
delay data so you can look at the delays on the railway network for a given given train over
maybe the last month and then you can see do I have a chance to even make that connection very useful
this is what it looks like so if you're interested in this there are a few links you can follow
up on all of them are in check as expected sorry but translators are good these days I'm not going
to get this you know need to learn check and my wishes for the future public institutions please
stop killing my attempts at getting data from you it would cost much less money to just
give neither thing instead of paying tens of thousands of prints to lawyers I never paid any
grants to lawyers and I want so screw you to check transport enthusiasts please I know it's hard
but try to cooperate with other people instead of starting your own project this is not something
you can solve solo I know I tried and for all of you international people trying to make pan
European projects sorry bearer GTFS is not going to cut it please let's make some extensions
or a new format thank you