Hello everyone, I'm Elia, I'll be speaking about visualizing mobility data, so the previous
talk was about preparing, collecting data, but we need to be able to analyze it and understand
it when it's good visualization tools as well.
So I am a social engineer, I've been working mostly on geospatial data visualization,
and yeah, I've participated in many projects, working with mobility data, representing
it in various ways, and my PhD was focused on that topic as well.
Also I was working for a company called Terreletics, where we visualized mobility in cities,
how people move in cities for different transportation providers to help them understand
the demand and improve their services.
So we built this dashboard, which allowed people to look at mobility data in cities, and the
company I worked for was a generous enough to allow us to open source this flow mapping
layer as an open source library, which we did, so there is a slow map jail library, which
is a DeGiel layer, custom DeGiel layer implementation, if you have familiar with DeGiel,
which is a framework for efficient geo data visualization.
So you can use, but then I realized that not everybody is a programmer, it was a couple of years
ago now, everybody is, of course, so I developed a tool which people can use without having
to know how to program in JavaScript, which is like, you just throw some data out of
all the data must be in particular format, to Google sheet, Google spreadsheet, then you
just pass the URL to the stool, and it will magically visualize it as an interactive map, and
people started using it in publishing stuff, that was pretty cool.
Then some people started to contribute in back, like there is, for instance, R integration
developed by the gore code of somebody from the community.
So yeah, briefly, what is a flow map?
Who's familiar with a concept flow map?
Not that many people, okay, so it's about utilizing numbers of movements of people or whatever,
it can be good, whatever entities between pairs of geographic locations.
You're not very interested in the exact route.
The people take is more about how many people people or entities move from A to B,
and then you can have additional attributes like time or mode of transport, whatever.
So a flow map is a way to represent that, and often a thickness of the arrows is used to represent
the amount of people moving all the color, or sometimes you see this animated particle animations,
like the number of particles will be represented in the number of people moving.
And the direction is important as well.
So arrow is like naturally understood representation of that.
And the kind of data which is can be represented way,
often called origin destination data OD data, so you have basically a table.
You have origin destination and count.
You can have additional columns for attributes like time or mode of transport.
So in developing this tool, I put a lot of effort into making the visual representation very readable, very understandable.
So in flow map jail, there is a double encoding for the amount of people moving, number of people moving.
It's the thickness and the color.
It is also the sorting of the arrows so that the most important points are at the top.
The arrows have outlines so that when you put a lot of them, they overlap.
You don't have a, they're still recognizable, this is individual arrows.
Then there is a fading, which is like you can configure basically the amount of,
downplay in the less important flows, you can adjust the get darker.
And this plays well with blending, which is used in flow map jail.
So basically it's a way to make sure that underline base map is still readable,
even when you have like thousands of arrows overlapping.
And it's not, it wouldn't work with just opacity because with opacity when you have overlapping arrows,
they would still, like the color would still add up.
So you wouldn't be able to read underline map.
But with blending, you can just use CSS blending because the arrows are rendered in the separate web jail context.
This is a technicality.
But never mind, you can make it so that despite thousands of overlapping arrows,
you can still read the underlying base map.
Then the location totals are presented as circle sizes.
It's a bit more complex than that, but I won't go into the detail.
There is an alternative way of representing the directionality of the arrows.
You can use this fancy animation.
Sometimes it's actually more readable, but often it's just like more appealing.
And still, like with all these techniques, you have too many flows which are kind of producing a noisy picture.
And reducing the performance is not a great way when you have to render all of them.
But many of them actually know it's like, here you see there are flows which start and end outside of the viewport.
Why, what's the point of rendering them?
They only obscure the picture.
So what's lemma, jail is doing is actually an adaptive filtering.
We filter the flows so that only those which are within the start or end within the viewport are rendered.
Also, we don't show more than like a certain number of flows.
You can adjust that.
So when you like zoom in, you see the detail for this particular region.
We also adjust the cables, right?
So when you zoom in, you see the flows which are the largest for this particular viewport will pop up.
So this way you can explore the regions in more detail when you zoom in.
And it improves performance.
Still, you often can get some messy datasets like this one which is bus travels in San Paulo.
Here, like the length of the aerosystem relatively long compared to the relation to the distribution of the locations.
So you get lots of over-plotting, right?
And it's not very readable.
Or here, the opposite extreme.
You have migration in Australia where most people migrate close by, right?
Not across the whole country.
So here you basically don't see the flows at all because that's too short.
So how can we address that?
Maybe we can like create a useful summary which would work at any zoom level.
And for like, independently of what the exact distribution of the locations and flows are.
So for that, the flow of an object is doing hierarchical clustering.
So who's familiar with the hierarchical clustering?
Some people. Okay, so that's fine.
Basically, we have the locations.
We calculate the total flows for the locations.
The total outgoing incoming or like, it can be different metrics.
But how important basically the locations are?
And we start with the largest so that when we group them together, the largest has the most,
like basically define where the cluster will be located, right?
So we don't move the Brussels to some smaller location.
Then we have some, for every viewport, we have zoom level.
We have a certain radius within which we group together locations, which are close by to the locations we start with, right?
So we group them together, we get the clusters.
We move them into the center of masses, weighted by the total flows.
And we get like that cluster locations at this level.
Now we have to recalculate the flows so that we basically aggregate all the flows between the constitutes of the clusters.
And we get the new flows at this level.
So this way we get aggregated summary for this particular zoom level.
And what it's called hierarchical is that we can repeat this process and go up and build a hierarchy of clusters for a number of zoom levels where I interested in.
So with this approach from this messy data says we get something like this, which is much more readable, but we like when we just look only a zoom level, we will lose the details, right?
The good thing is that this is fascinating so that we can do it interactively with zoom in and out and the clusters will expand and collapse depending on the,
zoom level. So this is a public transport and Brisbane and you can see like you can zoom in and see more details about the particular region, how people move there.
This is a true traffic. So with this approach, this is also true traffic.
It's not like origin destination, it's how many people move it within each segment, which can also be kind of converted into OD.
This is a public transport and Zurich and here it is also a temporal dimension you can and you can filter by the tram line or over time.
So by the way, this app is not open source, but much of the technology behind it is and I'm going to open source parts of it is a part of a new project I will mention, but this one is using duck DB, which for me they would duck DB.
A few people, okay, so this is a pretty cool database which has some advantages for this kind of use cases.
First it's made for analytics, it's using a columnar, data representation, which means queries aggregation for instance are pretty efficient.
It's also embeddable in really anywhere. So this example I showed you before that DB was running in the browser directly, so there's no back end serving this queries and every time I zoomed in and out or moved the viewport the word like dozens of SQL queries preparing the data for this particular viewport.
And that DB, since it can run in the browser, by WebAssembly, can do that and it requires very low inference setup.
So basically you just need to download the data from an S3, like a profile, add it to the duck DB running in the browser and you are good to go, you can run queries directly in the browser.
So yeah, in this app, like when you load the data set, it will prepare, it's too small, I know, but don't worry.
It's prepared like pretty aggregated tables, which will make it easier for to run later queries for particular viewport to get the data needs.
So like for different zoom levels, for instance, there is a mapping between original locations and clusters on each zoom level so it can quickly do the mapping and prepare the data.
So this is them. If you want to try it out, you can quickly scan. I won't be showing it for the sake of time.
So yeah, now I'm working on this new open source project. It's called SQL Rooms and it's basically a framework, which helps you to build data analysis apps and it's kind of backed by that DB.
And it's very modular, like the different kind of modules you can functionality, you choose to, you decide which functionality you want to add into your application.
So yeah, like basic data set browser, database browser, you can see the tables you have, you can have an AI systems, which will generate queries for you, SQL queries, which can still run in the browser without sending data to any, a lot of providers.
Many things, and I'm also, yeah, it's been used by a couple of several companies specifically for data intensive data visualization applications.
And I'm working now on, like, it will be an example showing a flow map, which serves the data directly from the DB in the browser.
And yeah, so yeah, and as part of this work, I wanted to have a demo for this conference, but I haven't managed live gotten tense.
Sorry about that, but it's like working progress and explaining what I'm working on.
So for even larger data sets, that DB is already pop pretty powerful. You can have like millions of rows in your, in your flow stable and will be good enough.
But what if you have like many attributes or small temporal back buckets, the number of rows in your table in a flow stable will multiply very quickly.
And at some point it will be too large to, to load the entire thing into the browser.
And you want to avoid that. So, but what you can do, you can prepare your data set in a way so that you can only fetch parts, which you personally need for the current viewport.
And that DB supports HTTP range queries. So basically, if you have a archive file somewhere in S3.
And it has a like column by which you can filter, you can say I want like this range for values of this column.
And if the rows are sorted in the right way, then it will only need to read those parts of the table, which satisfy your,
your query condition, right? So if you can do it this way, then it will be fetching data on demand as you change your viewport without having to load the entire table.
And so to do this, who is familiar with space filling curves? Just a few people, okay? So this is a mathematical concept, but it's actually actually easy to explain.
Like the purpose is to compress two dimensions into single dimension. So you have like two columns x and y, right?
Oh, let it you launch it better use x and y projected already. And then you want to, this are two numbers, you can save them separately.
But our purpose is like to have a column by which we can sort so that then it will be easy to query so that things which are closed by close to each other.
I'll also close in this table close to each other so that we can, we need to fetch fewer chunks, right?
And so this is the way to do that. Basically, you draw a curve, which would fill the entire plane, right, to the plane.
And then like at every junction you put a number one, two, three, four, and so on. And this number is the index, right?
Which you can then put in your column. And this way you can reduce two dimensions x and y into single index, right?
And this will be the number you want to sort it by, so that you then need to, to read a few, a few chunks from the table.
If you sort by this number. And that VP has a function as the Hilbert, that DB spatial extension, you need to load the spatial extension, to calculate this index.
And as part of the project I'm working on, there will be a Python script preparing like you give it a normal OG dataset with just like a simple plane origin destination count table.
This will prepare this kind of, part of our sorted and with the index.
Yeah, the only, yeah, so this is like kind of a SQL query you need to prepare this.
But the trick is that you need to do this calculated calculation of the Hilbert in its twice, because we have let alone, right?
But we have it for the origin and destination for OG data, you don't, it's not just one point, right? It's starting then.
So you can, but you can do it twice. You get it first for each for the origin destination, let alone.
And then you get two numbers, right? But you can also use as the Hilbert for these two numbers, you get for the origin destination.
And it compresses it again. So you get from four dimensions into single dimension.
You sort your table by the single, a column flow H in this code case.
And then the flows which will have, which will have the same or closed number in this column will be also close to each other on the map.
And then we can, we don't need to read as many chunks to represent this data.
So yeah, that's it.
Follow this word, yeah, best is probably SQL rooms, where new stuff will be appearing.
And otherwise, flow map jail, flow map blue, flow map blue are, flow map city.
And here's the example, you can try. Thank you.
Thank you.
There's a time for questions. No, okay, sorry about that.
But reach out to me.