I am very, very delighted to introduce Alexis Jacqueline, who is the author of an exciting
project and is told it's going to be developing custom UIs to explore graph databases using
sigma.js, take it away, thanks. Hello, so I'm Alexis, I'm a web developer at West Square,
who are a small firm in France in North, and we mostly develop web applications for data exploration.
We produce some open source code, including sigma.js and a tool which is named
defy light, the web version of jiffy, something else graph related, and I'm here to talk
about developing custom UIs for exploring graph databases. So let's start with Ricardo. Ricardo is a
research project from France to explore international commerce in the 19th and early 20th century.
It's been started, I think, at the Shonspomedia Lab, from people who are also big fans of
Phas Dam and open source software, and the main goal is to craft data sets to explore international
trades between sovereign entities and titists from 1830 to 1930. The core that a set is made of trades.
Basically researchers took huge journals from various countries where they listed
trades they had with other entities with varying currencies. In Ricardo, we don't care what people buy,
we just care about the monetary fixes. This is the core starting block of this project.
The main issue is that the entities that are reported as partners from sovereign countries are
not sovereign entities, and this data set is full of France, Southern France, Eastern Europe,
US, Atlantic coast, this kind of thing, and this is not good because people wanted to explore
trades between sovereign and titists. So researchers started another project, which is named
geopolitist, and it's a database of sovereign countries a long time, because the variety evolves
in the 19th and early 20th century, so it's even harder to apprehend, and we have this new
data set with antities and connections between them. Things like Paris is a part of France,
France is sovereign from this data to this data, etc., etc. You can check the page of this project
because there have lots of exploration tools for this specific data set. Another part that sounds
necessary since what I told you, we have trades reported in varying currencies, in varying countries,
in varying times, so people from Ricardo also drafted, drafted data sets to get all the
external rates a long time, and so we can finally have rural, girly trade reports,
we've normalized monetary values, and relations between antities, and that's what we're going
to focus today. We have this huge network between antities, and we have connections that are
trades, and connections that are geopolitical, kind of. So let's put everything in the
Neo4j database, put the slide with draws the rest of the old. We have a Neo4j database nice,
and we can open Neo4j browser. I don't know who is familiar with the Neo4j browser here.
So I opened it in the Neo4j browser, and the first image I had when I opened something with this,
because the issue is that for each year, for each pair of antities that reported trades
to with each other, I have one edge, and I have 105 years, so if I want to just draw the network,
it's directly unreadable. So we need to find better strategies to represent this message.
Also, we want to extract, so in the recap of the project, they have lots of heuristics,
etc., and code to actually generate network graphs of trades between sovereign antities.
But here, since I kept the trades, I, the road trades I, there were in the initial dataset,
I don't have necessarily direct trades. I can have trades between Paris and UK or Belgium and
North of France and this kind of things, and we want to be able to actually observe them.
And finally, the Neo4j console was very good to actually spot some issues, because we have
lots of antities that do trades with themselves, and that's probably bugs in the scripts that
took the original sources, and generated this road trades dataset. So it's a bit tricky to explore,
and it's a good use case for some useful new eyes. So I'm going to talk about Sigma.js now,
it's a JavaScript library. We developed a quest where to draw networks on web pages,
and it's focused within its ecosystem to build applications for network analysis.
It's not very good at drawing schemas, cytoskeptoj, it will be better, it's not necessarily
great at handling very custom renderings and interactivity within small networks as
difficult as well, but as soon as you want to display larger graphs, it's a very good tool.
And one of the main reasons for that, it's just handles rendering, and another or some tool
developed by Mediara of Sianzpo again, graphology, handles everything that is computing related.
So we have graphology, which is basically a graph model for JavaScript,
that provides a lot of algorithms to compute metrics, scores, to apply layout to the network,
etc. And then we give this refine networks to Sigma.js, that's just going to run the rates on web
pages, and then we can focus on interactivity. So I forgot with the slides.
And now, we have some, there's lots of features. If you go to the website,
you will see the list of features that Sianzpo provides, but let's dive into the application.
When I think it's already used in many tools, and actually here, Jeffy Lite could have been a
good solution, because we don't have that much of a big network if we merge the edges together.
I mean, we have around 4,000 entities, so Jeffy Lite could have been a good solution,
and also G.V., which is unfortunately not open source, but they do a lot for
an source, especially, they pay us to actually develop most of the recent features of Sigma.js,
so big thanks to them. And that's basically, yeah, the Neo4j browser on an
steroid that could have been a good solution. But let's go, Q-Stone. Everything, the code from
the application is on GitHub. I didn't want to put Neo4j server, so if you want to run it,
you have to run it yourself locally. But all the instructions are in the repository to build
that sets and runs the application. Also, I tried to keep as Vanilla as I could. I used Type
Street, because the other Street without Type Street is too painful for me now, but there's no
view, no React, no Angular, and I tried to just integrate Sigma with Web Components, which is not
something I'm very used to do, but it worked well. So, the first view I wanted to draw was
in good networks. Basically, what are the neighbors of a given entity and how they are connected
with each other? This is how it works. We have this Neo4j database. We will run a
Cypher query. We will extract some row graph data. We will use some draw flow demodic to get
a drawable network with everything we want, and then we will give it to Sigma and get this
interactive view. So, the Cypher query here is we have a center C. We want to get all networks,
and at this connecting my center to this neighbors. Sorry. And for each unique neighbors pairs,
I want their relations basically, and this will give me the Agonet work. In graphology,
I will aggregate all parallel trades. So, to avoid having all those parallel edges as we saw earlier
in the Neo4j browser, I aggregate them for all years, and I will have a size of the edge that is
related to its monetary value. And then we set various graphical variables, and that's it for
graphology. Then in Sigma, we just have to write a bit of code to handle parallel edges,
because we can it's easy to tell Sigma that all edges are curved or all edges are
eros, but here I want it to have straight edges, because I think it's more readable. That's
my opinion. And parallel edges as curved. So, there's a bit of code there. Adding some buttons,
captions, highlight neighbors, interactions, these kind of things. So, about the code itself,
we have at some point a Pupere graph, which takes a data graph that kind of directly comes from
Neo4j, and we run some graphology code, make parallel edges curved. That's what I said earlier.
For the position of the nodes, I will first put them all on a circle,
then this comes from the next feature, but if I have some fixed nodes that I want to
polarize my final view, as we will see later, I put them on a larger circle, and then I run some
false at last two algorithm. This is kind of physics algorithms to get some position for the nodes
based on the topology of the network. Most of the graph images we see in the literature
that are, if they look like hairballs, this is the algorithm that has been used probably.
And yeah, some interactions, I can show more code later if needed, and then we got this kind of
views. So, application itself looks like this. I want to explore reported partners of the entity
Belgium. I want to include the center of this Egonet work. This is not mandatory because
I know that every node I will have will be connected to this one, so this will bring a lot of
noise, but let's try anyway. And I want all trades on the whole period, but I will only keep
trades with over 500,000 dollars. So, if two entities traded for less than this amount,
I will skip the edges. And I will keep only exports edges for now.
Okay, and here comes the network. I have small gaps shown at the square. I can roll over
a node to see its neighbors and its context. This one is interesting because it appears a lot of
time in lots of networks. And I wonder what it is. Sorry. Okay. But yeah, we see that we have lots
of various entities that are not sovereign. I don't, maybe this one is, I don't know.
South Africa is, I'm looking for the weird ones. Okay. Is it? I think it's a weird one.
But yeah, this is a bit messy. Let's do the same one, but with, not the bellion, not in it.
Okay. Yeah. Well, it's a nice herbal. I think we might be able to see more interesting things.
This exact graph actually is the networks of, look, this is not supposed to be true.
Basically trades and geopolitical relationships around Belgium only in the first 17 years of the
data sets. There's still this world estimation, not that takes most of the information.
Actually, that just removes the trades to see how it would behave. Okay. This is an easy one.
So at least this gives me all the, thanks, all the direct relations, all the direct political
relations I have between Belgium and other entities and this starts to get informative.
But we can hope to do more. Okay. Yeah. I did this exact query, but for United States of America to
list of the political entities linked to America to United States of America.
No, we want to see indirect trades between two sovereign entities. So I will take two
entities and I want to see when they trade with each other. When one entities, one entity trades with
a part of the other entity, or when the part of both entities trade together.
So I have a new cyber query that gives me trades between the two entities and all the
path with depth two and all the path with depth three basically. Then I take the same
graphology and sigma scripting code and I get some new networks. So according to the sources we have
how does India trades with United Kingdom between 1833 and 1938? I decided for this network
not to display the direct trades. There is an option for it because it took too much of the
information. But yeah, I see the United Kingdom trades with Bengal, China and Mumbai while India
trades with British and South Africa and British Borneo. This is quite informative to me about
the data set. No, I know I can challenge the lyrics of the really of the code in regard to see
if the final trades contain all this trade I am observing right now.
Yeah, this was another example between United Kingdom and United States of America and
in this graph I kept the direct trades because for once I had significant edges that were
not direct trades and in the reports United Kingdom trades a lot with Atlantic coast United
States of America and this might be interesting for researchers to know how those trades are reported
etc. Again, so the more time I already demote some things, I can show more.
Yeah, I won't draw too much but I don't know if you would Belgium and United Kingdom
the main issue is that if I take Belgium and in United Kingdom I kind of expect just to get
the direct trades because the countries are very close to each other. Yeah, okay, because just
some trades very minor with British Western Africa in this time span. But most of the times
this network in this view will look like this basically, which is also informative.
And so I'm going to show a bit of code because I am early and I speak fast.
For the graphology I showed some things. Once I have this graph basically I just so sigma is
instant-hated. It's a class you you spun it by giving it some some settings here with things
or documented, but it says that yes I might want to order not the energies on the depth.
I want to run the edge labels etc. and then in my web components when I have a new graph
I basically just give it to sigma and I tell it to refresh. And this is some advanced code to handle
this. I want to highlight the networks of neighbors when I hover another end to display the
edge labels but it would be usable without it. Honestly, the part that was most annoying to me
was the safer queries because I'm not used to write them. But it worked well.
So one of the good things we the good thing writing custom you you eyes is that we don't have
control to most solutions that are plug and play like the browser or I also think about Neo4j
Bloom for instance is everything has to be done in the query and here we can cheat because
so that I will receive will be small enough to be displayed or at least we hope so because
then if it's not small enough to be displayed no tool can actually work kind of and we can do
we cannot graphology to do things that are easier to do in graphology after the query.
It's kind of splits the difficulty. So in my case here I just I didn't want to merge things in the
safer query because it was too hard for me and I just did it with graphology after and it worked
well and it was easier to implement than if I had only one solution which was the safer query.
Scripting graphology is really I really love this tool and that's why sigma is built on it as well and
there's very very many different algorithms and yeah having this I have in this library just handling
computation this makes things so easy I think then sigma just and does yeah rendering and
interaction also when I when I display the the second view where I wanted to see all in the
indirect trades between two sovereign and tight is I actually couldn't find examples where I have
three levels I have never found any case where two and two and tight is both have a part
that trades together and I was curious but it makes sense because the reporting and tight is
are most of the time sovereign and tight is so if I come back to this one the reports I have
from United Kingdom are from United Kingdom and not from British Western Africa so that explains
why I didn't find it and I'm sad because I wrote the query for nothing.
And thank you very much. If you have question we have to be the best thanks.
Yeah yeah so the question was can I explain what's the difference between sigma j s and
cytoscapes. cytoscapes is really focused on drawing networks for schema as I'd say so they have lots
of tools to actually draw lines in more schematic ways to get some huge diagrams but if you use cytoscapes
for huge networks and forth directed layouts first of all cytoscapes is all in one
I need handles computation and rendering to my knowledge and it doesn't scale that much
it's it's a generic tools to draw network diagrams as I'd say why sigma is a better tool to handle
visual network analysis. I don't know if that's clear oh yeah sorry.
Yeah so the question was is it possible with sigma j s to write applications where we
modify the graph runtime and have animations etc yes it's completely possible here for this
I wanted to keep the demo as low code as I could but I could actually go directly to the website
you can have you can run the layout algorithm in live this is an example
this kind of animations and about alterating the graph itself by adding data I think we have
the story somewhere like hmm okay I can move and here I delete some code to say when I click the
stage it pops the nodes and it connects to close nodes I see that the story book is broken but
you have you have examples of this kind of an interaction yes
so the question is how sigma j s behaves with graphs of feldons of nodes it behaves very well actually
the thing is it uses webGL so if you don't alterate the network if you don't modify the data
zooming and zooming in zooming out and panning and rotating the application it's all done by the GPU
so if I come to this example we have an example for this so let's put 50,000 nodes and
100,000 edges if I run the algorithm this is going to be very very slow
but for zooming in zooming out no problem so if you have if your nodes positions are already
handled yeah this works very well it's it's webGL too the labels are rendered in canvas so
this can slow things sometimes if you have too much of them that they're asked what it is to mitigate
that yes how do you compare sigma j s with cytoscape after you're here already
as the question but no no where is basically cytoscape I think has been really designed to
handle rendering network diagrams with hierarchy with more deterministic things things out most of
the time in squares or rectangle and this kind of things and sigma doesn't handle network diagrams
very well this is more done for what we call visuals visual network analysis so it's more for
I have I have a network of things that are connected and I want to see patterns emerge by themselves
so most of the time when you see sigma it's with this kind of visually appearing networks
I think basically cytoscape is way more flexible and you can render things in way more
different possibilities while sigma is less flexible but it scales better I'd say