So, all right, and hello, and thank you for joining us at our panoptic presentation.
So, as a quick introduction, let us introduce ourselves.
So, I'm a Phelic Salier, research engineer at Ceres, so it's a small digital method lab at
Saban University.
So, it's David Guelica, who is also a community scientist, and Edwach, who is a research
engineer, but more in humanities.
So, today we're going to show you the software we've built, we've been building for
a couple of years, and this talk will be in three parts, I mean, there are three of us
will be speaking each.
So, I'm going to first talk a bit about the context, and what is it all about?
I'm going to make maybe a quick demo, even as that's maybe a bit stupid to do this
live, but we'll try.
Edwach is going to show some research cases, and then David is going to talk a bit more about
the architecture and our plugin system.
First of all, I'd like to ask, if you're here, maybe do you work with images in your
work, yes, some of you, okay, okay, you might be using maybe, we'll see.
So, yeah, what's panaptic, so this is just a quick sneak peek, like a screenshot of what
the software looks like, but I'm going to dive into that a bit later.
First, I want to give a bit of context, so panaptic is a software for today, and make
exploration of a medium to large data set of images.
We say medium, like, from a couple of thousands of images to several hundreds, thousands
of images.
I mean, we've been working at maximum size, I think, like, 500,000 images.
It was working pretty well.
We've never tried a million, but maybe we'll find some day a use case that would work
with a million.
We'll see.
So, we've been building this since May, 2023, so soon two years.
I started a loan working on a prototype, and then was quickly joined by David, who is now
the main developer, and by Edwach, who is our main crash tester.
And then, some time along the way, we use also to work with a research designer to try
to understand really the needs of the researcher and not be only computer scientists in
a corner and just building all stuff that we found funny, but really try to understand
what our software could be used to.
And, of course, everything is open source.
We wouldn't be there otherwise, and everything is on GitHub.
So, one final piece of context, I'm going to talk about the origin of the project,
what motivated us to build this.
So, we had this researcher, it's called Dershini Shigia, that you may know or not, who
was working on large data set of Twitter images, like she gathered data for, I think, almost
10 years on different political controversy, and was trying to understand how far-right
movements were using images to communicate and share their ideas.
So, they collected data, a lot of data, actually 50,000 images at the end, unique images.
And they had a problem, they faced, they had no tool to really analyze them, because it
was quite a lot to just look manually.
And also, their goal was to try to identify redundant images, which was quite hard to do,
especially when images would have small variations, like cropping, textited, and something
like that.
So, yeah, they had Truppy, which is a great tool that you may already know, but it's
really well to work with already created data sets.
And it has the lack of automatic, maybe mentioning tool or something like that, to help
the researchers to dive into the data sets, which can be really exhausting when you
have to look everything by hand.
So, we had this that we wanted to, and also we wanted to iterate, because at first,
Vergeny would ask me a lot to do some Python trip to help use Python models and to do some
computer visions on the data set, but it was really hard to do, like, some discussions.
She would have new ideas after my work, then asked new questions, and I should do new
stuff, and it was kind of long and exhausting.
So, we thought, why not create our own software to try to work with images and to implement
machine learning to this insight?
So now, yeah, it's a part where I'm going to try to do a quick demo to you.
It worked well with Olivier, so finger crossed it's going to work well with me too.
So, yeah, yeah, you can see everyone.
You have here the interface of Panoptic, with, like, a really small data set that we've
imported to have a bit of context, this is images taken from Twitter, like, six-hour
images, about the timetic of the Korean-plus-mong, which is another field of study here in
our lab by studying the images of the far-right.
So you can see the images, you can also see on the left here a column with all what's called
the properties.
Properties are additional data that we can show in the interface, and that comes along
with the images.
They're really important when you want to study the images inside their context of
publication.
By the way, if you want to look a bit more at the properties, we have, like, a table
view where you can focus a bit more, for instance, on the text of the images.
Now what's nice with these properties is that I can manipulate my data with these properties.
So for instance, if I want to, I can create groups where I can create groups of images along
the time, and I can choose the granularity of these groups.
For instance, if I want all the images grouped by months, but I can make the same way
I can make filters, or I can make sorting, and choose a lot of different properties to
manipulate my images, and really create subsets of my big data to try to focus on certain
points.
I can also, of course, I directly properties already inside of Panaptic, so let's
try and make, for instance, something called category will be, that will be a multi-tag.
So now you can see that I have an empty field beside my, below my images that I can modify
directly.
So for instance, we can see here, it's a French politician called D'Armagnar.
This is another French politician who's called Maréchal Le Pen, so I can do my small
annotation.
Now I don't want to do this by hand for the whole data set because that would take a
lot of time.
So introducing what's really make I think Panaptic interesting is the use of machine learning
algorithm.
Like, see, when you import images inside Panaptic, we use a deep learning model called
Clip to compute embeddings of these images, and then to be able to use machine learning
algorithm, just for instance, like camines, which is a way of creating groups automatically
of your images based on their similarity.
So I can click on the Create Cluster button, and you see, for instance, I will have all the
images here of one specific street, I don't know in Paris, here only black background,
which is a bit useless, here more TV screens, here, for instance, more pictures of the
street.
You see, there is a lot of virity because it's a generalistic model, and you can have a lot
of stuff in here, and I can interactively say, okay, this cluster has a lot of things.
Then you create more cluster inside.
So I will do it again, and see some sub clusters.
For instance, I will find another group here showing cups.
So it could be interesting for me to annotate this.
So I can do batch annotation, and tag the whole group, can take my property here, and
create a cups, and see all the images are now tagged with a cups tag.
Now, another way to use our tools, for instance, would be to group again by the category.
See our cups, groups, and see, and ask ourselves the questions, do I have more cups in all
data sets?
And I can ask Panoptic to do some image suggestions.
And for instance, here, I will have on the top side of my screen some proposition where
I can zoom in, by the way, and I can accept them and put them automatically into my groups
when they fit my needs.
And I will add more and more cups to try to find a very coherent annotation of all the
cups that I could find in my data set.
So you could do that with a lot of things and interact a bit with your covers.
One last thing that I want to show you is the last similarity tool could be the one image
mode.
If I click on an image, I can have all the properties shown, and also I can see all the
others images sorted by similarity in the data set with by similarity score.
Here it's a cross-in similarity.
And I can see, for instance, if I want to, I can add more marion, for instance, and tag
them.
And, up, I apply this, and I get in my groups more marion than you can see.
And again, I could ask for image suggestions to find the missing marion.
So I can add them and find all the marion marishal in my data set.
And finally, of course, I can export all the data as I will be annotated in Panoptic in CSV
format to be used in another tool.
And now, I'm going to show, I'm going to take this to Edward, who's going to talk about research.
OK, so I'm going to show you a few examples of research project, currently using Panoptics.
There can be a group in two, a circuit of degrees, for exporting large web corpora, large
digitalized corpora or film corpora.
And it is, of course, possible to imagine applications far beyond them.
So here, as cases, I'm a personally involved in the exploration of large web corpora.
At CRS, we work in part on the study of online controversies, on the construction of
online public problems, and on conflictality around cultural events, for example.
So we collect material from social networks, and weather, it's Twitter, Instagram, or TikTok,
for example.
We work with police, and in these cases, images can be a relevant entry point for exploring
the corpore, typically, when we're looking at the visual dimension of the objects we study.
So you can see some examples here.
So first of all, the challenge is to be able to work with images associated with text,
data, to work with some publications, but above all, Netflix said it, the challenge is to be
able to explore and annotate large mass of images in a few a month of time.
That's where image grouping best on similarity gets interesting.
As do batch annotation, functionalities, so that we can understand what is in the corpus,
and objectify it.
And the tools need to be fairly modular on the question of similarity, because we
don't always want to put images together from the same reasons.
For example, on the left, it was my thesis work.
I was interested about the yellow vest movement, and about the, I was looking to find the
circulation of boobs of images, sharing the exact same origin.
For example, different squid and shapes of same videos, the same video of the same picture,
posted on another social network, for example.
And on the right, this is a work on the spread of the raticity
ideology of the breach replenishment, and this time, the problem is quite different,
because the idea is reserved to detect similar objects.
So for example, political personality, image of TV sets, or
semi-autic material for a political communication, for example.
And we also recently worked with the BB Attack National
France, the French archives, which is, there is the corpus,
have different, because we work with objects or images, which are photographed,
then digitalized, and the problem is the same to a group
images in function of the similarities, but the difference is about the noise.
For example, when you are in digital methods, the problem of the noise is in the construction of
the corpus, but not inside the images.
And it's different in digital humanities, or there is no noise in the
construction of the corpus, but you can have noise in the images itself,
because for example, you can have backgrounds, or no backgrounds,
when you take the photograph.
So this can affect the ways of digitalize them, put the images together.
And one last example, we have a doctorate student, Léonolfi,
who work on identifying this well-code in costume films.
She is, for example, interested in repeated shot composition in
her corpus.
So what we do is cut the films into images, take every five seconds,
and we imported them into panoptic.
And once in panoptic, we are able to find similar shots and
the visual codes of the film genre, along all the films,
and the almost 300 screenshots taken.
I think, and I will let David speak, what we need to draw from this quick presentation of examples,
is that there is a wide variety of reasons for exploring images,
which implies adding a tool, a lot of great modular things,
a use of similarity algorithms, that will never be totally adapted to all the types of images,
and to all the questions we can ask them.
So we don't have a lot of time left, so I will speak quickly about our architecture.
So the storage is just an SQLite database, the backend is Python,
and UI is a web UI, and we see here that we have loaded our plugin with
the key factors in the face index that can communicate directly with the backend,
and also insert data and the database if needed.
So for the SQLite database, the advantage is that it's very easy to install,
and also all the data is in one file, which means that if we want to share
your panoptic project, you can simply copy the file and send it to another person,
which can import it in the panoptic app, and it will work out of the box.
And for the backend, we choose Python because it's easy to develop plugins,
and it's a scripting language, so it works on every operating system,
and also we have all the Python capabilities, which makes it much easier to use machine learning algorithm.
For the front end, the idea was that HTML and CSS are well known,
we don't need to use any UI library, because everyone already has a browser,
and also we want to allow remote work as our next goals,
so a browser-based approach sets the foundations for it.
And now the most important part is the plugins, which are able to customize
functionalities of panoptic, and we have three main actions, which are the clustering,
the similarity, and also a more global execute function.
And to give you an example of a personalization, we had to do in the lab,
is the clustering of memes, and memes usually have always the same image,
so clustering on image similarity alone is not very useful.
We first have to extract the data and then do some special function that will cluster using
the image, and also the meaning of the text, and to show you an example how it looks like in the UI.
Here we have the create cluster button, where we can choose our clustering function,
and in this case it's our panoptic ML compute cluster function that can also have some parameters.
For example, the vector type and the number of clusters we want to do,
and we can see in the bottom we have the signature of the function we define,
and the parameters we give to the function will be shown in the UI, which is very convenient.
At the same time we also have the similarity view, so for example here we have two images that
use a different similarity function, and at the top image it's used the colors to find similarities,
so we find the blue butterflies, and at the bottom image we don't use gray scale image,
and we see that the results are different in the UI, and if you have special needs you can adapt
your function to show it differently in the panoptic UI, and the last action is the execute action,
it's more global action that is basically a way to execute any scripts on a collection of image,
and the idea behind this is that you don't have to go out of the UI to execute scripts,
generate data and import it back into panoptic, but you can do it everything from the UI,
so this was the quick overview of the technical things of panoptic, and if you have any questions
if you're free to ask, and thank you for listening.
Yeah, let me put it there.
I mean, can you repeat the good? Okay, so, yeah, so if I understand what I'm talking about,
oh, can you repeat the good? Okay, so, yeah, so if I understand correctly your question is like
can we trust the similarity score that's shown in the, yeah, okay, so the similarity score,
I would say it will depend on your data set, because sometimes like similarity score of like 99%
will be like really, really similar images indeed, but it will also depend on the coherence
of your data set, if you have like very variable images, then maybe the images that are really,
really similar will have a really high similarity score, but if all the images in your data set
really look alike, then maybe 99 won't be that representative, so yeah, you need to adapt this
to try to understand the similarity score. Other questions, maybe? Yeah, I should show
from the question that was just asked, are you aware of any studies using this tool,
right, so different data sets, that's a very informative, so we're interested in chemoinformatics,
because the way you build this embedded system and human-assisted exploration of the
data sets, and so the UI, it's very interesting. Okay, so the question is, are we aware of
other scientific like biomedical or bioengineering, like that research, biology chemistry research,
that are using panoptic, the answer is right now, no, we work mainly with digital humanities
and media studies, but it would be really interesting to try to apply it to new fields. I think
David had the contact with someone's working on a natural park, so it's not really biology, but they
were trying to study, so photo traps, pictures of animal at night and stuff and trying to categorize
them, and we're trying to write now to use that data set, but we don't have access to it right now,
but I don't see why it wouldn't work, actually, especially since you can eventually
use your own model, if the current use is not specified enough, so yeah.
Yes, I was wrong, it is an off-dig in taking a meter data into account, like to let's say
it's not possible that a Tesla is on the image that from 1991 is such a thing,
so the question is, how strong is panoptic to taking a data into account, for instance,
can we have a filter could be like, it's not possible to have a Tesla, which was posted in 1991,
that's correct, right, that's your question. I mean, the filters you define it to yourself,
so you can, but the filters are not used to compute the similarity or to do whatever you want.
I mean, if you want to filter all the images in your data set that are later than 1991, you can,
do it if you have the metadata, and if you want to cluster right them or do some analysis inside them,
you can, but you need to do it manually. I mean, it won't be automatic to detect fake data
because like it's an image from 1991 and you have a Tesla on it, so it's fake data to fake image.
That's something that you need to figure out yourself by using the tool, maybe by finding
all the Tesla in your data set, and then grouping by dates and seeing if you have Tesla images
that are in 1991, that you can do yes, but it's not full-years metric.
But you can do also a plug-in that could do that. Yeah, yeah.
Is it possible to use a weak plug-in UI logic, like for example,
suppose I extract something on the image, text for the previous thing, or objects,
but I visualize it or take it and you know how to make what has been.
So the question is, how easy would it be to visualize a data that I would extract myself in a plug-in?
That's right. Yeah. So yeah, especially for OCR text, it's actually quite easy because like in your plug-in,
you can create a new property, which would be called OCRized text, and then you would have access
to this property in the global UI. So you could see your text as a property and you could work with that in the UI.
So it wouldn't be a novel, we don't have the layers, we don't have the layers,
but we are really thinking about that, especially for objects' instructions,
that could be quite useful for identifying some part of the images, but we don't have that right now,
but we are really thinking about it, and I want to do it. I think there was a, do we have time?
Yeah. Yeah. I've been using penaptic actually on the database of the data. Really?
20, and it's cool, it's really great contracting with it, but I had the deployment issue,
because I was deep on a GPU to be able to compute it, should they print on a server with GPU?
And I would like to propose it to users and users, and I don't want to give the access to my server with GPU.
So how easy is it to take this, kill it, and have the only available property for instance,
and put this on some of the server where only the corporations would be computing,
but you don't need as much power as a calculation. So if it's a scenario, that's going to cost a lot.
So the question is, is it possible to share a project easily from a server where everything was previously computed,
and then to share it to a new user who don't have a GPU?
First, I want to say, this is a great question, and it panetic really works on PC that don't have GPU, actually.
So, a bit of context. But then it would work for you. I mean, you can embed everything in the
single SQL file, even the images. I mean, you have an option where you can say, can you store
the images directly in the SQL file, so that's really easy to share afterwards. So if you have
another panetic, maybe on another server or on another computer, you can just take the SQL file,
so the panetic points DB, and create a project and import it in your new panetic instance.
So yeah, it's maybe you will have some bugs, and you can write us, but it's supposed to be working
from scratch. Yeah. Yeah. Let's take a.