Hello, I am Dorothy Benamu and I am working at the National Library of France as part
of the web accounting team and this presentation is about how we manage the tension between
hosting large amounts of copyrighted data with limited access and meeting researchers
needs to explore these data with open source software.
So since the early 2000s, the National Library of France has been collecting the
French web, that is to say that we regularly collect samples of the cultural production
that are online and which are, as you know, very ephemeral, so this allows researchers
to study the political, social, scientific debates that either primarily take place
on the web or at least find significant echoes there and to study as well the major
transformation that the web has brought into a rare aspect of our lives.
So the B&F is authorized to do this under a low from 2006 and what does it mean to actually
collect the web so we use software web crawlers boats to do so.
We are using open source tools that we develop together with other institutions and
organisms that are part of the international internet preservation consumption, the
AI PC and so you might be familiar with the internet archives way back machine so
internet archive is part of the AI PC2 and we use the same kind of tool for accessing
and browsing past versions of the web.
The main difference is that we focus on the French web so we have more comprehensive collections
of websites, hostility or produced inference.
So what is the scope of the collections of web archives that are available?
So obviously we cannot preserve everything so we try to collect representative samples
regularly of the French web and our harvesting model is a mixed one combining two types
of crows.
Once a year we run a national domain crawl or a broad crawl for this call there is no selection
process.
We are collaborating with diverse organisms to gather lists of domain name that are hosted
in France and for each of these domains we are collecting around 2000 euro.
So last cherry representatives almost 6 million domain names and complementary to this
we also have thematic or curated crows so these are thematic selections made by a network
of librarians mostly but also researchers and associations within their fields of expertise.
We have selected websites that are harvested more in-depth and more frequently so fragrances
ranging from daily to annually.
They cover all disciplines, literatures, sciences, history and we also have a thematic crows for
example for the COVID-19 pandemic or for the European Games or for the electoral for the
elections.
So to develop this example for each major French election we collect the electoral debate
around the election.
It means we select websites, blogs, social networks when possible and diverse online content
from political parties, from unions, candidate associations, sciences, tumourist bolsters
and from any cities and expressing themselves on the internet about the election for example.
So how do we make this data more open to scientific research?
So these are great material for researchers but there are several limitations on those collections.
So the first one is that due to legal restriction, due to privacy and copyright concerns,
the web archives can only be accessed on-site at the B&F in the research reading rooms
or in a network of 20-regional library or questions that are specifically listed in a
decree.
So this is why they can be considered as close data and as a major challenge is that there
are massive data so we try to provide specific tools and services to allow to explore those
data such as Pondore within the B&F data lab and finally there are digital artefacts
so there are traces or recalls of what was on the web at a given period and if researchers
want to use them for their research they have to understand how they were constituted
the intellectual and technical choices that were made and have to deal with the biases and
they have to deal with the incompleteness, the multiple version of the same page, maybe time
and consistencies etc.
So making them more open for research remains a major challenge and this is one of the
issue that Pondore is addressing and I will let Gium explain you why.
All right so hi everyone I'm just going to try to stick that right here if it works.
My name is Gium Luvoye I'm a political scientist I'm a researcher and as such well again
we've talked about this earlier today already for me the practice of doing research is about
building a method that serves an epistemological goal trying to build something that we try to call
scientific knowledge. The thing is today that label is very rarely enforced what we call
scientific knowledge in the common speech is usually things that you find in articles that are
in peer review journals but what that means defecto is often left and said as in the reviewers
that are supposed to vet the research usually don't have access to raw data or to the method.
So it becomes more and again it depends on the discipline but it becomes more about the
plausibility of a narrative rather than the actual work. Hence the need again this is why we're here
for two that are free open source with intelligible source code whose execution can be decentralized
when it's possible and whose outputs are in the control of the user and then that's how we connect
with the previous presentation. We hope that having both the process and the tools available
over a long period of time enables to give us a better chance as being able to reproduce the work
and hence the very relevant question we had on the hardware is through that sometimes the hardware
can be a big issue but since it's the best we can do given the means that we have.
But as researchers we're always trying to reach all available data sources for once research
which collides a little bit with the idea of being reproducible and accessible to most
which is why I started building Penderway which is a software that does
three things or has many processes that can be abstracted into three things harvesting data,
standardizing data and exploring this data. So basically how it works is that it has a first
process called flux that is basically connected to different types of APIs that it calls in a way
that is respectful of the API in order not to drown it. Then so the data it queries has to be
abstractable as documents to be poured into the terro which is useful because researchers usually
even if they're not very good at using computer they know how to use the terro and the
terro is both a web service it's a desktop software it's an SQL database behind it so it provides
a lot of tools and all documents can have attachments as well and notes so it's quite a powerful
open source software and I think everyone is thankful in the scientific community for the terro
and types is a series of data vis systems that are calibrated to explore
copies of documents that rely on D3JS. What does that what does this mean in the context of the
web archives at the national library of friends so the BNF? It means that we need to first
identify that when you're on site when you have access to those to the archives that again are
protected you need to check whether there is something in the archive that's interesting to you
and then you do the full circle of querying the data abstracting it into the terro which is tricky
because you cannot get the data out then visualizing the corpus and then having the opportunity
of looking at each capture of web pages that you want to explore into the web archive browser.
So this is the tension that we had that the whole thing mentioned earlier it's open source
but it's closed data of course we're not the first people to have that problem but we have to
negotiate the specific constraints that we have both in terms of law in friends and in terms of
technical capacity from that institution to be able to provide access to such data so quickly this is
how it looks like it's the form it's a search engine you check that you have a number of results
that's interesting to you here I was looking for final in the 2002 election in France
and once in you are in that software in Pandora it detects that you are within a network that is
authorized gives you the opportunity to do the same request on the same data set tells you that you have
so this is a different request actually it's on dolly the little sheep it tells you that you have
a request under captures that are relevant to your request over time in the collection that's
relevant and gives you an example of the kind of websites that contain those terms because those
are full text indexed collections and then the solution that we found is that we sent only the
metadata to Zootero so we did not upload the actual content of the fields but we only selected we
remapped the documents and selected a series of metadata not all of them that were relevant to
to the corpus and so here you have an example have the website title the host the date
and we used to pass a JSON object stringified JSON object at the short title because it's
common to also Tero documents but now we cannot do that anymore we used to be able to do that so now
we upload notes instead and then we re-stand it back to paint a ray when you can see it as a corpus
here you have in blue the captures and the links towards documents that are different and so when you click
a link if you are within the bnf it sends a query and gives you access to the full content so it tells you
how many captures there are over time and you can look forward within the content so it tells you
and it gives you a short exert so you can have an idea of how relevant the term you're looking for
is in the context of that page so the way we found to conceptualize this kind of relationship
that enables us to build open software with in that instance close data is the one way mirror model
so in this model there is we consider the web archive as a bit like a suspect in an interrogation room
and they have a one way mirror on their side so they only see themselves and you can ask them
question so we build that model on metaphor based on three properties one is that you can harvest the
data when you're in the room but you cannot have right access to it so you can talk to the person
which you cannot change what the person knows you can take parts of each record so here the
metadata and not whole thing but only some fields that we determined but you cannot send requests
when you're away you have to be in here to ask the questions and you can come back for highlight
on a specific piece so that refers to the that part of the exploration when you click on a note and
send your request information on a specific note but you cannot take the whole body out so you cannot
ask for the whole corpus to to be extracted because there would be a risk that then you would just put
it on a USB key and walk home with it so that's that's how we build our model thank you for
attention we'll be taking questions yep hi so how much of this is opposed by right holders on
a continual basis so you always fighting the problem that you have to be careful as to what
people can take out of the closed room yes yes oh yes sorry the question I should know that so the
question is how do we manage the fact that everything is under the rights of the creators of the
authors and how much we have to fight that so there is a context that is still to be I guess proven
by practice but basically the idea is that you don't have you're not justified in taking anything out
of there except if you're a researcher and you need to take an excerpt to show a good faith that
the argument actually sends back to something that is empirically in the database so if someone wants
to if a researcher reads your paper and wants to know more or to check that what you're quoting is
actually accurate and you have that number of documents that are in the family of the phenomenon that
you're trying to describe they can come to the NF and use hopefully the same tool on the same data
and find the same results so it's extremely coercitive in a way there's no way of taking out the data
you're just you're supposed to be able to to quote it and that's the liberty I took as a researcher
to just show you those little sentences also I think this is still on the live web so you might
find it but so yeah it's very it's very narrow and it's a problem for research but for now
it's how it is I think maybe tell you one more question otherwise let's take this speaker