Hello, my name is Kasar. I'm a little bit stressed out about the time limitation. I've
been warned. I'll do my best and I'm going to tell you where to find more. So I'm going
to tell you about the project that I've been working on for several last months. It was
project with a community on open datasets for LLM training. This project I've done
it with a colleague at Mozilla, but also in partnership with Aluter AI. If you don't know
them, they are fantastic in the Penden AI research lab, open source research lab, and Sebastian
my service was part of this, he's sitting there standing up now so that you know that
you can bombard him with questions as well. It's much more technical than me if you
have any afternoon talk. So this project is part of a bigger series of efforts that Mozilla
as doing to create space for conversations with the leaders of open source AI space.
Because if you went to the keynote of Mitchell this morning, you definitely are aware of
that there's a lot of disagreement, different opinions. How does that actually
would open source and openness mean in the context of AI? The definitions are
flying around and so on and not everybody agrees, but we all part of a one
bigger tent. Mozilla wants to create space to have discussions to ask the right
questions and to find answers one by one to the Tony questions and possibly create
some common artifacts like definitions and things so that we can start looking
in a common direction because the kind of you know that the fight that we fight
in the enemy is really dominant and huge. So I wanted to tell you a little bit about the project.
We brought people together to talk about what are the challenges of putting together
open data sets for AI training, like what is the way forward, what are the successes
out there and what are the recommendations, where can the investment come. It's a story of
community, it's a frustration in the community struggle, PDFs and a lot of successes and
hope as well. So what's happening right now? Right now, no one almost no one releases
data sets that been used for to train AI anymore. And that's open AI doesn't do it,
but that doesn't do it, Google doesn't do it. We talk about open source models and so on
even deep seek, as you probably know, we talk about that is being an open source model,
but the data set that they used to train it is not revealed is natural use, it's not
even open in any way as you see later according to one of the definitions we have here.
And while the data, of course, is the fundamental element of AI. So what how did that happen?
In the beginning of January, January, January, January, July explosion, so on there were
still some data sets being released by Lama One or RT5 by Google. However, as you probably
are aware, the copyright, outrage, the copyright loss was that followed with a lot of
not be happy with data being scraped by the big companies and exploited to train
a hugely successful generative AI. This copyright lawsuit created a lot of legal risk and
a lot of fear in that industry from releasing data sets. So most of this big company
stopped releasing the data set because the lawyers basically told them, that's just to
be not doing that. That, of course, in addition to the competition pressure in the industry.
And that pertains both to the big companies as well as small research lab as well.
There's a lot of legal risk in releasing the data. However, of course, we know
we are here in the open source community. There's a lot of advantages to open data sets,
to open data sets, bring about competition, so that smaller players can build on the data
that is out there. It's about accountability and transparency. And of course, also
research. So we can, and we have to do better. So what we did at Mozilla, we organized
this data set convening in June 2024 together with a literary AI when we invited
people who are actually building the open data sets who struggle with that and who succeed
in that. And around 30 leaders of that field from a range of organizations here,
you can say, how are you hugging face, a grattle of a synthetic data.
Play us that was featured here before mentioned as part of the builders,
product spawning, really great organization as well. So a lot of people,
we even like reached, I reached out on LinkedIn to people and everybody was very
eager like, yeah, let's meet and let's discuss it because it's a real mess and we
trying our best but it's hard. Although, as you can see, we're very happy at the end.
And as a background for that, we interviewed a Lutter AI,
Stella Biderman, who was leading the building of the open data sets at a Lutter
and play us as well, the French organization to understand what the challenges
are and to have a common background for these discussions. Based on that, we created
the research paper that you can access later on and I'm going to,
it's just made out of the community insights that we derived from that workshop,
but later on also like asynchronous collaboration.
And I'm going to give you a little overview of what's in that paper,
of course, it's just not that much time and I'm scared already.
So it's just a short overview. So first of all, that used to be a graph.
So all right. All right, okay. Well, it appeared.
So we try to order kind of the space for us. What do we mean when we talk about the open data sets?
And there are these three tiers of open openness and data sets for AI,
starting with the sufficient documentation as the replicable data sets.
That aligns with the open source institutes definition.
All right. So you need to document data sources and the processing steps of somebody
could replicate as substantially similar equivalent data set based on that.
And these are the data sets such as CR4 and C4.
The second tier is the open access, so the data availability.
So the data set is out there for everyone to download,
but that doesn't make any claim about the licensing of it.
And the third one, what we call here, the fully open is there are all three elements.
And there's the legal side of the usual open data definition that we know from open knowledge
foundation, where you can reuse, share, and modify data.
And that pertains both to the kind of licensing of data sets itself,
as well as all the components that go into image text and so on.
And then of course, we didn't only talk about open because open data sets alone are enough.
There is also the aspect of like what makes the data set fair, just equitable and ethical and also
compliant. And this is something that is important to remember that this has three different notions
that sometimes are even attention with each other.
They intersect, but sometimes you have to make decisions as you build the data set.
For example, offering and going opt-out might go against you wanting to have a stable version of the data sets over time.
So a router and play has told us about a lot about like what are the challenges right now.
If you try to put together an open data set that just made out of openly licensed content and public domain content,
and these are a lot. So a lot of stuff that's making the data have exploded every day.
First of them is that loss across very across jurisdiction and time.
So if you try to assemble an open data set that will have global implications, you need to look at multiple jurisdictions and geographies,
but that also change over time. And that requires usually legal expertise from different lawyers from across the globe,
which is of course very expensive and very time intense. There is also a very big challenge around the data data being incomplete.
So what constitutes a work in a copyright law doesn't always translate so neatly into the different components.
For example, if you have a few crawling automatically website or common crawl across it and see that there is a CC license on a website,
but in an automated way, it's not really possible to say if that pertains to the whole or the assets of the website or is it just an image or a text.
So you can make money mistakes there and make yourself legally vulnerable.
The same problem with the public domain where the status isn't always so clear and you need to do a lot of manual digging,
which I know that Sebastian is really spending his days and months with manual digging.
The same goes, the other problem is just because the document is actually in the public domain doesn't mean that you can get a copy of it.
A lot of stuff cultural institutions and so on aren't digitized or even if they digitized, for example, the project of Google Books,
you can't always get a full public access to it because it requires different arrangements with Google and so on.
The problem with PDFs that I also learned from Sebastian about is that extracting data from PDFs is extremely difficult.
There are no real tools that help you do it in a scalable way and it requires a lot of manual labor, which of course requires a lot of resources and people and so on.
Many, as you know, many open source projects that made out of Google and tears, they don't have a legal entity.
Being exposed to legal risk requires a lawyer, a lawyer, a lawyer, a legal entity and so on.
That's another problem that arises in this AI context and finally there's something called the consent crisis right now where people don't want to,
even originally, open data, they don't want to, the AI scrappers to scrape the data because they are, of course, annoyed with, with data being used into the big data sets.
And that is directed at the big company, but at the same time, for example, robot, robot that it takes still and so on, that blocks out also researchers and independent developers and so on.
But of course, we don't want the big tech to exploit the open data that is out there.
There's also that kind of problem that we might maybe use with a solve with an AI common that is set up in a right way.
So we spent eight, really intense hours together.
I have to say the food wasn't really good.
Unfortunately, but people were amazing and we had a lot of discussions around the pipeline of producing such open data sets that a lot of exchange.
And we came up with the back space practices that are really grounded and what people are actually doing, the different organizations building the different open data sets.
And I'm not going to go through all of them because that maybe would be too much by inviting you to check out the paper in detail.
But one thing that comes ahead of everything is really encoding the preferences in metadata, the problem of not being sure if something is licensed or what is the preference of the data owner.
Going forward is that it doesn't allow building data sets in a very scalable way.
And other than that, there's a lot around working with communities, around a lot of documentation, making the open data sets reproducible and so on.
There is also a lot of emerging examples.
Alluterious building players building common corpus that are already training, training models on it.
They also have a so-called toxicity classifier.
So an open source, the pipeline of how to identify toxic content.
The same hugging phase that I do, I do web classifiers, also open source pipeline.
They just like open source in the methodology of like how to go through the data sets and remove the harmful and toxic content.
Of course, documenting the definition how they defined at harmful content.
And so on, there's also experiments around data trust of how to organize the governance, around data for communities.
So they have say about what's happening with the data sets and so on.
This is also an amazing organization working on data governance, letting people opt out of the data set and then creating API for developers to run through the data set and remove that data from it.
So a lot of great examples, we have even more of them all with links in the paper.
And finally, we also identified what needs to change in terms of policy and what needs to change in terms of tech investments.
But we can move forward as a field.
And I identified here three main points, so to say.
So increasing open data availability, again, here, making it easier to identify the status in a reliable way of public domain data having maybe registries, working with cultural institutions,
in partnership with cultural institutions to digitize data and establish the metadata for it.
A.I. data commands is something that's been discussed all the time by, like, who can finally do it.
Anyway, that is reliable and such concrete things, as I mentioned, the tools to extract,
to open the license content from difficult formats like PDF.
And of course, clarifying the legal status of the data.
So one proposition there was an policy makers who could create a safe harbor for, especially for smaller organizations such as a Luther AI,
that they can make some mistakes around license things.
But they don't need to feel that immediately they will be slapped with loss.
And of course, that they don't have the resources to fight.
So having a safe harbor for that would help a lot, as well as invest in this global metadata standards to manage licensing and consent and scale.
And here the consent is meant as a new one's consent that goes beyond the robot takes the crawl, not crawl, but, like, for which purpose can this data be actually used.
And finally, as always, the money, ensuring the sustainability of the ecosystem.
So if we want the open datasets for AI to be out there, to be really meaningful, so that we can build open source AI on them,
they need to be treated as public goods.
And that means they also need to be financed as public goods with a long-term perspective of sustainable funding.
And alternatively, also thinking about sustainable business models here.
So what would that look like?
That's always a problem, I guess, in the open source community because things are open to use.
You can't really sell them, but there's a week media experimenting, a week media enterprise from what I know.
And then there is also spawning thinking about the premium model and thinking about how to give back also to the communities that are giving out data and how to develop something that is more sustainable, so that we cannot move forward.
So that's very short.
And thank you, this is the QR code.
That would be too much to type.
If you want to check out the research paper, also if you want to contact me, I'm actually living on Zilla,
but I'm connected with that project and I can answer your questions on direct you to people at the company.
Thank you very much.
I make it.
Do I have questions?
How do you plan to handle the attribution requirement for open license content?
Yeah, I have the microphone.
Yeah, it's a true.
I've been thinking about it, maybe Sebastian also can give his opinion on that.
But I heard about people actually giving attribution on mass listing or one as part of the documentation.
Of course, or linking to this kind of repositories, but I don't know if there is, I think it's like one of,
still maybe some of the unresolved issues, but Sebastian, maybe if you have an opinion of how to handle that at the letter.
It's one of the hardest problems, so let's talk about that.
I think you know, best practices right now when the status has released effectively mean the part of the parquet file, when you have a data set or something.
It's just another column, right?
But is that enough?
Is that sufficient or something that the community really needs to talk about?
But we really try hard to get attribution for every single item that is licensed under Creative Commons licenses.
Thank you.
Do I have more questions?
That was great, thank you very much.
So quick question.
Are you going to turn, I've not read the report, but I've got to turn it into something machine readable, so that another machine couldn't fully understand.
And that means that new providers, not the ones that just talk to you, can also implement it as well.
Machine readable, can you?
So everything that's in there described what the intentions are and stuff like that, it's machine readable.
Yeah, the metadata being machine readable.
Yeah, so the machine couldn't also read it.
Yeah, yeah, no, that's, I think that's the whole crooks on it that it's not, it's requires manual labor, so the trick would be to make this.
So you can pull the data that you need and be sure about the license, sure about the preference signals, so that it must be machine readable.
Otherwise it's just like picking or going one by one and checking and requiring a lot of resources time.
And that we'll never reach the same level as OpenAI and so on without that.
To waste more questions.
And I don't, thank you very much.
Thank you.