Hello, good afternoon to everybody here. Good morning or hopefully not good middle of the night to folks out on the internet.
I am so obnoxiously excited to be here with you all today.
I'm going to talk to you about multi-lingual speech technologies for a global world, which is a bit like saying ATM machine, global world.
It's a world world, it's kind of the point.
My name is Jess and I work for the Common Voice Project over at the Mizzilla Foundation.
I am absolutely needlessly intense about Birmingham being the best British city.
For folks watching on the video, I'm the only one might, but somebody just like explained it to your agreement.
I'm also really excited about communication and human languages.
I'm really bad at learning languages and I have no self-control.
Please, please do not talk to me after this and tell me anything interesting about your language.
You will ruin my summer. Please, please, please don't.
We're going to be talking about data and models and multi-lingual models.
So the first thing I wanted to do was just go ahead and make the nod to AI hype.
Are these models going to change the world? Are these part of Gen AI? Is this the most important thing?
Are they going to replace programmers?
Joyously, this is so far above my pay grade that I just get to sort of nod and bump along.
In a non-editorial way though, I'd like to suggest that maybe we could talk about something practical and useful.
I want to talk to you about multi-lingual speech technologies.
And I want to talk to you about speech technologies because hopeful technology in 2025 feels a little bit rare.
So I'd love to see if we can all be excited together.
If any of you are eventually going to be building some of these, I'd love to kill any excuses you may have to not make them linguistically inclusive.
But for those of you who are using these speech technologies, I'd love for you to be really, really aggressive out in the world about why doesn't this understand me.
And why doesn't this understand people who aren't like me?
Speech technologies are so boring. They are joyously boring.
Almost all of them have terrible diagrams, and this was my favorite one which is quite good.
A research team of Arshalan, I'm in Ramsey and Allen.
Did this really fantastic one, which I think walks us through how ASR, automatic speech recognition,
and speech to text, turning speech waveforms into a type of input a computer can recognize and act on.
So we've got a bucket because that's how we like our data, just in a cylinder of transcribed speech data.
So that's going to be clips of people talking and then text associated with that.
And that gets built into a model to say, hey, when you hear these waveforms, it's associated with these words, this orthography.
And then we've got text data that also feeds into models for what we think that's likely to be saying, and a model for pronunciation.
So in this dialect, in this language, how are things pronounced?
When a speaker comes to a speech technology, the technology will extract out the features.
Usually, we'll get a cool look at the shape of those words.
And all three of these models are going to feed into a decoder, and the thing I love best is, human speech is messy.
Human speech is wild, even in the best of times.
And speech technologies, automatic speech recognition is going to spit out hypothesized text.
What we think that person said,
is anybody Scottish?
Cool.
This works better or worse in some languages.
And this works better and worse for some accents and variants and dialects.
I'm not picking on Scottish people.
They just seem to have the hardest time for speech technologies just really aren't voice assessments.
When I give this talk in other types of mainstream conferences, I love to ask,
like, hey, who uses those little things that live in your house?
And totally don't listen to you when you don't use the key words.
Who uses speech assistance, voice assistance, or those, oh, oh, cool.
I mean, I'm, yeah, me too.
I would definitely do the, so speech technologies are boring, but they're also not just
Siri or Alexa.
They're really, really things that are often exciting.
So they're not perfect, but real time interpretation applications are magical feeling.
I've seen them use together and share livestock disease information for pastoral farmers.
Use for remote medical services platforms.
So the ability to say, hey, I'm bleeding really bad.
What can I do?
It can be quite good when you don't have both hands free at the moment.
Used in assistive technologies, which I absolutely love.
Power is the automatic transcriptions.
We're seeing more and more on videos for whatever value that's worth.
And really at the end of the day, any time you talk to a machine, we're seeing ASR or speech-to-text at work.
I'm hopeful because this is huge for lowering barriers.
You don't need literacy skills.
And all of us hanging out in well-developed Western European city.
We know, oh, people who, a lot of people can't read.
This could be due to cognitive challenges.
This could be coming from a background where there's less literacy.
I kind of don't care what I do care about is people deserve access to this technology.
And if we can remove literacy as a barrier and let people interact through speech,
at least to me, that's so dazzlingly exciting.
You don't need to use your hands either because you don't have use of them now,
or you're doing something else with them.
You can use them.
You can use speech technologies while probably not driving your car, please.
And thank you.
I'm not sure about local laws, but while using equipment machinery,
while doing other things, and speaking requires less directed focus than writing.
If I have to type something, I mean, a screen in my field of vision.
I should probably almost definitely not do this while driving or operating machinery.
But I could probably use speech technologies while chatting with you all.
And interactions with speech technologies mirror the ways many of us communicate with each other day.
This isn't a main point, but the ability to talk to our computers is retro sci-fi magic.
They're probably not going to love us back, but the brief opportunity to feel like they talk to us is pretty cool.
And linguistic include exclusion, so saying,
hey, these speech technologies work, but not for everyone has huge impacts.
When we're seeing people come to the web for the first time, whether in speech or text,
we're seeing people join us in equal hardware, in equal connectivity,
but we're also asking users to join us, often a second or third, or in a language they don't primarily speak.
At this language is, this is an English spang on about all the time,
but languages die every day.
And the languages that die are the languages that don't get used.
The languages we use are the ones we include in our tool line.
And accent and variant-based barriers to speech technologies mean that they develop additional barriers based on class and race and region.
Speech technology isn't just voice assistance, but voice assistance are a fantastic sort of lens through which to see what works and what doesn't.
How many languages there are and what's a language, what's a variant, is kind of the tab's versus spaces of linguistics.
But there's definitely more than 7,000 languages in the world.
Voice assistance work really, really well with about 20.
I'd go ahead and gather that we've got more than 21st language speakers of different languages here today.
This means that the datasets that power them are coming from backgrounds where they underrepresented and lock out people of color,
people from indigenous backgrounds, not just different language backgrounds.
While I'm super excited about speech technologies and how hopeful they are, they're not being built for everybody right now.
When we saw this diagram, I was pointing at all of the things I was excited about.
One of the things I'm also excited about is this research that produced this diagram that I love is actually about the impact of gender and dialect and training size.
Gender and dialect in the training data for Arabic automatic speech recognition.
And this is broken because of the datasets.
When we come back here and see that we've got this transcribed speech data that begins all of our processes for doing automatic speech recognition,
that data not being there or not being included for these models to be trained kind of breaks a lot of things.
Almost all of these datasets are proprietary, close source, and they are expensive as hell.
Often they're limited in demographic scope.
If you gave me a project why I needed to collect 17 hours of Belgian French data right now,
I'd probably run a university campus.
I'd probably go talk to younger students who are chill with casual work.
And often times when we do get regional languages, these are being collected by folks from outside the region who don't understand the language, and are going to wind up with weird data.
I can talk to you about what we did, but in doing so much less of a hey use this completely free and CC0 dataset pitch.
But more of a let's talk through the different aspects of the data and the different aspects of linguistic data that I'm really excited about.
So what common voice did back in 2017 is start collecting speech data via crowdsource platform.
The big thing I want to stress is this is not our data.
We never had a language unless someone from that language with community asks us, and we see ourselves as librarians of it.
We release that data every quarter under CC0 license because it belongs to the people who made it.
If you did want to collect your own data, let's kind of walk through how we did it and how we've been doing it.
Somebody asks us to add a language right now we've got 131 languages on common voice, which doesn't really stack up to the 7,000 I was shouting about earlier.
Because people are donating their voices on a website, we then ask people to help us localize the platform.
We ask you to donate your voice, and I ask you to agree to the terms and a language you don't speak, that is exceedingly uncool.
Because we're a red speech corpus, we also need copyright free sentences for folks to come to the website to read in their home language.
We launch a new language, we have a little bit of a remote party in the office, it's fine.
But people come and contribute their voices.
Other people validate them because that data needs to be validated to be really valuable.
And then because it's not our data, we just go ahead and kick it out into the world every quarter.
If you thought, I want to do this, I want to go ahead and collect my own data, love it, love it, love it, do it.
Let's walk through all this stuff you got to do.
This will absolutely show exactly how old I am.
But common voices gone for a CC0 license, which is the most yellow license you could possibly source.
It means that folks can package the data, resell the data, use it for whatever they want.
We thought this was a really good fit for our project.
The big thing I want to shout about for folks looking at speech data collection is, this is not a good fit for every language community.
We've talked to folks from indigenous language backgrounds saying, how do I keep big tech from accessing this?
Say fantastic, not by using a CC0 license.
So really looking at where the data comes from?
While I could not definitively speak to it, when we look at a lot of big pre-trained models these days,
there's a lot of supposition that they're trained on non-consensually access data.
So thinking about, I'm using speech technologies, where do I want to get my data from, and is it okay to use the data?
Is something that I really like to encourage folks think about?
Think about the age of the data, the speakers in your speech data is massively important.
For common voice, we ask people be adults to contribute their voices for a range of different ethical and legal reasons.
And this is true of a lot of open data sets, but this means that speech technologies overwhelmingly, working incredibly poorly with young voices.
Speech technologies also tend to work very, very poorly with the elderly, also because these are underrepresented demographics.
Right now, we've got, so license, age of your speakers, literally just the languages you want to include.
And this is massive. Right now, we've got 130 plus. I love to ask folks, and y'all have been chatty so far, to guess which language you think we have the most data for.
I don't know y'all can y'all, I mean, that's fine.
Oh.
Oh, okay, with that English Spanish Chinese, Hindi.
What?
Yes.
Okay, look, it's cheating, if you know. So we got a lot of really good and usually folks guess English first because tech tends to optimize for English data first.
But we found that folks from research and language communities tend to be really, really passionate.
I can't pick favorites.
But folks like the cut-alone language community, the Welsh language community, have just been really, really passionate about.
Hades are languages. We want to make sure we can use tools that represent it.
I'm also a little bit active in English.
Yeah. Sorry, I'm trying desperately not to get sidetracked, but I was like, yeah, did you know?
No, no.
You haven't been doing that for an instant longer than?
Maybe.
See me after class.
So we don't just have to think about the language.
And one really exciting thing is if you open this up to communities and say, hey, what's your language?
You are going to get a ton of very interesting controversy around standards and what is a language.
Love it, love it, love it.
But also variants.
So the way I speak English may be completely different from the way someone in Glasgow speaks English.
Major language variants can be hugely different.
And then we come down another level to accent.
So having a look at how to map metadata for accents, define variants and split those out,
and find a way that includes languages in a way that represents what people think of themselves as speaking.
What we've done on common voice is we tend to disambiguate languages based on ISO code.
So the Internet National Standards Organization often gets to choose who has and has into language.
Variants we have a different BCP47 views.
And for accents we've got a combination of an optional chance to set your accent.
But it's also a free text field.
So you get a drop down of American English, British English.
Or you can type whatever it is in your heart.
And people do.
Right now the way we collect data on common voice is a red speech corpus.
So people come up to the website, they push the button.
It's very technical.
And they'll get a sentence, which is always extremely normal.
Like his research largely concerns the eco-physiology of legends.
Like.
I was going to say this is the first linguistic heckle I've gotten.
But it's the first non-academic linguistic heckle I've gotten.
And I quite like it.
So our red speech corpus is fantastic for its simplicity.
Folks can come in.
They don't have to think too much about what's going on.
Hey, just read the sentence.
It's not great for a couple of different reasons.
First of all, the sentence is extremely weird.
And I think that speech recognition technology will very rarely be asked
to identify the word likens.
But also the way you read something and the way you say something is a very different vibe.
His research largely.
What are you doing?
That's weird.
Oh.
Also, a problem that I'm so, I'm so sorry if I look really excited about like
this technical problems, but they're fantastic.
Oh, this.
And like the language ones are even more complicated because they're people at the end of the day.
So a red speech corpus as well.
All you got to do is write it down, right?
That's not a problem.
People know how.
Like once you write it down, it's not even a big deal.
So for example, if I wanted to write something in Tajik, I would use the Tajik alphabet.
Yeah.
And that's not a problem at all until we get to the next step where this is also the Tajik alphabet.
And really if I wanted to, this is also the Tajik alphabet.
So looking at how to handle multiple orthographies.
So different communities, different contexts may use different characters,
which is one of my very favorite language problems.
When and how do you give the people give folks the opportunity to switch between those?
Sorry, I'm, yes.
But also who here comes from an English first background?
Like, I'm sorry, I need to.
So from folks who come from an angle phone background, we might get very used to using the same language
throughout our conversations with no code switching at all.
For folks here in Brussels or for somebody in Nairobi, the opportunity to pop different languages words in and out is called code switching.
And is immensely common.
So one thing we've just piloted for common voice to deal with the different orthographies,
to deal with code switching, to look at making it a little bit less like-any as well,
is we've got a pilot coming out where we've asked people to spontaneously tell us what they think about something instead of.
So instead of reading what's easy.
This is a bit disappointing to give somebody who lives in Britain.
Because it's a very, it's a very short.
But it's fine.
People instead get to record their responses.
Oh, do you know what?
I live with the equator, so I'm a reasonable person who is 12 hours of sun every day.
But it gives folks the opportunity to talk about something they care about.
It gives other folks the opportunity to transcribe it and learn about it.
And folks get to speak more naturally with umms and aas and aas and aas and pops and code switching.
And this could be a fantastic way to make a ton of money.
You say, oh, Jess, you told me how all these proprietary data sets are really expensive.
Like I did, I did, yeah.
And all I got to do is these very simple things and I'll make a ton of money.
I'm like, oh, cool, yeah.
It's very easy. You have a good day.
I wouldn't even worry about the marketing.
Wait, wait, see me after class.
But the big thing I would want to say is if you are using speech technology,
if you're building something, please, please, please come and voice us free.
It's thousands and thousands, it's tens of thousands of hours.
There's no license on it. There's literally no excuse not to use it.
But it's not just me sort of flogging our data set.
There are so many language demographic and domain data sets out there that are free,
that are open source, a lot of them are academic.
In 2025, there are very few excuses.
If you're building models and training models on languages, not to be adding these.
Very, very free, permissively licensed data sets.
And why I'm pushing this on you so hard is I like to stand up and be intense about languages,
because one of you is going to come tell me about Georgian verbs after this.
But if it's okay, it's not desperately uncool.
I'd love the opportunity to be excited and hopeful about tech with you all.
But really, I'd love to see some of you and some of you out there in the big wide internet,
building speech technologies.
There's a ton of open models. You can come get our data right now.
I can't stop you.
But also, even if you're not coming in telling me, giving feedback to people building the speech technologies
and building the data sets is critically important.
When I screw something up, when we screw something up, please yell at us.
Oftentimes when you see well funded technologies coming out of the west, you see a rest of world mindset.
Well, we're going to build this for the California market,
and then we're going to come into Europe, and then do you know what?
Then we're going to talk about rest of world.
And that's not the way the world really works, especially if you're building open source,
especially if you're building interesting, beautiful, useful things,
that you hope would change the world, thinking about where and how language comes in,
whether this is text, whether this is text localization, whether this is speech,
is something I would like to politely beg of you.
But even if you don't ever build anything with speech, please take into the world my permission
to get as loud and to get as mad and to get as weird as you want when you're talking to your computer
and it doesn't respect you.
Thank you so much.