We'll speak with and prompt high fidelity text to speech with minimal supervision.
And when we read the paper, we kind of found out that the eye texture that they presented
is simple.
It's basically just like true transform models stitched together and to end.
The samples that they released also sounded really, really good.
Unfortunately, they only released the paper and the samples.
And we thought, like, we give it a shot to basically replicate the paper.
We had no previous experience with TTS, speech, or even with training big transform models.
And SS had this is basically our journey.
Let's see if that works.
Hello, Fauston. This is the first demo of Wester speech.
A fully open source text to speech model trained by Calabra and Lyon on the jewel's supercomputer.
This is basically just like one example that Wester speech is able to to produce.
And we think it's basically on par with the text to speech that you get from Amazon Microsoft or Google.
Previously, I basically said that we build Wester speech on top of the S.
That's only half of the story.
We actually build Wester speech on top of some amazing open source projects.
Mainly Wester from OpenAI.
That's also like where the name Wester speech comes from.
And codec from meta, which is the the neural codec and vocals from Gemilo AI.
We also in the process of like implementing Wester speech.
We read a bunch of papers from like different AI labs.
Namely, of course, Spirit TTS from Google.
Music can from meta and tensor program five from Microsoft and OpenAI.
So this basically shows you that the rough overview about the architecture of Wester speech.
So on the left, of course, it detects a speech model.
You have the input text, which is then in fact into like the first transform model, which kind of creates a phonetic representation of you in
Protect.
And since it's kind of difficult to figure out like emotions or property from the text itself,
we also embed this into like the phonetic representation as part of the first transform model.
Then the phonetic representation is fetch into the next one, which then actually creates the actual speech.
It's still like compressed speech, but it's speech.
And we also, since the text doesn't give you anything about like the speakers, emotions, whatever.
We also add speaker embeddings on top of it.
And then we apply the vocals vocoder to actually get the audio out of it.
Now, I mentioned that we kind of get like a phonetic representation.
And phonetic representation actually comes from from Wester itself.
And Wester is a very easy, just encoder decoder model.
And if you just look at the rights, the Wester decoder just takes the Wester features and creates the text tokens,
with like varying speed because you don't really have an idea from the text itself,
but from the audio sample, how fast the speakers.
And on the left, of course, you have the input speech, which is like an audio file,
goes into the encoder, and this actually creates the phonetic representation.
And what we did is we trained a quantizer in between the encoder and decoder to not only kind of reduce the number of features,
but also to kind of force the encoder to focus on the phonetic representation.
Now, we basically have our architecture in place, and this was basically our plan, how we can basically train Wester speech.
Starting from the right here, we have the speech waveform, which is just audio files,
and you can find a bunch of audio files like on the internet.
The problem is, if you want to train a model, like a text speech model, you also need the transcription.
Luckily, we have like the open-eye Wester model that can take audio and create the transcription files.
And then we just do the reverse, we have the transcriptions, and in the training process,
we just used the audio file in the transcription to train it.
But when we actually started the training process, we kind of figured out that it failed us on all fronts,
and yeah, like let's just go through the different parts.
Starting with the speech waveform.
I mentioned that we just used audio data.
Luckily for us, there's like this really great status at out there.
It's called liberalites.
It was released by meta.
It has 60,000 hours of English speech in the public domain.
But then, like the first problem that we encountered was, it comes as a single 3.6 terabyte archive,
which is kind of dumb.
Just if you want to download this, and if you can, like, set a rate, you're 200 end, it's like connection.
It takes you like 5 hours.
But if you're like us, you want to try out different things.
You don't really download it like once you don't like multiple times.
So it's a problem.
If you want to unpack the status at, it takes like over eight terabytes on your SSD.
Just yeah, so you have the data available.
The other problem is read amplification.
So the 3.6 terabytes are actually 220,000 files.
And the liberalite data set, as I mentioned, it's fairly audio books.
So they're kind of like split up into two chapters.
And one chapter has around like 16 terabytes on average.
And in machine learning, what you want to do is you kind of want to randomly access your data,
you want to shuffle it.
You might want to just extract like 30 seconds out of like a single chapter.
So on average, you kind of have to read like eight megabytes, and the rest is just waste.
And if you do the mass, if you want to try in this, you kind of have like an SSD that can read like six gigabytes per second.
You have like HEPUs and like use a batch of like 32.
You arrive at like three iterations.
And in practice, what we have seen is more point five iterations.
Because like you get like file system overhead and just like the system overhead in general.
Which is really, really bad because like you would wait like months to just finish like a single iteration.
Luckily, if there's this really cool project, web datasets from TMB, that's the GitHub user handle.
And what the web datasets actually allows us to split up our 3.6 terabyte file into like charts.
It's just like like splitting up in multiple tar files.
And we split up the whole dataset into 623 5 gigabird chart files.
And the web datasets had then allowed us to read eight random charts to country and also shuffle.
And if you do the mass again, like you basically arrive at 330 iterations per second on the same system.
And that is something that you don't really see because like most people just maybe deal with like a gigabyte of data.
And that's it.
And it just fits in RAM and we don't really have a problem like you don't have copies whatsoever.
But if you deal with like a lot of data, large data, the disk actually becomes a bottleneck.
So now we have a way to read our audio files, our audio books.
Next thing is that we need the actual transcriptions, right?
We need it to try and the whole whisper model with the speech model.
And luckily, as I said, we can just use OpenAI whisper.
It's like a state's great of the art model.
But if we actually use it just like the plain whisper model that OpenAI released,
it's like 50 times faster than real time.
Which if you want to process like the 60,000 hours, it takes you 1,200 GPU hours, 50 days.
I'm not waiting like over a month to kind of get the transcriptions.
Especially if you mess something up along the way, you probably do this like multiple times.
So make it faster.
Yeah, you can use batches.
You can also use faster whisper implementation, which is using the same model,
but it's just like a faster implementation.
You can also switch to a smaller model, which then basically puts it down to like 78 GPU hours,
which is still like three days.
But remember, we are using web data sets, which allows us to parallelize this over across like multiple GPUs.
So if you do this across like 100, for example, you can kind of process the data set in over and in another one hour.
And that's not only works for the transcription, but also for voice activity detection, speaker embeddings, and so on.
Now we have the transcription.
But yeah, whisper is like kind of a state of the art model, right?
But we encountered several problems.
One of it is like you also get timestamps from whisper.
But what we have seen is that they're off by several seconds.
Luckily, it's kind of consistent, so we just applied a constant offset.
What was even more puzzling was that in the transcription itself, there were some parts missing.
And SSH had been using the liberal light data sets.
And this is like an example, like if I hear, like if I'm listening to the one of the shepherds,
it says like, shape the five of the things in our garden by author random,
this leap of works recording is in the public domain, and so on.
And what whisper heard was basically this.
So it ignored the first part completely.
And we kind of like wish this art because like you want to train something like on on top of it.
And we have a really good idea like how this happened.
So basically, open AI, it's very likely that they use the liberal light data set.
It's about like 10% of their whole data set.
So it's not likely that they ignore this.
And but they also needed the transcriptions, right?
But what they did was basically they used forced alignment.
And the actual e-books from project Gutenberg.
So they aligned the text with the audio.
But in the e-book, there's no like chapter five.
You don't really see this.
Even like the same four footnotes as well.
So what we do to basically fix this if you just ignore the 30 seconds.
Now like to this training itself.
Just to sit for me, what's the most important thing when you need to solve a difficult problem.
And it's basically iteration speed.
I'm just skipping this because like I'm running out of time.
But yeah, what does it mean in our case?
So if we want to train like a large model on like 80,000 hours on a thousand speakers on like 96 GPUs,
we take like six hours to train.
Which allows me to basically train two experiments to two experiments per day.
One in the morning, one like after lunch.
Which is not really great because like if you like us, you want to try out like a lot of different ideas and like 99% of the ideas I garbage.
So what you do is just you train it on a very small data set.
Single speaker, you can train it on the 40 90 and 50 minutes.
And that allows you to train basically 48 experiments per day.
And then like you just like train a little model like add somewhere depth and profit right.
Now the last part.
I mentioned like oh like you need GPUs.
96 GPUs 100 GPUs.
So doing our journey, we basically export like different companies.
We started this like data crunch on the lips.
Which kind of give you like visual pricing.
But the problem is that usually you can only get like one or two GPUs.
So they're all booked.
Another interesting platform is vast AI.
Where basically users provide their their GPUs that they have under the desk.
It's a lot cheaper.
But the problem is like bandwidth.
And they also don't support like network drives.
And again like we have to download the data type like multiple times.
Like you have some machine crushes and all of this.
And something that we figured out later was if you do like open source work.
You can talk to the line community.
And they have access to like the dual super cluster.
With a lot of GPUs.
And then yeah, you basically profit.
There you go.
Give them a round of applause everybody.
Good morning.