So, hello again everybody, my name is Cedric and I'm joined today by my good friend Carol Chen and today we would like to present and actually do some demos on the concept of synthetic data and synthetic data generation and this has been quite a hot topic recently and you might have heard about it in the news, right?
If you are on YouTube as much as I am, you might have seen interviews with Sam Altman or the Zuck as I like to call them talking about synthetic data, talking about model distillation or, for example, you might have seen in new model cards that a lot of the frontier and state-of-the-art models are now being trained with synthetic data, right?
You start thinking about that. The AI is making content for AI, right? Are we going to have model collapse or in another light? Are we going to soon run out of training data, right?
If you take a look at this chart, there's essentially a prediction of about 2027, 2028, where we might run out of available tokens to use for model training.
It brings us to this general idea of synthetic data and for our talk today how synthetic data is powering the next evolution of language models.
Gardner here reports that by 2030 synthetic data might overshadow real data in language models.
All things to think about, but we'll walk through this today, we'll talk about challenges of AI and production,
how synthetic data can help, and where it fits in for the LLM pipeline of pre-training, post-training, and model evaluation, and do some cool demos, and show how, say, for example, if you're using a model that's maybe been tuned with synthetic data for a particular use case, you could have 1,000%
times less cost, for example, than using a standardized chat GPT. So saving money, saving the environment, this is all great things that synthetic data can do,
feel free to check out the slides, and we'll do a quick introduction. Carol, would you like to go first?
Thanks, Edrick. So I'm Carol Chen. I'm also from Red Hat. I do a lot of support for upstream communities, and some of them are listed here, including a strong lab, which utilizes synthetic data.
So if you want to find out more information, it's trucklight.au, but you can also find me a sidebed on many different networks.
Thank you. Yes, Carol's awesome, and my name is again, is Sedrick. I'm an engineer and advocate, organized KCD in New York City.
And I make YouTube videos if you want to learn about AI topics on those whiteboard things, feel free to check that out. So, Carol, pass it back to you.
All right, thanks, Sedrick. So before we dive into synthetic data, let's quickly look at some of the challenges that of AI adoption. First of all, it's cost.
So the cost of influencing a scale to training and everything. However, this is changing.
I'm sure most, if not all of us here have heard about deep seek. Well, there are some questions about, you know, some of their claims.
The reality is that innovation, especially open source innovation, can impact costs and efficiency both at the model level and beyond.
In addition to saving energy and other resources, performance is also increased in these new open models, and deep seek is definitely not the only one.
For example, there's the grand models if you're afraid of them. And this is a good trend, and open source is fueling it.
We're looking at several orders on magnitude here from billions to millions.
Although I've just came across some article debunking the, you know, $6 million claim by deep seek.
But anyway, according to a recent gardener report, more than 30% of January projects will be abandoned due to cost being one of these major factors.
Next, we look at complexity. Let's say we want to customize our lens with our own data.
The curation of data sets is usually non-frivial and very complex for non-data scientists like myself.
So whether it's business-centric data, or, you know, the main specific data we need to incorporate this additional knowledge,
because some of the limitations are due to those models being to generic.
And not only that, as more and more companies and organizations need to align models with private and sensitive data, such as in healthcare.
This complex training and tuning processes, you know, you have to do with that.
And not to mention lack of quality data that, you know, due to privacy reasons or, you know, very few specialized,
don't mean every experts that you can get a data from. So, like you said, the scarcity of data.
Finally, there's the, you know, different types of, in many cases, we require the flexibility of deploying on hybrid clouds and using different types of models, such as, you know, smaller scale models on edge supplementary larger, larger language models.
So the control and of these can also help with dealing with tricky regulations, such as data privacy laws, handling data anonymizations and so on.
So these allows the flexibility allows organization to take advantage of innovation while keeping private data secure.
If you have to cover some of these common AI challenges, what are some of the, you know, the role of synthetic data can help, you know, in addressing some of these, and I'll hand it over to subject to share with you about that.
Thanks, Carol. Lots of, lots of good points on why synthetic data generation is important. And it kind of boils down to this, this idea that data curation is difficult, right, hiring data scientists in order to do the collection and
other types of engineering you need to process and filter and annotate data and and essentially refine it into an end result that you could tune a model with is expensive.
It's prohibitive and for organizations that want a domain specific model, it's just, it's just difficult. So synthetic data has been kind of a solution for this issue, you know, you could create domain specific data for health care for enterprises for.
Industries where you might not have it, there might just be data scarcity and for regulations like HIPAA or the EU regulations where you just can't work with medical data, for example, in certain types of ways.
It allows you to substitute and have a complement for that real data that you can then test and train with.
And for cost and efficiency, elements have been proven to be a great annotator. And that's why you already see synthetic data being used in the state of these art models.
Because they're annotated perfectly, they're high quality and elements when they actually go through this process as we'll see in some examples in the future, tend to explain things better than we do as humans.
When we explain things, we think, you know, of different, how do I say, we kind of relate ideas in a different way than a model would for example.
And so by using models as annotators and to explain and refine ideas, you can get better results as we've seen with the chatGPTs that have been trained on synthetic data.
And for example, deep-seek itself was trained with synthetic data generation. And this brings us to the idea of pre-training for foundation models.
So there's tons of different examples of refining, classifying and working with the available tokens out there on the internet in order to create and curate better data.
So some of these might be, for example, a fine web that you might have heard of, which is a collection of kind of the crem de la creme of the web that is out there, right, for education and just for general purpose, hugging face to the cool experiment called CosmoPedia, where they generated a pretty large data set in order to train their small models.
And they kind of took webpages, reprompted them, and used that to fine tune a model. And then even in video, rewrote almost two trillion tokens to remove low quality data, right.
And what we're seeing is this trend for synthetic data generation of continually having more efficient and refined data. So it's more of a quality over quantity type of thing.
We're instead of taking the whole internet and throwing it at a model for months and months while we're taking a specific portion in a fine portion.
In addition to pre-training, there's post-training in the fine-tuning process that we're going to look at today.
For example, Red Hat and IBM, we've got this project called Instruct Lab, which uses this student teacher approach in order to teach a model the way that you and I would learn something.
There's also Microsoft's Agent Instruct, which is a similar process of taking in all of your raw source materials, transforming that, and then having that as seed instructions in the taxonomy, which is a great way to organize data for training to make sure that there's no biases or lacks in the models capabilities.
And there's also the stage of model evaluation where you can see synthetic data being used. So for example, in various benchmarks that are out there, but also for replicating rag data, so making sure that you don't have limitations or blind spots.
So for example, in general, sorry, LLMs have been proven to be pretty good for annotating that data and we're going to see that here in two different examples to essentially go through and be able to label existing data and create new data from seed examples.
So who's ready for a live demo?
Awesome, we've got four minutes here, so I'll try to speed run this.
So the first example is, oh yeah, I thought our talk was 15, sorry about that. Cool, so that's exciting.
Our first demo is going to be a bird style model.
Yes, it's perfect, because I'll explain the demo real quick. So bird was like the OG chat GPT, right?
And it's, it's a little bit different of an architecture because instead of going left to right, decoder style in encoder, so it takes the entirety of the text that you're processing with it.
So we're going to use it for an example to take a large corpus of text, let's say it's like an investment banking company and they have a portfolio.
They want to make sure that hey, analysts, are they liking things or they're not liking the market.
So what this is going to do at the end of the day is provide almost over a thousand times cheaper model costs than say taking a general large purpose LLM.
It's going to reduce the amount of carbon dioxide that's being produced by the model and it's going to reduce the latency for responses and be able to perform on par with one of these day of the art models simply by doing synthetic data generation.
So we'll go ahead and hop out of here and I want to show you.
My apologies, coolab.
Well, let me, sorry, I have to hop in it.
I thought I had the notebook ready.
Oh, you're joking, okay.
All right, so here we go.
I'm going to head over to the notebook.
So essentially what's happening here and I hope it's big enough for everybody.
Let me make sure is that we're importing some requirements logging into hugging face and we're going to use the open source, a
extra model to be our annotator for this specific example.
From hugging face, we're importing a financial phrase, big data set.
So this is examples of analysts who are actually in this domain who are saying, hey, this news is reputable or this not reputable, but this is in the positive light or a negative light.
And so essentially we're dividing this into the different labels of the data set, so the sentence and the label that we want to provide.
And we're providing kind of a chain of thought template here.
So we want the model to explain it's thinking.
We're providing examples a few shot prompting with, hey, if the operating profit increased, this is a positive example.
But if the decrease profit was the result, then it's negative.
Oh, shoot, okay.
All right, no mind, one minute.
Okay, anyways, all we end up doing is going through the data set.
So it's already been labeled, but we want to see, hey, what would the annotator do?
And be able to, at the end of my notebook here, be able to label the instructions here, positive, negative.
And once we actually take this data set that would be extracted from a larger corpus of text, then we would do fine tuning on that model.
Say, for an hour or two, with some maybe compute that you could rent from hugging face or AWS.
And have a model that could actually do that.
But the other example I wanted to provide really quickly is in struck lab, which is another way to do this for a conversational element.
So the first thing was classification with text.
This is for an actual conversational element where you can represent the questions and answers you want the model to provide.
And then with a locally running model as we have here, be able to generate a data set.
For example, here, like a JSONL to have the questions and answers that stem from the seed data.
So I know we're out of time, so sorry, but that is our talk today.
That's also another way to save costs and create a domain specific model either from classification or conversational elements.
And on behalf of me and Carol, thank you.
Sorry.
Thank you.