Okay, my name is Joelson. I'm going to talk about a new open source data engineering framework called data prep pit.
Okay, so this data prep pit was released by IBM that Apache 2.0 licensed last week.
It was used internally by IBM to prepare their granite LOM family.
So it was a data engineering tool that they used. They released it open source last week.
There's three value props with this framework.
Data engineering built on cube flow pipelines. So it's workflow based, which means you can define steps or flows made up of steps.
You take the output of one step and you send it into another depending on what the output is.
So it's a little easier to use than a little more flexible than just raw Python.
The second value prop is scalable compute. So you can build and test your workflows locally and then easily migrate them up to much larger cloud clusters.
And then the third value prop is community.
We can collaborate since we're workflow based. We can collaborate and solve and complex data engineering problems facing GNI such as determining licensing and copyrights.
Compliance GDPR identifying personal information hates speech and bias.
So we're just shoveling all our data into our LOM.
We can figure out and design pipelines and try to detect these things before they go into the LOM.
The potential user base for this.
GNI value creators. So if you don't want to get bogged down in data engineering, you're looking simply to maybe set up a rag with a bunch of documents in it.
This would be a perfect tool to do this. There's many examples and tutorials for processing that kind of data.
Maybe you're a professional data engineer and you are bogged down in data engineering.
I develop workflows, Python locally, easily migrated up to spark ray, Qflow pipelines.
There's a catalog existing transforms that operate on big data. So you're already can come out of the gates with a bunch of stuff you can already leverage without having to rewrite it yourself.
And then the third potential user of this is the AI researchers.
And if you want to collaborate with other researchers and solving some of these data problems, this is a good framework to do it.
And I'll just quick shout out to the AI Alliance. The AI Alliance was started in December of 2023.
It is about 100 members right now. It's a nonprofit organization that is promoting open-interested MLDI data engineering.
So thank you.
Thank you.