So, I have a pleasure to interview Sedrick, and Sedrick will tell us what he said,
like, okay, who invited the intern on stage?
So, he worked on a draft for ordinary three years, as intern now, he's full-time work,
but he's presenting something that he did during these three years.
And, you know, thank you, yours.
So, yay.
So, my name is Sedrick Clyburn, and actually, first and foremost,
I want to give a big thank you to the organizers, and all you for being here,
we couldn't be possible without the community.
So, big round of applause for the organizers.
I'm a first-time attendee, and first-time speaker.
Here are a pause down, and really, really excited to share with you the session on building AI applications
from your local machine using an open source project called the podman, AI Lab.
Unfortunately, my co-hosted pod, which is the PM for the project,
could have joined us today, but I'm sure you're watching, hello.
So, let's go ahead and hop in.
So, today's schedule, we've got a lot to cover, and I'm glad the air kind of was before me
talking about containers and talking about AI, because it's a perfect match.
We're going to introduce podman as a container engine, and podman desktop and show you
the podman AI Lab, and how it solves a lot of the challenges that developers like myself
have faced when trying to build applications with AI models, and take those applications
and move them to a production environment.
So, we're going to have a couple fun demos.
We're going to look at different use cases, open source, and open weight models as well,
and do a final crem-de-like crem-de adding AI to an existing application.
If you want to check out the slides or try podman desktop feel free to.
But without further ado, I want to introduce some friends here.
These are the seals that make up the podman and the podman desktop projects.
And recently, he was accepted into the CNCF as a sandbox project just last week.
They approved it, so that's very, very exciting.
They had a long night yesterday.
This guy in the top right was a delirium cafe, and so on like four.
And, you know, they're big fans of fonds them as well.
If you haven't used podman before, podman is an open source container engine.
It's open, extensible, and it's Damon.
So, instead of a client server approach where you have a Damon running in the background,
what happens is you actually do a four-click exec.
That child process then becomes the container.
And because of that, you can run containers in a non-group fashion by default.
So, it's a great way to use container technology in larger organizations, for example.
And it's open source, and it works with OCI images, containers, volumes,
registries.
There's a little bit about the differences between, say, for example,
Docker.
If you've used that before, and if you want to learn more about it on the left-hand side,
there's a 10x engineer, really smart guy, tell my boss, who made a video on it.
So, you can learn more online if you haven't tried out podman.
But, podman desktop is this visual of graphical interface for working with containers,
for working with Kubernetes, and as we're going to see today for working with AI models and applications.
And it's paid for deploying to Kubernetes.
So, you can spin up a local kind, or a mini-cube cluster,
or you can connect to your cook and fig, and deploy something straight to your cluster
without having to leave your container development environment.
So, it's very easy, and it's all open and extensible by default.
And it's built on a really nice stack.
So, you've got spell and tailwind on top of the electrons,
so you could use it, Mac window is Linux.
And now, recently, for Mac, we were using lib-k run to enable GPU passed through,
so you can use Lama Cpp with, like, bokeh, and then make those API calls to use the GPU for model inferencing.
So, that's very excited.
Now, the main focus of the talk is transitioning to this new era that we're all living in,
which, as you can see from our lovely little mascot, is the world of AI,
which, for me, in the past few years, was kind of daunting,
as application developer myself, because I would be talking to the data scientists on my team,
trying to see what they were working on, and they would send me a Jupyter notebook,
and I didn't even know how to open it.
But now, things are a little bit different, and we've got all of these open source tools and technologies
that we've talked about today to spin up local AI model, as an API,
and make those requests without even having to need Wi-Fi.
So, I think it's amazing, and it all starts with enabling developers to use models
from their local machine.
There's a lot of benefits because, for example, we could run an AI model as a service,
just like we would with a database, like a test container, for example,
and we have control over that model, right?
So, no data is leaving our local machine, but also is completely for free,
and this is amazing.
The PodMan AI Lab is what we're taking a look at today.
It's an open source project and it's extension of PodMan Desktop,
so that you can have an environment to develop applications that use AI,
and run it all in containers so that when you want to move to a development or production environment
on Kubernetes, everything is running in the same predictability that you would have throughout the process.
So, I think it's cool.
And as a developer, it can kind of feel overwhelming to start working with AI.
You might be looking at different AI models.
You might have picked one and then bound there's a new deep seek that's out and release the next day,
right?
So, we want to have a little bit of predictability and standardization, right,
with working with models, and that's the whole ideation and prototyping videos.
But then you have to add things on, connect with services and data sources,
with drag or agent to connect to APIs and change things together.
So, it can seem overwhelming, but we want to kind of make it easier for developers like me,
and you guys to try out prompts and to build applications and run models.
And so, they're going to show with all the demos today is the AI lab.
So, it includes recipes, so, for example, with drag, you would have a inference server,
and you would have a model and you would have a vector database,
all included in containers in one singular pod,
so that you can quickly spend things up without having to worry about dependencies,
and do your iteration of build, run, and test all in containers from your local machine,
using the same workflow that you've been using for container development.
So, if you want to learn more about it, feel free to check out the slides,
but I love demos, and I hope you guys do too.
So, who's ready for some live demos?
To be, all right, so here's the extension.
I'm running pod and desktop here on my Mac, and so the container engine is running itself.
Under the hood, what's happening is we've set up a virtual machine.
This was configured all for me, just with the click of a button, so it's very nice.
And we have GPU passed through, so that we can inference with this new M3 chip,
which is very nice.
So, let's go ahead and check it out.
This is the extension where we've installed it,
and there's tons of other similar types of extensions.
We'll also kind of look at bootable containers today with AI as well,
which is pretty cool, but the recipe catalog.
So, let's say you want to build a new POC,
or you want to test something out.
As a developer, you've got to think about the models,
and the frameworks, the language change,
and you've got to think about an inference server.
It can be a lot to think about two complex, right?
But with these recipes, we just have the ingredients
that you need to set up this application.
So, you can learn about it here in this little catalog,
but I'll go ahead and start one up and show you how it works.
So, there's a list of different open source,
Apache 2.0 models that you can download straight from Huggenface,
the click of a button,
and you use that model in this recipe.
So, we're essentially pulling down the chatbot,
GitHub repo.
We're mounting that model to the inference server,
and we're starting in it.
We're starting in the ChromaDB vector database,
in order to chunk in our data,
and we've got this app running.
So, I'll go ahead and show you this.
What is it?
And second to go, it's all running in a singular pod,
and let's check it out.
So, it's a simple streamlit application,
let's ask a question.
Who is keynoteing,
phased in this year in 2025?
So, this is a standard example of,
right, asking the model a question that it wasn't previously trained on.
The answer we get, hey, we don't have information about who's keynoteing.
I'm sad now, right, I want it to know.
And so, I've got a little brochure in here,
actually, I'll show you,
that just has a little bit of information about the event.
And let's provide that to the application,
and let's go in the background here,
and see what's actually happening.
So, here we have all the logs.
Oh, that was supposed to happen.
Okay, so let's say our applications are working,
and we want to come back here and restarted.
Well, it's very easy.
We just restart the recipe.
Let's say we've made some changes to our code base.
We've added a new feature.
We've started iterating.
Well, this will be a great way to just, you know, code,
and restart code and restart.
And then, hopefully,
not have the same issue.
So, we'll give it a second.
The containers have stopped running,
and hopefully, it'll just start out for me again.
And we can try to ask the question again.
So, just like that,
all the containers have restarted.
So, for vector-day-to-base,
the stream-lit application,
hey, who is keynoteing,
Faust in this year?
So, we'll try it.
Faust in.
Okay, so, we'll get the same answer.
Doesn't have the context.
Fingers crossed.
I'll give the same answer.
I'll give the same question.
Mmm.
Good thing I haven't back up.
Okay.
So, now we ask the question,
provide the context.
And give it a second.
Now, while it's processing the document,
I want to show you the magic for
what's going on behind the scenes.
So, here is the source code
that we've cloned when we started this recipe, right?
And we've got a container file to package our application.
We make this a little bit bigger.
We've got our rag application in order to,
say, for example, connect to the model that's running locally.
And we've got all the code here to accept, say,
PDF, text, some front end here,
and be able to pass a prompt template.
Hey, don't answer the question.
If you don't have the context,
I don't know which one.
One of these was supposed to work.
Just trust me.
We'll have to another demo.
So, code base container file,
our application package, just like that.
We've also stopped here in these recipes
in order to, say, for example,
take our application and run it as a service,
say, using quadlet, which is a feature of podlet,
or to take it and actually run it as an operating system itself.
With, say, for example, we take
sent-to-stream as a bootable container base image,
then we overlay, as containers,
all of the different components,
the model, the inference server,
the database, and the application itself.
And we can actually take this and run it
as an operating system wherever we'd like,
you know, the edge, IoT devices,
etc., etc., where the model needs to be as close
as possible to the actual hardware itself.
So, that's pretty dope, as well.
Let me show you a demo that hopefully works here.
A lot of our applications for AI might be working with text,
but it's really important to remember that audio,
vision, etc., are becoming more and more popular.
So, here's an example of audio to text.
We're using the whisper port for that came from OpenAI
to power this application.
So, what we're going to be able to do is, say, for example,
if someone is out in the field,
maybe it's a doctor, and they need to transcribe their conversations,
we could build some type of application to be able to
understand the language in real time.
So, let's go over here,
and I'm going to talk a little bit about
my first time here.
It's very crazy, and I have a hard time navigating.
But, maybe OpenAI, I'm too close to it.
The results were almost Ablar in Espanol.
You know, a couple of different languages to test this out.
And let's go ahead and upload this file to our little applications.
So, let's go over here.
And fingers crossed.
We'll give the MP3 file, and let's see what the result is.
Yeah.
Pretty cool, right?
So, you can take whatever language whisper has about
80 languages that was trained on,
and start to build an application just like this.
Click on the button.
You have the model, the server for inferencing,
and the front end to open, and then you can make it,
it's not very wrong.
Because what we're using at OpenAI,
that you saw earlier, was Lomacy plus plus.
But if you like BLL and where you could replace,
and kind of build your own recipes,
and use kind of this model catalog here,
of these open, open whisper license,
poshi 2.0 models,
or maybe import your own.
So, for example, deep-sea,
I've been playing around with it recently.
It's very interesting.
You know, we could test that out here.
Then, or...
Let's see.
I'm not liable for the output of this model.
So, I love how it thinks.
It's like, it's a wrapper, like I said.
So, fun to play around with models.
But, of course, the real use case is,
and let me see if I can open this other,
or give it a refresh.
The real use case is doing some experimentation
and prompting.
I can't open up my other playground,
but it was just some kind of messy run
of the temperature and doing some prompt templates.
Five minutes here.
Let's do the creme de la creme demo.
I want to show you the model survey inside of things.
So, here we go.
We've got a couple different models that we've kind of spun up.
Let's take, for example, this granite 3.1 model.
We've been spun up.
This is just running inside of Mama C++ container.
So, we've containerized this infant server.
We could check out the, for example,
Swagger API and kind of, you know, try it out.
What's the capital of France?
Hope it says Paris.
That was close.
We could say, take some code and integrate this into our application.
I play around with Langshane and Langshane for J on forks.
It's a pretty good way to start using AI if you're a job developer.
And this model is running on this specific port.
I could copy this code in and start using it into my application.
But let's see, let me hop over to, here we go.
Cool.
So, I have to do a quick step and actually I need to change one more thing.
So, this is Langshane for J.
Let's see, I'm going to replace the port here.
So, the model is serving, right?
We just tested out the endpoint a second ago.
Let's start off our application.
For eight thousand six, cool.
So, let's say someone in this organization, for example,
has to go through a large amount of backlogs of claims, right?
Everyone's getting in car accidents for some reason.
So, here's a new claim.
Marty McFly, crashed as DeLorean.
We want to be able to have a model to be able to ask,
hey, who is it fault here?
So, let's try it.
You saw the poster quest go off in the background.
Okay, enough audio.
And here we have, in token, the response back.
But it's really cool, right?
The models running locally, we're making that poster quest,
getting the response back.
And hopefully this is a great way to start experimenting and
getting from our local machine and building some cool applications that use AI.
Let's see who is at fault.
Okay.
So, you know, very nice, very nice.
So, I hope you guys have enjoyed this little kind of live demo for the AI lab.
There's a lot of cool stuff you can do.
And if you want to replicate a lot of us for it,
and be able to call models and actions with the API,
or you can replicate it specifically here for this local server.
Let me kind of close up here and let's see.
Cool.
And as you might know, containers are a perfect solution for AI in the moment.
Because of the portability, the ability to scale up on demand,
let's say when our application is in use nine to five,
or be able to scale down afterwards and be able to save.
Not only money, but maybe the tree is outside as well,
and be economical.
So, this GPU acceleration, whatever platform you're on,
and a highly recommend that you try out the pop-end AI lab.
There's a great community behind it.
It's an open source project.
It just joined the CNCF.
And it will free to provide feedback on the repository
to contribute, create PRs, whatever you would like to do.
And be happy.
On behalf of the whole team, I want to say thank you.
And this is a pleasure.
This is really fun time.
And thanks for coming to Faustin.
Thank you.