Yeah, I'm going to skip this slide.
So hello, everyone, at the next talk, we have Eric from Green Island, coming to talk to us about Ramalama.
Eric, it's all yours.
Thank you very much.
Let me just clip this in, so hi, guys.
I normally start this song with a phone a little jingle, but we've only 20 minutes, so I'm going to skip it this time.
I'm a little sorry, guys.
Okay, so yeah, this is a project I created with another engineer called Dan Oach.
I'm a recruitment, I'm a self-engineer.
I guess from the AI perspective, what I've worked on is co-created Ramalama, and I'm also a LAMSIPP upstream maintainer.
So yeah, Ramalama is all about making working with AI models boring and simple for everyday users like you and me.
Yeah, so as I said, this was, I'm going to fly through this stuff really fast.
I want to give you a short demo and give you guys ask some questions or whatever.
But this is our own goal, let's start.
Now we're exploding on all sorts of different features, but the goal was to make AI usage boring.
Boring is kind of a term I stole from another famous engineer called Jam Masters.
Well, yeah, to allow the simple use cases to basically just work.
So like Ramalama, let's say, over the weekend, this cool Chinese model gets released called Deepseek.
You can just run this one, and it's dark man, and this one line run command with Deepseek, and essentially it'll just work.
So some of the goals, some of the goals are to be OCI centric by default.
So to provide user space runtime environments as containers.
So there's a huge array of acceleration stacks for AI.
And they're not actually the most straightforward to install.
Some of them have lots of dependencies, some of them have not so many dependencies.
Well, one of the things Ramalama will do is when you try and run it, it'll look for the primary accelerator on your machine.
And based on that, it'll pull the correct runtime environment for that accelerator.
So it'll just work.
And sometimes maybe you want to choose your AI stack, maybe you're like, I want to play with Vulcan.
You can manually specify what AI stack you want to use also.
So another feature is to provide models as OCI artifacts, because I'm going to go through that later in slides,
but basically that you can push and pull models as from OCI container registries,
because OCI container registries are all over the place in public and in private networks.
So it's just there well suited handle multi gigabytes files, which models tend to be in multi gigabytes files.
So it's well suited for that use case.
Some containers are not strictly required.
You can actually run Ramalama with no containers.
Sometimes I do that on my MacBook here, because if I want to try to run Ramalama natively on macOS,
which doesn't have containers built into the macOS kernel, this makes life more easier.
This is something I read and lamissy people before.
So AI models aren't deterministic by design.
So it's probably a good idea to isolate AI models anyway in either a container or a VM or something.
And this was some advice I read on lamissy people.
Documentation always execute untrusted models within a secure isolated environment, such as a sandbox.
And you might say, well I trust my AI model, and another line I saw a couple of lines later,
is that the trustworthyness of a model is not exactly binary anyway.
So you should probably always encapsulate your AI workloads to a degree.
So this was another goal to be deemless by default.
So a lot of AI tools use client server approaches, which is fine.
But by default, if you use Ramalama run, it just spins up one process, which is quite useful.
And when you tear it down, it's using zero resources on your system.
But of course, we can run Ramalama as a server to using commands like Ramalama server.
So goal is accelerator support.
There's a lot of AI projects out there, because the likes of a video dominate like 95% of the market.
There's a lot of AI tools that say, we're only supporting Nvidia, or we're only supporting Kuda and Rakan from Andy.
In Ramalama, we're actually quite different.
If you're willing to do the enablement work and open a PR,
we'll review it and get emerged run on your accelerator.
There was the guy recently, he opened a PR to get Ramalama working on his Intel based AI accelerator.
And yeah, he's having great joy with that.
So yeah, and if you put in the work to auto detect your primary GPU, that's kind of useful as well.
Because Ramalama is all about kind of usability.
OS support, so we support Linux, whatever Linux this year, you run.
Mac OS port natively, and also Mac OS via virtual machine called Padma Machine,
which allows, which is basically Linux VM, which allows you to do all the cool containers of Mac OS.
I also know guys running it on Docker desktop.
Mac OS there haven't good successes as well.
Our Windows port is via WSL too, and there's some documentation on that if you want to run it on Windows also.
So this is another goal of Ramalama.
So we want to be open multiple model registries as I call it.
So you can push and pull whatever location you choose really.
An interesting comparison here is the comparison between OS AI container registries and the alarm report call.
Because I examine the alarm report call and the alarm report call is very interesting.
The alarm is a company formed by some extra guys.
And if you look at the protocol, they slightly fork the OS AI transport layer,
but just enough so that it's only compatible with the alarm registry.
And that's fine, but it just means if you're going to build up all your infrastructure based on Olamah,
now like either you're going to surpass the alarm registry usage limits which I will introduce at some point and I'm speculating.
And you're going to have to pay them lots of lots of money or else if you're deploying in the data center where you're not allowed to pull from external networks.
You're probably going to have to spin up Olamah servers in your house and buy all those servers and change all your firewall rules.
Yeah, the idea and all else you can just use your existing OS AI container registry and you don't have to pay for new servers or worry about surpassing usage limits.
So what I'm trying to say is you can either use plain old OS AI container registries or you can pay Olamah ton of money and spin up new infrastructure in the house.
So the goal, yes, one transport layer for all, so a lot of technologies are using OS AI container registries to push and pull all sorts of artifacts.
For example, there's a product, there's a distribution from the company I work for.
Red Hat calls really I and it's a booty based operating system and the main advantage is that is you can pull your system updates from your in-house OS AI container registries or under public internet.
So you can use that same transport end point to pull your operating system.
You can pull your AI user space run times from this same transport endpoint.
You can pull, you know, your application containers and now you can push and pull models as well.
So that's kind of the goal and it'll it'll simplify your infrastructure quite a bit because, yeah, you can serve all sorts of files on your.
Both the registries which are well suited for multi gigabytes artifacts and.
This is one of our newer commands.
So let's say you found a cool.
Modline line on either hogging face or alarm or whatever.
You could run a command called Rammalama convert, convert to the OS AI format then you can push and pull to your local container registry rather than pulling from the internet the whole time.
And this is.
This is a new feature from Padmai.
This is quite cool.
It's called Padmai.
And what Padmai.
Can do it can push and pull files as OCI.
So we're going to integrate that basically to.
So you can push and pull model files.
And another feature we support is switchable inferencing run times.
So these are kind of two that are quite popular at the moment.
So you can look quite well on Apple Silicon for example.
So you can look caught away and test your models.
You can learn the CPP for example.
I know runs quite well and it spreads great.
And video accelerators.
So yeah, there's there's a flag that's actually on time.
So you can basically choose your.
Inferencing engine.
And this is how the most basic usage of it works.
So you run Rammalama run.
That's an IBM model.
It tries to detect your primary accelerator,
pulls that inferencing run time,
which in the diagram is CUDA.
And then it pulls the model from.
Some place.
It could be hooking place.
It could be alarm.
It could be.
Queda.io.
Docker hope whatever.
And then you start the container with the model.
And yeah, everything just works.
Here's some water commands.
The first one is basically how you install.
That's how you run.
That's how you serve models.
I'm showing switchable run times there.
With VL 11, LAMS, CPP.
You can generate quadlets.
Or cubeless files.
If you want to deploy this on Kubernetes.
And that's where I run command is.
I'll skip past that.
Bramalama pull.
That's how you pull models.
This example is a hooking place model.
And Rammalama list is basically how you list all your various models.
So yeah, our community is growing.
And I encourage you all to get involved.
Like going back to the accelerator support.
That's one of the areas where I find the community is super helpful.
Because I don't have access to like thousands,
all the thousand different types of GPUs in the world.
And the community are great helping for that.
Much of the solution is written in high level Python.
So it's it's very accessible.
And we're an open ecosystem.
You know, just just open issues, PRs.
And when we get time, we'll take a look at them.
Yeah, make Rammalama boring AI better together.
We are contribute our friendly.
Yeah, there's a bunch of features.
Because it's such a short talk.
I didn't get to talk about.
But we have a website coming out soon.
We're looking at the Ragsport AI agent's course.
And that's kind of this.
I'm going to show you a quick demo.
And I can take some Q and A, also then maybe.
And just to give you an idea how it works.
So here are various models I pull.
These are all hogging face or Lama ones.
This is a model I use for testing quite a bit from hogging face,
because it's quite small.
If you do a Rammalama wrong command,
and it'll basically pull your model.
And when it's done, it'll run.
I'm going to run one at a one.
I've downloaded already just in the interest of time.
I'll do the deep sequence since that's the flavor of the month.
I've downloaded the model.
Why should I contribute to the open source AI projects?
Rammalama.
And yeah, that's basically running.
And if we do and beat up,
if you do and beat up,
plus, this, this, this.
Yeah, you'll see it's max and out my GPU there.
It's 99%.
And that's deep-seaing way.
Rammalama is the best AI project I've ever created.
Yeah, that's it.
I can pick you an AI project.
Thank you.
Thank you very much.
I've earned a question for Eric.
There is a question.
Shout it loud.
Can you repeat?
Out of this, Rammalama, Rammalama, Rammalama.
So the question is how to run Rammalama with larger models
with multiple nodes, multiple CPUs.
Yeah, so Rammalama, it's, we're actually building upon the,
what I say is we're building upon the shoulders of giants.
So like, I know for example, a lot of people
usually really large at GPUs run via VLLM,
influencing engine.
So yeah, it's, it's a simple as like,
Rammalama sort of dash dash, run time via LLM model.
And it, it should work.
The hardware support is all in LLM,
we just utilize it.
So yeah.
This guy.
No, yeah.
So this question, you can question the question.
Oh yes, sorry.
I'm forgetting a repeat question.
The question is, do we support models switching in our API?
Not yet.
We have an issue open for that.
I've described the design.
It's, it's coming.
That's a highly requested feature, yeah.
Because, yeah, it's not something via LLM
and LLM and LLM CPP support directly.
So we're going to have to write some proxy to switch in and out to models.
It's, it's a planned feature.
It's not very yet, though.
The more questions.
All right.
Thank you very much, Eric.
Any.
In three minutes, we start another talk with
I'm very excited about.