Okay, so the next talk is from guys from Padler.
They actually have done it all across the year as well.
So, a part of this year would be about what happened during this last year.
So, welcome, so, my question is heating.
Hello, nice to meet you.
We are extremely happy to be here.
My name is Mateur, this is Gosha and Miguel.
And we are co-authors of Padler.
Which is a tool for a self-hosting lens.
We are all in the Lama CP ecosystem.
We have introduced Padler last year here on Phosphum.
And now, we wanted to talk about what worked for us,
what didn't work, what works well with hosting models or not.
Because I think we have this kind of unique perspective,
because we are working both on the infrastructure,
but we are also creating products for casual end users.
And generally, we can see two paths that open-source elements can take.
And one of them is doing hobby projects,
and we need topics that are only relevant to us.
Which is fine, of course.
But we also notice that LMS, an open source,
LMS have this potential of being this experimental frontier
for research for products.
I wanted to focus on this more.
So, Padler, we wanted to create a tool that can show that Lama CPP
can be used successfully in production at a scale.
That it can, we can use to the fullest its potential.
The fact that it can run anywhere.
Because we also initially saw a third path with us privacy.
But unfortunately, most businesses just don't care about it too much.
I'm not judging it, but this is the fact.
I mean, most businesses we approached would rather use
some big vendor like OpenAI or Claude or whatever themselves host.
Which, again, it is what it is.
But still, I do think open source LMS has this great potential
as this experimental research frontier.
So, in this context, where does Padler fit in?
So, Padler is our open source software that allows
to self-host large language models.
And we have been developing it with the thought of having something
which is very easy to handle that follows the best LMS practices
and that you can deploy it anywhere as well.
So, there are several environments that you can put Padler in.
You can use that on your premises, on your server racks,
or even on the cloud if you really want to.
And if you got really clever with the idea of Padler,
you can even use that to create some company-wide second-brain.
So, everyone in the same office would be able to connect
to the same Padler balancing stance and share the resources.
So, this venue here, for example, everyone would be able to connect
and share the same computational powers by using Padler.
And that's very interesting.
And talking about Claude, we are actually using Padler on the cloud.
And there are some features that we have created to make such deployments easy.
Like, you can use your metrics from the Padler status D
to, in the case of AWS for example,
set some cloud watch rules to bring more or less GPU instances in if you need.
And the cloud where actually one of our priorities,
because even on the cloud, you can have some benefits
over using your own infrastructure.
And with Padler, you can even set some
multicloud setups where you keep your Padler
a balancer on AWS, but keeping your agents on OVH, for example,
or any other kind of cloud.
So, that's very nice.
So, initially Padler started as just Lama CPP load balancer.
But since then, we evolved a lot, changed a lot.
We still keep Lama CPP, but just as an inference engine,
inside Padler, so now it is baked in,
there were several reasons for it.
One of them is, we needed semantic versioning
to do like predictable deployments.
Lama CPP has a rolling release, which is fine, has its marries,
but we needed to have this ability to limit
to have back what's compatibility or to,
like, have predictable deployments of predictable versioning.
So, that was important to us.
Second, we have this liberty of adding more custom features.
Initially, we relied mostly of Lama Server HTTP
and points to monitor its status, its internals.
But now, we are using C++ by next directly.
It's baked in, so Padler is also easier to deploy.
Thanks to that, because it is contained in just a single
binary that you can put in a server on a laptop.
So, it's very simple to handle.
You can argue that we decoupled from Lama CPP server
and stuff, but now we rely on C++ bindings.
But still, it is much easier to maintain Padler that way.
We can, for example, pick some specific snapshot of Lama CPP
to make sure it works correctly, release it with semantic versioning
and so on.
There was also another reason, because at one point,
there were deemed slots.
Feature of the stamping issue didn't really use in production.
But we were using it a lot in production.
So, we pretty much had no choice.
At that point, we just rewrote it and slots.
I mean, the feature that allows you to divide context,
memory context into several slots to handle request
inference requests in Padler.
It's something really crucial to us.
I think we must have done something right, because we got some positive feedback
from you from the community.
And I'm especially grateful for that.
I'm especially grateful that it was to us.
I know it is not easy to put your production service on some open source
projects from some guys, random people.
So, I mean, thank you for that.
And we treat this seriously, because we really want to take
make sure that, I know, memory, existable.
It can run anywhere.
So, thank you again.
And this is kind of a tool of our UI and some work capabilities.
We have this work admin panel.
We try to make it really accurate.
So, it represents the really how the system works.
So, it always shows you in real time how many slots
we keep either the inference addresses that and so on.
And we use it in various ways.
We use it in production.
We use it locally.
So, for example, sometimes when we work together,
we connect to the same wife.
For example, Goshia has a Mac.
We've unified memory.
She can load bigger models.
And we can assign like eight slots to her Mac.
Like it was close to mind PC.
So, we use it like that locally.
Nice thing about it is, when you deploy something,
you need some staging environment, some production environment.
So, Lama Cvp can run on anything.
So, you can assign some agents to like cheaper CPU instances
and staging, when you don't need to have as many users, for example.
And you can assign it to GPU instances on in production.
We also have features for scaling like buffer requests.
We expose them data for stats these.
So, you can automate the process.
And this is the panel for loading models with parameters.
So, we support about hugging phase and local files.
And so, you can either provide a link to the GF model on the hugging phase
we use hugging phase API.
Or you can just deploy the model on the server and set the parameters.
So, some of those parameters are local to Lama Cvp.
Some of them are local to padler.
But you wanted to collect them in this single place.
So, you can just change all of them with a single API request.
And what I think is interesting, we try to follow hugging phase conventions
because often model authors put parameters.
In red means it is in specific way.
And they put some specific parameters recommended for their models.
So, if you deploy something, you can pretty much copy paste them.
We also have a feature.
So, you can provide your own chat template.
Because they are often baked in into GGF models as a metadata.
But I don't know, let's say,
in the template exchange, it doesn't support some obscure feature
that the template in the model uses.
You had least have a chance to do some work around or anything.
So, you don't have your hands tied.
Because, you know, situations happen.
And we really want to feel comfortable deploying this to production.
And you can preview, of course, how the model works.
So, this specific model has a thinking phase.
So, we also try to look nice by the way.
So, we really wanted to make it look like the thinking phase.
Is this note on the margin of booklets?
So, I hope you like it.
We also keep everything documented.
So, we keep documentation always up to date.
We also documented the internal,
so, if you want to dig deeper into that, you can.
And we have some plans for the future.
So, Lama serve an implemented multi-model capabilities.
And since period route, most of it, we need to port it to parallel.
So, this is our top priority.
We also need some more compatibility and points.
But, we are open to adding.
I don't know, Gemini or Crowdle, whatever.
This is also a good first issue.
If you want to contribute something.
Because I think it is quite easy.
You only need to map our internal data structures into,
like those third party data structures.
And we also want to experiment with browser agents.
Because I think it would be interesting, for example,
to have some of the agents from a cluster,
to be spawned in browser, some of them be part of the server site.
So, we are very flexible with this.
Okay.
So, now I want to talk a little bit about what kind of projects
become possible when we have this stable self-hosting foundation,
like Padler.
And first of all, this begs the question,
when does self-hosting make sense?
First reason is, of course, costs.
Of course, when we use huge models that are comparable
to what we typically get from closer providers,
we have this significant cost in the form of the infrastructure,
that's the upfront cost.
So, we normally need to generate a little bit of scale
in our product to start noticing the lower costs.
However, the cost efficiency gets much more immediate
when we start to use some specialized smaller models.
And also, with self-hosted approach,
we get far better cost predictability,
sometimes even also fixed costs, regardless of the scale.
Then there's data privacy, and this is as we mentioned,
maybe not something that is entirely recognized by the majority
of the businesses.
On the other hand, there are certain use cases,
certain whole industries, even like the medical field,
where the data privacy matters a lot.
Then there's the question of the performance control
over the latency.
Here, the reason is clear, you only infrastructure,
so you'll have full control over it.
And finally, there's the better user experience.
So, this is something that we get from being able to use
a smaller specialized, finite models,
but it's also something that we get thanks to the fact
that we are not dependent on some silent updates
in vendor models.
So, I think that the reasons for self-hosting
are strong, but they want to make them very, very concrete,
and talk about a few projects that we are building on top of
Padler, and that we hope represented this experimental
frontier of using LLAMS.
And my hope here is that this is going to give you
some kind of inspiration for your own projects,
and that this is also going to give you some solid arguments
for when you need to get the buy in from the business,
or from the product, and when you need to convince someone
for the self-hosted approach.
And the first project is poet.
This is another open source project that we're working on,
and this is a static site generator with a built-in
MCP server, and we are now combining working
on adding some LLAMS-based features.
So, poet is designed to give full control over the abstract
syntax tree of both map-down and HTML,
and what we can do is that it exposes LLAMS-based API,
and we can do things like text analysis,
or cross-referencing between documents,
and Padler can very easily plug in two such an environment
and be used to, for example, analyze text.
So, for instance, we can check that the metadata
matches the content or we can identify
a single source of truth across our documentation.
So, why poet may need self-hosted models?
One is that different content requires different understanding.
So, if you have a blog about Rust programming language,
this is going to give you a different vocabulary
than let's say a legal website,
and with self-hosted models, users can find
you on some small specialized models,
and get far better consistency checks.
Another thing is that with content,
not all commercial models work well with content.
Some kind of content, like, for instance,
imagine that you are building a website
with some mature themes, or legal content
that describe crimes.
Some commercial models are going to just refuse
or hedge, and specialized self-hosted,
find two answers or model is just going to do the task.
Another project, this is something we're working on
that I want to start up for context.
We are building a no-code tool that is going to allow
our users to build systems by describing
some general business terms business domains
instead of having to explicitly define
every database schema relation or work.
And for that, we need to come up with
some way of different way of representing systems
and applications, specifically,
we need a different way for storing data.
And we came up with the idea of using RDF,
so there is a description framework.
And because we also want our users
to build using natural language,
we need something that is going to take the user input
and translate it into Sparkle queries,
and Sparkle is the query language for RDF.
And we came across the project called Fire Sparkle,
and this is a research project,
and it's goal is to take the user input
and translate it into Sparkle queries,
so this is exactly what we need.
And for the researchers,
it was enough to use a 70 billion parameter model
to prepare the data,
and then just eight billion parameter model
final model to process the user input
and translate to Sparkle.
Also, with self-hosting and specifically with
MSVP, we can use features like grammars,
and this is something that is going to
further constraint the model output
and get what we need.
So, could we achieve that with commercial models?
Theoretically yes,
but the self-hosted approach is far better,
because let's consider what this approach means
from the product, business,
operational perspective.
With self-hosted, fine-tilt models,
and features like grammars,
we get far better model output,
so this means far better user experience in our product.
We also get better stability,
like if we used commercial API,
we could risk some changes,
some updates in the vendor model,
we would change the translation quality,
and we would risk random regressions.
With self-hosted models,
it's much cheaper and much more convenient
to self-host the small model,
like the eight billion parameter one,
data privacy is also fully within
our control.
So, we can see that we can build
a complex but coherent system
that is going to serve multiple users concurrently,
and open it for that,
our models from having faith,
patler, and a little bit of patience.
So, of course, those were just two examples
of what you can do,
once you have a control of your
inferencing infrastructure,
but we really hope that when you leave this room,
you will keep this thinking
that the experimental front says much more accessible than you think,
and that you can, for example, use some
function to build a model to power a real product.
So, if you need something which the commercial API is missing,
that can be some more specialized models,
some constraints,
or even if you want the overall predictable behavior over the time,
you can see that the open source is not just an option,
but it's actually the best choice that you can do.
And if you want to reach out to us to talk about the project,
about padler,
and point our sensors and our mobs,
feel free to do that.
We will appreciate a lot,
and we are also given away some pins.
So, if you reach out, you can get one as well.
So, that was it.
Thank you everyone for listening.
Thank you.
Any questions?
We have one minute for questions.
No?
Okay, one question.
Thank you.