there serve hostels loud slang with model materials it's all years
hello everyone
Hello everyone, so my name is Mateusz, and today we will be talking about
Padler, which is a tool for self-coasted large language models at the scale.
And to start with, as an introduction, I would like to mention that Padler is a tool
for LLM-OPS practitioners, and this is important because a lot of people do not even want
to acknowledge that LLM-OPS exists in itself, but I really think it deserves its own niche,
it has its own place, because the tool link that we need to deal with when it comes
to AI and the inference and production usage of those is much different than what we
see usually see in DevOps, and what we see when deploying for example classical websites.
And we are still figuring out as an industry, what works, what doesn't, it is still
an infancy stage, but at the same time, businesses who want to use models in production
self-coasted models, yearn for some kind of stability.
And this is kind of what Padler comes in, Padler is primarily a load balancer, custom
tailored for LLM-OPS.
We are working on more tools related to the infrastructure, but at the time it is primarily
a load balancer, and it aims to provide scalability and some resilience in production.
And to explain what Padler is and how to get there, we will start from the beginning.
So if you are wondering how to end up with a project like Padler, you can follow these
few simple steps.
So let's say you want to host an open source large-fog which model in production.
So you will probably want to start with a single host and you need to pick a runner.
So you can pick between like VLLM, LM, LM, LM, or LM, which is also based on LM-CPP,
maybe something else.
I chose LM-OPS-CPP because of how fast it implements new models, of how fast it moves
forward, and how many internals it exposes.
It is really easy to build custom tools on top of LM-OPS-CPP.
And let's say we don't know that LM-OPS-CPP, we don't know that weights from hugging
space or whatever else, we installed all that on a server, and we can quickly notice
that something does not add up quite.
Because even if we take some smaller model, like 10, 12 billion parameter model, like
Mr. Nemo for example, it requires about 10 gigabytes of beer.
So you have 10 gigabytes of beer, I'm the cheapest solution, is like something with T100
GPU, which currently costs about $300 a month for NAWS, and for that we can have an
inference speed of about 200 tokens per second.
The way concurrency works, it more or less divides tokens between users of the service.
So if you have two users, we will have less than 100 tokens per users, in general, it
is a generating speed.
If we have about 80 users, we will have like 20 to 30 tokens per user, more or less.
And to me, like 20 to 30 tokens per second is something on the verge of usability.
And if you can lower than that, I think that makes your service unusable.
So does this mean we can deliver this kind of poor quality experience for 300 dollars
a buck, is this it, and the answer is resounding yes, we are coming back kind of to the
90s with this, and what do I mean by that?
We are so used to the fact that we can buy a server for five bucks a month, and we can
install some kind of web framework, and we can handle like hundreds or thousands user
concurrently on that service, and we are back to the situation where we need to expensive
hardware, and we can handle maybe thousands of users at once.
So in this sense, we need scalability from the start, and this is not only a technical
issue, this is also a mentality issue, because when we start a new service, everybody tells
us, you do not need scalability from the start, just push it on production, see if it
gains some traction, maybe lighter your needs scalability, but we need that from the beginning.
So if we need any kind of scalability, it looks like we need some of the balanceic also,
and there are options, there are a lot of load balanceic algorithms, but let us go through
the most popular ones.
So the simplest one would be round robin.
So the way this works, let's say we have a few service before I'm at the CP and some
modern running, and the way round robin works, it will distribute requests in circles.
So request number one goes to server number one, server number two, server number three,
and then back to server number one, just in circles.
And this approach works well, if the response times are more or less the same all the
time.
For example, in rest API, so, but the issue with LLMs is the response times are really
variable, because there is a seed of randomness, so even the same like the same prompt
can produce various results, and if we are exposing those prompts to users, one user can
for example ask a yes or no question that model can answer, and the other user can ask
model to generate something like half of the book, really, which we'll take a few minutes.
And if you are unlucky and you will be, those requests will pile up on one server, while
the other servers are not busy at all, so that is not good.
So maybe we can monitor resources used, and this is really tricky also.
It is possible, but it is really hard to get it right.
So the GPU usage, for example, if you use it, will stay more or less the same, no matter
how many users use the service, and you use, for example, continuous batching, as an algorithm
for parallelization, and so you would, so to match those differences, you would need to have
some kind of high resolution monitoring to check very often the GPU usage, and again, we
are used to the fact, for example, with websites, that the more users we have, the user
gradually grows, and is rigid of the text moving to scale up or down, but it is much harder
here, so maybe something else.
The option number three would be to use least connections.
So we can, for example, maybe serve inferences over HTTP, and from the load balancer perspective,
maybe we can issue new requests to the servers who are handling the list amount of requests
at the moment, which in itself would work well, but the issue is, the resources are so limited
that we need something application specific, we need a lot of blancer that knows what it is
balanced things exactly, so it can react accordingly to this situation, because again, it
is really easy to clog up servers with new requests.
So this is finally where parallel comes in, so it uses different balance categories depending
on the end points, so when there is an end point that uses inferences device, it uses
least connection, plus it uses LMSPP internals to know how many resources the server can
handle, so it doesn't bug it down further, it introduces some kind of resiliency to the infrastructure,
which are very explain later, and also it is capable of sending stats to the metrics if the
or infrastructure needs to scale up or down, which is, for example, usable in stage
environments, because it can also scale from zero host, and the way parallel works in general,
you do not specify the workers or LMSPP workers from the load balancer level, instead you
stop the load balancer, and the agents that are installed alongside LMSPP, register
themselves in parallel, so it is kind of reverse to what most load balancers do, and this
allows parallel to scale from zero host, because which means you can start some chip compute
instance, and it can start more expensive GPU or GPU or whatever interface device instances
when they are actually needed, it can also pose as LMSPP server itself, because it
for example, it can also aggregate some metrics, for example, if you have two LMSPP instances,
and one of them is configured to handle eight requests at most, and the other one for example seven,
it can pose itself as a single LMSPP instance with 15 slots to handle, it can also buffer
requests, delay them or redirect them, to LMSPP instances that have free slots to handle them,
so for example, if your infrastructure is scaling up or down, it can buffer the request,
it can wait until a new host appears in the infrastructure, and only then it will direct
it to that LMSPP instance, so it helps to avoid situations, for example, it would be enough
for user to wait just for a few seconds, and the other request would be handled, and otherwise it
would be dropped, in its self-pudder is lightweight, it is based on a pink or a framework from
cloudser, so if that kind of network framework is good enough for cloudser stock, it is also
most likely good for your stack also, and you can make mine, and the heaviest pink it adds on top
of LMSPP, I just have checks, which I think has standard pink in application lower balancers,
so it does a health check once 10 or 30 seconds, mostly for the integrity sector, see how many
slots are actually left and LMSPP, and that's it, so scalability is also simple, because
it can pose since it can pose as LMSPP itself, it can aggregate some statistics, you can experiment,
you can, for example, maybe balance paddlers with paddlers, you can use maybe some other
products in front of it, like actually proxy, so you can experiment, this is also useful for
the stage, you can buy elements, because you can set it up, so only a compute instance is running,
and it will only start more expensive instances when they are actually needed, I have also seen
people who are using paddler in their home labs, for example, so you have Raspberry Pi 0 running all
the time, if it needs to do some inference, it starts your GPU device, before you can learn or
stuff like that, so it has lot of uses, high availability is also kind of easy, because
you can keep paddler instance in your, for example, backup region, because sometimes there are issues,
maybe I am unlucky, but in the last 16 years, the data center in which I had servers burned
down, so you do not really need high availability until you do, so the idea is, for example, you
can keep your infrastructure in the primary region, and you can have a backup region in the
on a stand-by, and if your primary region goes down, you redirect requests to a backup region,
and in that backup region, you don't have to keep your entire GPUs, like GPU, infrastructure is
stuck up all the time, and how does paddler compares to other projects, so paddler was not
made to compete with any other project, paddler was made to help projects cooperate,
as still we can compare it to some other projects, and people often ask me about LAMAS CPP RPC,
so LAMAS CPP RPC is built in LAMAS, it does similar thing to paddler, but in the opposite way,
you define the hosts in LAMAS CPP in the network, and this is a static setup, so you just attach
inference devices to LAMAS CPP, it will distribute requests, but paddler is more focused on the
infrastructure, it does those metrics, it can scale from zero hosts, and it makes sense to use
both really at the same time, because you can configure an infrastructure, for example, in a way,
when you can bring up small clusters of LAMAS CPP RPC and let paddler manage them, and balance
requests between them, so you can kind of look at paddler as reverse proxy, by LAMAS CPP RPC's kind of
a forward proxy, and for the future plans, we are considering bundling LAMAS CPP with paddler,
because that might help, because currently you need to have compatible version of LAMAS CPP with
a compatible version of paddler, and if we bundle both, it will avoid such issues, we are also
considering adding support for other runners, for example, LAMAS, and we are planning to expose
semantic versioning AI, API, but not only for inference, but also to manage servers, because we want to
go further into this infrastructure and operations path, and I would like to use this opportunity
to give a shout-out to contributors, especially Luis Miguel, who recently contributed a lot of
awesome stuff, like console dashboard and initial sketch of Supervisor, but really to everyone,
and if you would like to join our community, we have a GitHub Lab for Discord server, you can
reach out to me personally, and thank you for your attention, if you have any questions, I will be
happy to answer them.
We have a few minutes for questions, so if somebody has a question, please, and shout it.
Okay, no questions so far, thank you very much Matos.