Okay, let's quiet the room again, so we're back to Long Talks, the lighting talks are over.
And the first one would be on Dynamo, so part of this film and video, and actually the first
time I heard the Dynamo talk was a few months ago, so it's another albumer, since
the difference is clear, and actually very curious to hear what changed, what's new.
So please welcome, daughter, talking about Supercharger and LOM, sort of in this Dynamo.
Hello, hello everyone, yes, I'm Paul, I'm a software engineer at Nvidia, and I mainly
contribute to Dynamo, specifically for the design-graded serving part, and as a part of that
I also contribute to Viel LOM, which is one of the infrasemges that runs under Dynamo.
So what is like, where does Dynamo place, it plays basically on top of infras frameworks,
and infrasemges like Viel LOM is DRONK and Lama CPP and others, and this is a serving
library targeted mainly for large-scale distributed serving, but not exclusively for that.
And it's actually a collection of libraries, so there's the main one Dynamo, but there are
a lot of tooling libraries around that, and all of those are open-source, and we already
have over 200 contributors, and we try, this is mainly not like originally in the Nvidia DNA,
but we try our best to be, to work in an open source manner as much as possible.
Yeah, so like the ultimate goal of Dynamo is to speed up the inference and allow large-scale
inference to run as fast as possible, and this is our main objective, this is taken
apart from the semi-analysis inference marks benchmark, so basically in the x-axis we have
how many tokens per second per user can generate, so this is from the perspective of users,
how fast the tokens are generated for you, and then only y-axis we have how many overall
tokens per seconds are produced by our system, so this is basically how cheap is the inference,
right, so we want to be top-right, and we have some success in that matter. So maybe I will just
quickly preface what was the original motivation to create Dynamo, and it was to allow to create
an open source framework for disaggregated serving, and basically if someone's not familiar,
we have two phases of an engeneration one would be to compute the context, which is like pre-feeling,
and then once we compute the context, the prompt we generate token by token, and those problems are
computationally very different problems, one is more compute-bound, the decodes is manually bound,
so we want to optimize them separately, so we need to separate them first, so that solves some
problems, but creates many, many other problems. One of the points for example is that we create
different generation, different configuration for pre-feeling, for decode, they can span
different number of nodes, on the data center, we need to scale them independently, we need to
route requests between them, transfer, decay, decay between them, and so on, so a lot of work to
be done to actually make it worthwhile, and why it's so hard in practice to have efficient
large-scale system is, for example, in the benchmark mentioned both, it was running on
the 300 GPUs, so we are so need to be able to route between all of those, we need to transfer
decay between all of those, and the throughput on the front end, on the routers, and so on must
be, must be really high, right? And also since the models are spanning, sometimes two nodes,
sometimes eight, 16 nodes for single-inferencing, the photo runs also becomes quite a big
over problem, we don't want engine failure to compromise our system, and then another one is that
in practice the load changes over time, right? So that's why the good model is not enough to
have a successful product, successful deployment, you also need the good system built around it,
so we can, in fact, serve fast inference, but also tip. So these are the technologies that
dynamic uses, so going from the top it's main, main design to be deploying Kubernetes,
not again, it can be extendable, but this is our main news case, then we have the observability layer,
then we have discovered a plane to discover different dynamo components that also
have been mainly in Kubernetes, then the components must communicate on the communication planes
for the communication plane, and then we can run actual inference on the endings, and then we have
other tooling in dynamo like nix and kvbm to enable kvc transfer and other kvc manipulation
and optimizations, so these are main components, and we try to be a dynamo to be as modular
as possible, so basically other than the bottom layer to discover the communication plane,
you could take out any of other components and work, and you can also implement your own
components, because basically each main component like previous decode router, front end,
is basically only an async generator with multi-in and single-out generation generator,
so that's also for example for inference engine like vl, and it doesn't have a thin
wrapper on the end because the engine is async generator already, right? So we could go through
for the components, so maybe some of them will interest you, some other don't,
they could also some of them be used like planner or router outside of dynamo, for example,
lmd, which is another inference framework mainly developed by Red Hat, so yeah, so the
discovery plane, it's very simple when you spawn new components, other components must
be aware of that, and when some components are scaled down or fails, also other components need
to be aware of that not to router, and for that we simply use Kubernetes, but also outside of Kubernetes,
we have a CD implemented as a backend, and for single node or local development, we also have
simple file system, backend as a discovery plane, not much more there, communication plane,
this simply TCP protocol to communicate between workers, between components, for example,
the request, the prompt, the response, and so on, right? Also there's a front end, which is again,
simply open AI, compatible, front end, again, not much there, I will mention that most of the
components are written in, and the whole dynamo is written rust, but the public API is maintained
in Python, so for example, the specific implementation of workers is done in Python already,
right, but everything underneath here runs on rust, for example, for the front end, so first,
more sophisticated component would be the router, and basically router's goal is to have as high
of a cache-hit rate as possible, right, because for example, you are cutting with your
cloud GPT, and your first quest landed on a specific node, and then you come back two minutes later,
right, and you want to continue your conversation, so we want to route to the
probably to the same node, that it has already kbcached from your previous conversation available,
and not to route somewhere differently to recompute or have to transfer the context from
different nodes, right, so that's the goal of the router, and there are actually two main variables
that router act upon, one is the kvindexer.tracks, which which basically which prefix is available
at which node, and also slot manager, which tracks how busy is a worker, right, because maybe
cache is available, but worker is too busy to accept new work, right, so that's what the router does,
so it just combines those two variables with some weight, which would be a tunable parameter,
but you can also use round robbing or random routing, right, and supposedly it works,
it enables, we've heard from some customers, it enables much, much improvement in time to first talk
and basically from higher cache hit rate, then you have the disaggregation self, so for the
disaggregation menus, the nixelibrary, which is like part of our ecosystem, and nixel is just a
peer-to-peer communication library, so you'll probably heard about nickel, which will be collective
communication library from a new video, and nixel is designed to be a peer-to-peer communication
library with nixelibrary, so you can write your own plugins to support different, for example,
file system, so we use nixel to communicate between tip-to-tip, but also from tip-to-host memory,
and from host storage, lock-astorage, or shared network storage, right, so for disaggregation
menu uses for tip-to-tip communication, so that's how our disaggregated flow looks like,
so first the request goes to the router, router will pick the best prefil worker to handle,
though, that request, prefil worker will do, it's think it will compute the context,
fill the cave cache, and then router will send forward this request to some decode worker,
which will allocate locally cave blocks, it will meet those cave blocks using nixel from the
prefil worker, and then to start generating also in form prefil worker, that it can free up the
decade is already, so this is something, for example, we've implemented together with red-cut team
in NVLAM, so we added what's called nixel connector and some scheduling optimization to NVLAM,
we've done similar work with SGLANG and TRT LLAM, okay, so I will mention nixel, also on top of nixel,
we've built something called KV block manager, and this is also a tool that can be plugged into
VLAM or SGLANG in form of the cave connector, and it's goal to use the hierarched call memory
to reduce number of eviction you have to do in the cave cache, right, because for example,
you will return in one minute, maybe the cave cache will see be there, but if you return to your
cloud in five minutes, maybe there are so many requests for other users in between,
that there are no space for the cave's, so they're got thrown away, a victim from the cave cache,
and now this is to compute that, and yeah, that's costly, right, especially for prefil, it's
the attention cost is n square, right, in terms of the input size, so we want to first
instead of evict move from device memory to host memory, then we want to move to local storage,
and then probably to the network storage, right, if they did the context is long enough,
and the file system is fast enough, it's still makes sense, sometimes it might take longer
to read from the external storage, but you still save a ton on the compute, right,
so that's what cave EM is designed to do, is to extend the life of your cave cache.
And when you use that, you can again see improvement in time to first talk and because you avoid
the computation, which is both costly and time consuming, okay, so then we have some
totally loosely connected with dynamo, that can easily be used outside, like I mentioned before,
one is plan, right, so one of the challenges I mentioned is that the load of the system is
staging over time, right, so we can assume there are some peaks, peak R, peak usage hours,
and we want to scale down profile decode front ends dynamically, preferably, so that's what the plan
is doing, it's confirmed offline, before start deployment, given SLAs, proposed starting configuration,
and for that we also have this tool called air configurator, that you can simulate what will
be the best configuration without even using the GPU, but given hardware and model, we should be able
to wrap the estimate the best starting configuration, and then in real-time planer will
scale the decode, preferably, because up and down to satisfy the SLAs, yeah, and for that, again,
we have small Kubernetes enhancement called growth, and that is used to scale the
multi-node deployments easily, right, because since for example our inference entry is using
four nodes, we need to scale the group of nodes together, and that's why we also develop this
growth, so for example, maybe you've heard from red hudders leader workerset, this is something similar,
maybe more tuned to 20 video hardware, so, and to end user experience would be they would create
some configuration file, they will specify, they want to use front end, and decode worker,
they will give the dynamo command that they can specify, which model, the parallelization technique,
and that will be parsed by dynamo operator, that will enhance that with some
inference entry specific arguments, for example, to handle the setting up master,
opening master ports, passing master addresses IP address to workers and so on, and that,
then will be passed to the growth operator that we actually create all of the resources,
and then deploy them to to your Kubernetes cluster, yeah, a pair of simply a small benchmarking
library that you can then benchmark your open a endpoint, and then more to express this is another
very tiny library that we have in our ecosystem, that's meant to quickly help quickly load
engines, and also developed engines, right? So, yeah, and that's basically all the components,
that's it, thank you.
Any questions? We have a couple of minutes.
Yeah, my question is, how much of it is LLM specific versus more generic to be used with
reduction models, VLMs? Is it LLM specific?
No, so one thing is that dynamo is really nothing specific at all, so like dynamo runtime
itself, so the interface that you can implement for components that we get discovered by each other,
and then on top of that is what we have already implemented, right? So, we've focused on LLMs,
but also on multimodal models as well, but there's nothing dynamo itself that's specific to LLMs,
if that makes sense.
Any more questions? No, last chance. Thank you. Thank you, brother.