All right, time for the, for the next talk, just quiet, quiet everyone, all right, so every
year we try to find, you know, a presentation and sometimes they get submitted, sometimes we
have to find, you know, people introducing a new and exciting inference framework, because I think
it's all about those types of things these days. Like again, you know, you can sort of do a lot
of things with PyTorch, but that is super boring. And it's not really inference optimized. So last
year, it was the ML this year, it's tracked. So Julian and Matthew take it away. Hey, um, no, no,
okay, well, thank you very much for attending this presentation. Julian and myself are both
working as software engineer developers at Sonos voice control in Paris, and we are here today
to introduce two open source libraries. One has been in a for a while, it's tracked, and the other
is a companion project that we open source last summer. So we, just quite seeing the surface, we have
just a couple of minutes to introduce tracked. So what you have to remember, it is, it's a generic,
neural inference neural network inference library. One of these characteristics is that it's tracked
all the way down. It goes from passing your model to executing it, so it contains the matrix
multiplication stuff and optimized routine that we need. It's not new, it has been around for nearly
ten years, and it's been used extensively in Sonos, but some of the company have been using it.
It's written in Russ, it's meant to be very integration friendly, and it's released under a very
permissible instance. So just to give you an idea of what it looks like to use tracked,
there's a bunch of code here, but very little is actually tracked code. The first two lines show
how to instantiate the NNF parser of tracked. The second line show how to load a model,
transform it into a renewable form where, where, which you can actually run, which means finding
out the right order of operations, stuff like that. So that's the first two lines at the top of
the screen, and then you have a bunch of stuff which is just getting an input ready for processing,
and finally you get a model.run, or you actually run some model on the image, and the rest is
post processing and displaying to make the example walk. So very simple to use. Another facet of tracked
is it's common line tool, so it's designed to be able to first dump a model, so that here you can
see the big enough of a mobile net model. You can see the convolution with some details, so it's useful
when you're given a model and need to figure out how you can make it run, that's your first stop.
But then once it's running, you can actually get insight on the performance, here you can see
a profile of the same mobile net network, where you can see the depth-wise convolution takes
nearly 10% of the entire execution time, and you also get some insight on the arithmetic's
intensity of the operator, and so you can design if something is what you expect or if you need to
invest in it. As I said before, it's meant to be very easy to integrate, so one of the key
point is it doesn't have any big cumbersome sort of party, it's very easy to cross compile because we
are using rest. It has support for resin, so you can actually run some model, a new browser,
and we have a bunch of targets that we optimise for, so I'm 32 and I'm 64, I'll abuse because
well, so knows, but also Intel and wasum, so wasum works with CMED, I mean all of them are CMED,
but that was was MCMED, we are also optimizing for metal and QDAR. Tract as a CAPI and on top of this CAPI
was some Python bindings. Recently we've worked on adding to tracks on GPU support and we still
want it to be very easy, so you can see the difference in the first two lines of code and
so one in the middle of the screen what it means to go from CPU to GPU and everything goes right
on Tract, it's still all the fact we're still working on support, but you can see that in this case
we get a nice improvement on the performance and you can also see I'm in social education to show
how the command link can be used to bench a model. Here I'm going to show some of the
web thing or thing that you might find weird if you play with Tract and on this instance we are
showing the dump of Yamamoto and one thing that I want you to look at is the source as this S thing
which means that Tract supports symbolic dimension of shapes. Tract likes to know everything about
to network before running it so we need to know the rank of each tensor and we need to know
the shapes of each dimension I mean the length of each dimension, but there is a twist you can
use symbol so we don't have to have the exact value we need a symbolic expression to know
how to reason about your model and I have another example here so that's in the same
model it's a Yamamoto, the dynamic key value cash operator what you can see here is the shape is now
S plus P as being the number of token in the current terms or of and two on if you're doing
token generation and the P is the best sequence so here you can have all you prompt
or everything accumulated before and one other thing which is different about Tract compared to
the engine is that the state I mean Tract can do state management so here what you can see is that
the dynamic key value cash operator just have one input and one input it doesn't have to be plugged
into extra model input and outputs for storm management for statement management so state can be
managed by Tract inside a state object that Tract API expose so all of the examples so far
that I've shown our about NNF and we come back to that but Tract also has some support for I mean
it has first class it isn't for onix we support like 85% of the operator set and it pans out to
more model than that we have to do some adjustment because of the knowing all all about the
tensors before hand thing and the Tract protocol before format doesn't contain everything we need
so here you can see the setting putfact line for five where we actually load the model in two
stage first we load some kind of protomodel then we add some type information shape information here
to specify exactly the shape of one input and that's enough in that case to help Tract
infer everything else about the shapes in the model so it's really analog to when you add
when you need to add some type information in the C++ program or in REST program for to add the
compile of figure figure out everything more recently the LN thing happened and it came with
engine that we are no longer generic but we are taking a new approach where basically you
what got the model in the engine and then you provide the tensors on the side and that doesn't work
for Tract because Tract wants the model as an input so for some time we try to manage with
onix but we like that we want to not go going anywhere with this and we add this internal tool
that is called Torch to NNF and that is what Trilion is going to introduce now
thank you Matthew
so we will discuss about now shipping neural network from Torch so by Torch to NNF format
and NNF format which is highly compatible with the Tract inference and giant which is this
list so before going on we just want to set what is a model as set for us so something which
is completely aside from a neural land giant is a list of tonsile names and a graph of
computation that have been defined correctly which inform you the set of transformation that go from
input to output and so this is what you need to have something independent that you can consider
an asset so what is NNF and NNF's and for neural network extension format it addresses the same
core problem as NNX and was specified by the coronavirus group it's something that didn't
match as good as for the community as it could be because it landed one year after
NNX and it didn't have a proper inference on giant to run on but it has some very interesting
key features namely it have a readable graph structure that you can see on the right there so
it's very easy to read you can just it's textual so you can directly use it with your text editor
and it have composition available directly so as you know neural network are repeating blocks
over and over again and so you don't have to repeat over and over again this especially for us
what was interesting is that it's an extensible tonsile format as well that is defined in the
specification which are as well to have extended quantization logic and so now what is dodge
NNF so dodge NNF it's a project that was born in 2022 out of frustration trying to export
NNX quantized model for internal project and it aimed to be a strong bridge between PyTorch
and NNF and more precisely tracked NNF extended version with check upfront at export time that the
equivalent between the two and friends on giant that are the PyTorch one and the tracked one
equivalent for some specific example and so you can see here an example where you export a model
which is very similar from the API of NNX you may be familiar from PyTorch except now you have this
inference target which is there and which allow you to specify any inference on giant but here
tracked with a specific version which allow you to unlock some specific features and so why
it holds to an NNF so as we discussed it unlock some specific quantization and the quantization
function which are more advanced it can relieve you control specific module say realization if you
wish so you can take the and on some specific stuff you can add additional expression for
the symbolics which is discussed about tracked directly baked in the mobile assets and finally
it's more focused on single based neural networks so it will work better compared to NNX to
export this model for tracked so thanks for listening if you have any question fit free to reach us
on a GitHub or just ask us now areas of the documentation as well if you wish to contribute to
use the project thank you very much we plan to stay around the room if you have questions
stuff I also jumped into the matrix group so thank you so much guys I don't see any questions so I
guess will we'll just let you hang out with the people