Okay, so keeping in line with this idea of various types of inference frameworks and what
you can do, for the next talk, we're going to somewhat extreme low end of compute capabilities
that can still do a lot of interesting things in AI and ML.
With that, I would like to introduce Anastasia and Anastasia, who will be talking about
tiny ML, so take it away, guys.
Thanks so much for the introduction.
So I'm Anastasia, so along with my colleague Anastasia, we're going to talk about how we balance
accuracy and inference latency in really resource constraint boards like IoT,
like as expressive boards where we want to run in much inference.
So a bit of information about us, where we're really small team, we do research, we focus
on hardware abstractions, on low level, we do low level OS work, we have gone into the
cloud native space, we do containers, contain a runtime send, we bridge all these things
into a coherent end-to-end ecosystem essentially.
Of course, this work was also done by colleagues that do not work with us at this time.
First, let's see why we think it's an important issue, what we tackle.
IoT devices are everywhere, there are sensors, they have some compute capability, but unfortunately
we can not run large models, we can not run like inference with accuracy on their compute
capabilities, so the reason for this issue is that we don't have so much memory, we don't
have such compute capabilities, and of course there are issues with a network latency, giving
data from the sensor to a more compute capable infrastructure.
What happens at the moment is that we need to shrink ML models that will run on IoT
devices, and we do that using various methods, with the quantization, we prune, we do a lot
of stuff that actually hurts the accuracy of these models. If we don't do that, then we
get a really increased latency, so there is a trade of that we need to consider between the
accuracy of the inference that we run on these devices, and the response latency that we
want. If we want real time, we cannot be accurate, if we want to be accurate, we cannot
get real time responses, in general, so we consider, we will see that in one of the next
slides as well, we consider IoT devices like ESP32 devices, microcontrollers, we don't consider
IoT devices at us, but we will consider that device as an edge device. In this landscape, let's
say on the IoT device, we need some kind of compute capability to be able to process
the input, and we consider using edge devices like Raspberry Pi, and then with the adjation
with the GPU, or even a cloud device to be able to do this actual computation. Of course,
in order to orchestrate all these deployment, let's say software deployment and do the
actual inference and get the results that we want, we need to consider as well the security
aspects, so is this device legit? Does it feed me with data that I trust? Can I trust
the other device that's running the actual inference? It's kind of a mess. I think that's
what I kind of said. What we want essentially is we don't want to change the application
code that we deploy, even if it is on the IoT device, on the ESP32, even if it is on the
Raspberry on the jet zone, wherever, we want it to be secure by default. We want to be able
to run in the whole continuum, it's a buzz word, but what we mean is that we want to
be able to be cloud native, so deploy something and make sure that this can run on a cloud
server, on a Raspberry, on an IoT device, wherever, and we want to be able to run inference
efficiently in the infrastructure that we have available. What we actually built for that
is a ML IoT. This is a large framework, in this talk we're going to talk only about the
inference of loading, but just to give you an idea, we securely onboard devices using this
framework, we have a mechanism to attest both the device itself and the application that
runs on this device, so both with secure boot and with EATs to make sure that we run
what we want to run. We package all this, all this firmware and the application that running
on an Edge device, let's say in an OSCI image, so it's a container-based thing, and we
have this framework, we excel, where we use it to transparently upload, compute intensive tasks
like image inference, to enable a node that is able to run this kind of models. I'm not
going to bore you with the cloud native framework, I'm just going to say we use dice
for the device identification, to generate a certificate based on a unique device secret,
some of the ESP32-based devices do support that, so we are able to make sure that the device
is legit and we can onboard it into our cluster, into our trusted cluster. We use a EATs
to make sure that the application that runs on the device is legit and using this cloud native
framework, where we built an open source component called Acre to identify the devices
to do the inventory, and we end up with a cluster using IoT devices, Edge devices and
cloud nodes, and we are able to run applications there and make them communicate with each
other. This is a rough sketch of the architecture of the cloud native framework, I'm not going
to bore you with the details just that we use Acre for the inventory, so we discover these
devices, we onboard them using dice and EATs, and then we can deploy over their updates on
this device, so we can repurpose these devices purely in a cloud native way. We consider, as I
mentioned earlier, these kinds of devices in the continuum show we have ESP32 as the IoT, and
point we have Raspberry and the Jet Show, and then via Jet Show with a GPU as the Edge devices
and the cloud server with a GPU, a desktop-like GPU. Now the framework that we use for the API
remote operation is something that we have been building for quite some time now, so we
we call it the Excel, we started developing it for GPU sharing in VMs, we ended up using it with many
many ways, so the core concept is that the application consumes a high level API, so it
imagines inference, torch, load, torch, run, a TensorFlow, load, or session creator run and stuff
like that, and we match these API operations to the underlying plugins, so we have a hardware specific
plugin that could implement torch run as the actual torch run. We have a hardware-based plugin
that implements image inference using the Jet Show inference framework or using a torch
implementation to do image inference and all that stuff. What is really interesting about this
framework is that we can pack the actual operation into a small number of bytes and forward it
to another instance of every Excel application that has access to an accelerator, so we can
do that completely remotely. This is what we actually leverage here, both from ESP 32 devices
and from Marathbury Pi that they don't have GPUs and we can forward whatever we want to run
on GPU on a Jet Show, on a GPU in a cloud server and get the result back and all that with the
same API, and that's what we consider as important that we don't care where the actual
inference runs, as long as we get our SLA, as long as we get low latency where we want it or
more accuracy where we want it. A bit of information how this RPC thing works,
so there's the actual VxLAPI on top, and when we specify that this operation is going to run
using the RPC plugin, we forward that using GRPC, we'll see more information about that in
the next slide, to another VxL application, we call it the VxL agent, and this this application
essentially runs on the device with the actual hardware capabilities. Now our initial implementation
with VxL for the transport layer was with TTRPC Rust, which is a modified version of GRPC,
to be able to do things efficiently, however, in the special case of IoT devices, this was still a bit heavy,
so what we did is that we got TURP to make sure that we can run light with remote operations
from the IoT devices to enable a node. What we tested, so we're going to show you some numbers,
and Anastasia is going to elaborate on how we took them, and what we did, we used these devices
that I mentioned earlier, so we need speed 32, Raspberry Pi, Edge Edge and the Cloud Server,
and we offloaded from the IoT device, from the Raspberry Pi and from the Edge and to the Cloud Server,
and we measured, we saw the accuracy versus latency rate of, as well as the accuracy not only
in terms of the model itself, but in terms of how many images we missed. Anastasia is going to
elaborate more in that. Hello everyone, so for each model, we captured some metrics, we evaluated
the 15 images from a free Modnet test set, we, of course, record the inference latency and
20 latency, the top one and top three agreement with the high accuracy model, the model we took
for reference model is the mobile net V3 large on an X86, and we also took a efficiency,
which I'll talk about later, as a metric, and the confidence deviation, which is the difference
between the predicted class softmax produced by the reference model and the quantized model.
So here we see the key results that we got on the local versus offloaded implementation,
note that we also tried pruning and the head of time, compilation, which produces a shared
object and not the graph, which is better. We actually saw that the latency drops at approximately
47 milliseconds for our best offloaded implementation, which was the mobile net V3 IoT model.
Yes, one thing that we learned along the way is that the post training
quantization that we use can be very unstable for some architecture architectures. Mobile net V3
is a good example, it actually uses this squeeze and excitation blocks, which take a very small signal
and use it to scale whole feature maps, and when that signal is quantized, numerical errors
are amplified across the network. This made worse with the H, we say activations that have very
narrow ranges and when we quantize their flattened. So in the figure on the right, many local
microcontroller runs either miss the correct class entirely, or predict the wrong one.
So this color plot actually seems a bit complex due to the many different configurations
we tried, but to make it more intuitive, the model runs the light down, right, are the slow and
inaccurate. Then we have the slow and accurate on the upright, and we finally see both fast and
accurate model runs on the uplift. The plot is for mobile net V3, which had a high quantization
error as I said, and we actually tried to dispatch some layers to 16 bit the integer, but we still
had very accurate results. If anyone wants to recommend something for post training, in this model,
please feel free to contribute. Now something that we wanted to show as a metric we derived,
and think it's very important for edge devices, is the correct prediction per minute second.
We want this device to be as responsive as we can, but we don't want them to output garbage.
So for the ESP of loading, we're getting approximately two times improvement,
while for the raspberries and zetion, we get three to five times improvement.
So this is also a graph that shows a efficiency, of course, the higher the the better, and the
bar blood, the here proofs that remote of loading is the winner.
So what we learned, we learned that local email on microcontroll is fragile. We have severe
memory constraints. Quantization can break semantic classic classes, even with mixed precision,
and of course, we also show that network overhead is outweight by latency and accuracy.
Thank you. Thanks so much. So, of course, the the whole framework is open source.
We believe it it makes sense because we can we can actually do real deployments using it.
We can scale the actual inference to many nodes. We can reuse hardware in that sense that we
can repurpose the actual device to a larger model or a smaller model or a completely different
application, and we don't using this VXL framework, we don't we don't have to to rewrite the
application. So using the same API, inference can unlocally or remotely, completely in a seamless way.
This work we have we have sent it to a peer reviewed scientific conference, and we're going
to present it in Florence this this May. So if you want one information about how we got the
numbers, what we did, we're going to share the link in the slides that we will upload to the actual
paper. Thanks very much for for listening. If we still have time, I think we can if we have a couple
of minutes I can show where demo of this running.
Okay, so this is the the actual device.
Yeah, it's okay.
I'm going to show it regardless. So this is this is the local inference, and these are the
seconds, this is the classification ID, the confidence score, this is the local on the ESP.
This is the local on the ESP, and I do a config command to say that I want this offloaded to this
inference server, which are on it on the background with this model. So I do that, and
inference is like 50 milliseconds, 70 milliseconds, I'm going to tell you.
It's a little bit easier. It is in the jets of alone, good enough for that, or it's the latest,
or the overhead. Jet jam is great for that, but Jet jam, but in the ESP, it was like that.
It consumes like 100 of the power.
Yes, but I don't know, but I suppose the jet's anyways.
Correct, yes, yes, yes.
But does it use less power in that way?
For the actual ESP, yes, for the jet, so imagine if you have an oboken thing and not a
speed, it's a logistics space environment, you have two directions, and a lot of these things moving around.
So you don't have to put it down to the bottom.