It's currently estimated that it takes about 50 terawatt hours of electricity to run the world's AI data centers with estimates that this is going to go up to even closer with estimates that this is going to go up to
400 terawatt hours by 2030. As well as the cost of running these data centers we have the cost of getting data to the data centers.
It also often said that it takes about 0.03 kilowatt hours of electricity to transfer just one gigabyte of data and if you're transferring you have a cost in time as well.
For many applications these trade-offs are just not acceptable and a great deal of what we are talking about today in the AI diagram is how you own your own AI and how you can do things locally yourselves.
While there's a lot of very good tooling and infrastructure to do things locally and to do things yourselves for some particular use cases the infrastructure sort of nascent and underdeveloped.
And one particular use case that is of interest to us is the microcontroller class of processes.
Where if you if you want to run things on a sort of microcontroller which doesn't necessarily have an operating system a file system.
It is possible at the moment but it is certainly difficult most especially if you're trying with your personal project or with your business to do something interesting and novel.
So as we've been introduced I'm Dr. William Jones and I'm with my colleagues James Lattery and Pietro Ferroa and our goal today with him because and by the way.
And our goal today is to show you that while the tooling for this microcontroller class of processes is nascent and underdeveloped it is very possible to bring up any AI project that you're doing the personal or business on.
With a good AI modeling framework using only three and open source tooling. In our case we will be talking through a project we have done with a novel with a high processor with a custom accelerator bringing up the executive torch framework.
And Pietroa is going to start our talk about that.
Okay, say can you guys hear me well.
Okay, so what are Pietroa tuners at the torch?
So our high level AI inference falls a predictable path as most people may already know.
Monera work is represented as a graph.
So here we have a graph evaluator which traverses the graph performing intensive assignment and operation evaluation which means allocating memory and doing the maths.
And then the departure sends this task through the best hardware, the CPU or a specialized accelerator.
Crucially before, as a torch, runs the program.
We can apply graph level transformations to simplify or fuse the graph, making the model linear before even reaches the chip.
So why do we need a new tool?
So stand a pie torch is massive.
Is a Python based framework built for both training and inference.
Which means they should have it for embedded systems.
Executive torch however, it's like way sibling handling inference only.
And our goal is to take a model trained in the flexible pie torch environment and hard on it to run on tiny resource constraint devices.
Executive torch follows a build and then customized approach.
So the build phase is handled by the out of time compiler.
In your desktop, it does all the have a lifting, simplifying the logic and stripping away Python so the chip doesn't have to.
And then we have the cost of my face, which is bought lives in the hardware.
And to bridge the gap between the two, we use a backhand delegate, which during the AOT phase, the head of time phase.
We tell us that torch, don't run as maps on the CPU, package them from a specialized accelerator, allowing us to plug the hardwares unique bring into this torch body.
So the build pipeline, so the journey from research production follows a linear path.
We have the part which model, which is the starting point, is the full, full of Python overhead.
Then we have the AOT export, which compress the strips all the Python and serialized the logic into a PTF file.
The PTF file is the universal language for a Zacot torch, so it contains the graph, the weights and instructions.
Then we have the runtime, which is the lightweight engine, on the device that reads the PTF file, allocates the memory and executes.
And the result of all of this is fast inference and devices that don't even have an operating system.
We have some constraints.
We applied all of this to a risk five base platform in a bare mat to environment, which means the radical strings are very absolute.
So memory is a luxury, we work with megabytes, non gigabytes, and by using the AOT phase to pre-calculate memory of sets, we save the chip from having to figure out at runtime.
We have few cores, few accelerators, and most importantly, we don't have an OS.
So there's no Linux, no file system, and no dynamic linker.
So yeah, the Zacot torch runtime is statistically linked directly into a binary.
And because it's modular and dependency-free, we can run the PTF model directly on the metal.
And turning a specialized chip into the dedicated AI engine, which gives us a lot of sorts of minor changes.
When it comes to customizing the performance, we have two main methods, we have dropping replacement.
So if you have a hand-shoon kernel for a standard operation, for example, a convolution, you can simply swap the default version for the optimized one at build time.
And we also have more advanced optimizations, for example, graph level optimizations that can fuse layers together.
And by the end of this pipeline, we move from a generic model to a highly tuned system where software and hardware act as a single cohesive unit.
And now I'm going to pass them to my colleague Shane. We'll go each more detail about the observations we have worked on.
Thanks, Pietro.
So yeah, we've been working on as Pietro said, a risk-by-processor with a custom NPU.
And we've employed a number of optimization strategies in order to get the model working as fast as possible.
So the baseline of executor's will essentially, it will take the, it has the operators all built in from Pietro's.
So anything you bring down from Pietro's to executor's, it has an implementation from a lot of the main operators.
But obviously, you want to optimise this.
So the baseline here is where a single core CPU will, it'll take the sensors and work on them.
So the first optimization strategy we employed was tiling.
So tiling allows for breaking up the problem and essentially making it so that you can do things concurrently.
And this has been pretty instrumental in terms of getting the operators to work as fast as possible.
The next optimization strategy we worked on is multi-treading.
And this again, then allows you to take the tiles you've just done.
And split this across multiple cores on your CPU or on your cluster to allow for, again, working as fast as possible with the resources you have.
So a lot of the tensors in Pietro's and executor's are all float 32, which are 32 bits.
And these are expensive operators to work with, especially on an embedded platform.
So the other optimization strategy we have worked on is quantization.
And this process is essentially where you take the, do you take the 32 bit float and transform them down into 8 bit integers for more optimal and faster performance.
So the first optimization is in regards to memory.
This is where we have been using the multiple cores with the, specifically L1 and L2 memory.
So, through the use of a DMA, we can break the down into tiles and bring these into L1 memory for faster performance.
And yeah, to work on things quicker.
So this then allows us to, the tensors generally tend to live an L2 memory because this is where you have the most memory.
As your L1 tends to be pretty small.
And with the use of the DMA, you're allowed to then break these into the tiles and use the tiling algorithms that we have been working on.
So, pretty, for the likes of an image convolution, this lets you pre fetch that to break this down into sub-tiles.
So, as you can see, this is a pretty straightforward algorithm where, as you are computing on tile n, you're able to then use a different core to load tile n plus 1.
So, in that sense, you are essentially double buffering.
And with the use of the DMA, this lets you quickly swap between memory as needed.
So, as you're computing tile n, you're way fewer results. You can store your, um, store your results into the output tensors.
And then you can compute on your next buffer straight away. And this essentially loops around.
You can load the next tile as you're waiting for the computation of the previous one.
The next authorization that we employed is quantization.
And as I previously mentioned, this allows you to take the 32-bit weights and biases and other tensors that you may have.
And break them down into 8-bit integers.
This is a pretty core optimization that is very necessary for these embedded platforms.
Because essentially the, um, working with 8-bit integers, it just tends to be much faster than using floats.
So, you can calculate the scaling parameters needed and a offset, which it will let you take the 32-bit floats and roughly map them to an 8-bit integer.
And this then can be used post-training to put into your operator and get out results much faster.
So, based on these optimizations, this is the benefits we've seen so far.
This is ongoing progress, so there's other optimizations that you can make.
But with the current optimizations I've already outlined, as you can see, there's a 3.5-ish performance benefit with a convolution,
sure the use of, um, tiling and acceleration.
And with softmax, there is a, um, at least a 2 times benefit, but we've seen higher benefits with bigger tensorsizes, um, where the green is the baseline, which is the operators,
that executive torch implements, and the blue is our versions of the operators.
So, we'd like to thank the team at Mosaic SCC, who supported our work with this processor.
And yes, so this is being, uh, this allows us to take those big PyTorch models that are really energy, um, not efficient and make them energy efficient on small and better devices.
Thank you.
And I think we have five minutes for questions.
Yes, if anyone.
Now, uh, so we've great quantization comes great loss of accuracy, so I wonder just what can you run up yet.
We quantizing at eight bits, we have similar issues, not being able to run, uh, sound related models, so I wonder what are your numbers on this case.
We're still developing this fairly early on our numbers, our numbers look good, but we can't conclusively say that we can't conclusively say that we've solved the problems that everybody else is struggling with yet.
Sorry, I don't have anything more concrete in that.
Uh, something that just, while you, while you find the next, there's a question, um, there is one.
While we're doing this, um, this convolution graph here, where we're comparing data, this is quantized convolution versus quantized convolution.
So, if we were doing quantized versus on quantized, it would be expected that we'll solve a full full increase.
Best, because we're going home.
Quantized versus quantized, it's very, very close to the six.
Uh, so I'm wondering, we'll catch.
So, uh, have you looked at scaling this up to larger rest five systems in chess?
We've explored, we wouldn't use the executive coach framework like this for larger rest five systems, because it's specifically aimed for doing things at the framework control over class processes.
But we have before, but secondly, I'm doing things.
At a much larger scale.
Um, and we, we would use a similar approach.
We would use normal Python to do it, I suppose, is there, is there, is there where we'd got to do with that?
I think the system we looked at when we were looking at doing this, a larger scale had something like a thousand cause of all.
So, yeah.
Um, so, so the question was, um, when you're using TF light.
So, the question was, um, when you're using TF light.
So, the question is, um, with TensorFlow, you can convert all the operators into TensorFlow light.
When using TensorFlow light, um, and this is the same with PyTorch to executorch.
Um, for the most part, yes, it's, um, very, um, capable of, um, as much operators as you need really.
There are, our limitations, specifically, were operators that you'd use during training rather than inference, but, um, I believe TF light tends to have those exact same issues.
Um, and it's, it's still early days where exactly Torch, I believe it's only a few years old.
So, um, it's, as the PyTorch develops and as executorch is developing, um, you're getting more and more optimizations, more and more operators and, yeah.
Something I'd say on that as well is that the, the way executorch does support for this is it maps down the very, very large set of all PyTorch operators down into a sort of smaller core set.
And it's, as Shane said, it's still early stages. It's pretty good at reducing things.
Mapping things from this big set of operators down to the smaller set and in theory, everything should be supported.
Our experience is, it's a little rough around the edges because it's still being actively developed.