Very much, Mark. Good morning, everyone. I'm sorry for the remote attendees. I'm not sure.
You can see my slides. There's mainly images. And I will, by the way, put the slides, also
in GitHub, just after the talk. I mean, as soon as I will have a working internet connection.
So I am Sylvan, folks for Golf Kiroradio. And my presentation this morning is a kind of follow-up
of something I presented last year, but I will come back with this soon. And it's called
using AI hardware accelerators for real-time DSP and embedded devices. I was looking for
even longer name, but I didn't. Okay. So more seriously, the talk is organized as this.
I will briefly explain how this side came. Explain, share with you the, something I had
doing this. And show you what is the outcome. And what can we do with this? What can
be done with this? Sorry. So last year, in the previous episode, I presented a GPU based. So
NVIDIA only, I would say, solution to process a very wide bandwidth and do multiple digital
down converters in CUDA. Just to be very honest with you, this code was extracted from some
code we using my company. And it's working. So the idea was to use, so basically the idea
is as follows, you have a very wide stream coming from a radio. And you want to extract from
this radio slices sub-bans. You have many ways to do this. And the algorithm I proposed was to
use the GPU. So you basically push all the samples to the GPU. The kernels are running inside
and just extracts the sub-bans. That works. And one of the points was why should, what could be
done or what could be used as another solution to do the same? And what if we would use this
super fancy super nice AI chips that they are promoting a lot. You see now that if you buy a new
PC, you have an AI PC. What is the AI PC? It's a PC plus some chips inside of pure ships.
So my point was, can we use this specific devices for signal processing? And the show
times are yes, but it's not so easy. It's not difficult. There's tricks as usual. So the idea here
is to share with you the tricks I used. So the motivation is to be able to use some
either specific chips like the Google Coral, you see on the right, which are just inference
when the consist tensile processing units will come back to this in a minute. Or devices like the
Rochip, some of the Rochip CPUs will have multiple arm machines and have a small units dedicated
to AI. So these are very cheap. I mean you see the prices, the Google Coral stuff is 30 euros,
depending on how you buy it. And the Rochips you have these are complete boards. Means
CPU plus RAM plus IOS, everything you just have to connect your screen and
Ethernet and it's a full computer. The RK 3588, you can see this as a Rochip as a Raspberry Pi
computer module 5 or 4. That's basically the same. But there is something specific in the chip.
This is this AI unit. You also have these other things. The Accelerar for example, they claim
very big figures. I will show that. And on the right, something that we're testing in my company,
which is this 10 store-ins machine, which is a monster. And the good thing with this is that
it uses only the Rochips 5 hardware. It means that this chip, the
sorry, I can't read the name because there's different ones. The chip is using a network of
REST 5 units. And you code this in C, full C. I mean, as long as you can code in C, you can do
something else. I would say. Which is quite different from the artificial intelligence accelerators.
And I will come to this now. So the promise they have is, sorry, I will increase the OK.
So the promise, if you look at the red hash sheet from the Rochip RK 3588, it claims 6 tops.
What is a top? There are operation per second. We are not talking about flops, but tops.
There is a small difference. It's not a difference between the two. But basically, it means a lot of
a lot of math in real time or fast, at least. What's interesting is that they claim triple
court, I say, sorry, excuse me, I say they claim. I'm not saying that it's wrong. I'm saying
that's what is written in the commercial sheet. So you have, basically, what we can understand
from the second line, include triple MPU core and triple coba, so I mean that you have kind of
polymism. We can understand from this that we can run multiple streams simultaneously.
And just for those who are attending my speech last year, the fact that you can use multiple streams,
simultaneously is very important. If you want to use multi-threaded or multi-task DSP systems.
The thing also is that if you look at this last line, support deep learning frameworks,
TensorFlow, Cafe, TensorFlow Lite, PyTorch, when it makes an end and really an x, et cetera.
That looks very open. Thank you.
So the promise is that if you look at the models, these things are running,
you don't think there's something? No, no.
If you look at the performances, you will see that these things are able to run some
AI tasks, classification, object detection. So this is the well-known yellow, you look only once.
This algorithm where you feed in images and it's able to look in image and say you have a
catch, you have a dog and so on. That's what these things are made for. And in this column,
the third, sorry, the, if I can move the mouse on the good screen, yes, here, this column,
the last column shows you the supported platform, that's the list of Roxip CPUs that are doing this.
And if you look on the table here, you see some figures like this is able, for example,
for the yellow, so that is the one that is doing object detection. This is able to run at
something like 33 frames per second. That moves amazing. First of all, it's flopping, it's 6.16
or 8. Means that we're processing bytes. I mean, integrers or small or floating point numbers
with limited resolution. This has an impact. So first of all, you need to be aware that
the performances that are claimed are usually for int 8 or FP4. Of course. Of course.
So if everything here is said to be ON and ONX and you look at the files here, you can download,
they say you have a lot of models, you can download and this model are ON and ONX files. So what is
an ONX model? This is an open source framework and I copied this image from their websites.
So you have on the left those tools, which generates where you design the model. So you have a
pie torch, TensorFlow and blah, blah, and so on. On the right, on the right hand side, you have the
platform. So you see CPU, GPU, FPGA, blah, blah, blah, and in the middle, the ON and ONX one time is that
magic group that somehow makes an obstruction of the hardware below. So far, I haven't been speaking
at all about radio, but stay tuned, it's coming fast. So in fact, the ON and X is the ONX is the
coldest and execution provider. So you have your model. So if the file that describes the
process you want to perform. So you can see this as the microcode that we run inside the processor.
On the left, you have your input data here, on the right, the output. So for example, let's say that
you input, you inject in this, the samples coming from the radio and at the right hand, for example,
you want to have the audio. And you want this chip to do everything inside without choosing the main
CPU. And the idea is that this execution provider is able to read this file and configurably
underlying hardware for you. That looks very interesting, doesn't it? The good thing is that this works
for GPU chips. So the former, I mean, what I'm used to use, that is the NVIDIA GPUs
should be working, and that should work also for these nice chips.
A very brief introduction to ONX. You describe the execution flow as a graph, where you have
more than one input, multiple inputs, multiple outputs, and you describe the execution flow like a graph
and each node in the graph is an operation. So for example, this graph, which is from the
documentation, is doing a match, multiply and add operation. And that's a very simple ONX file that
just explained this. They don't speak about numbers or vectors, they talk about tensors.
And you will see that sometimes this is a mess. The other good thing with ONX is that it's
quite widely documented. There's a lot of versions, 11, 12, and so on, but basically you will find
online, a lot of documentation, explaining what is, what each operation is doing, and how it works,
and what is the inputs you need to fit in, and what is the, without the outputs you would get.
So just to make a very brief summary of what is ONX, it's designed for AI, means that the
operations you will find are basically those who are used in deep learning, and mainly in image
processing. And of course, those chips, for example, the raw chip ones, are usually used in remote
IP, IP cameras that are doing motion detection, object detection. So the idea is that these chips
are used in these cameras that take image, real time, check what's going on, and if there is
a movement, then it can send something on the SMS connection. So these chips are optimized for
that kind of input-fix, so it's matrix of pixels with either black and white, which usually
three colors. So the dimensions of the inputs and the data structure is organized to process images.
And in our case, we're not processing images, we're processing streams of complex signals,
so there will be some tricks to make this work. So if that works, what would we love to have?
So the idea would be to be able to run some of these, some of our DSP blocks,
inside these chips, and for example, that could be audio processing. If you've seen recently
that's, or that's it here as a plug-in system, and they're using, they're not using Linux,
they're using another one, but basically that's the same ID, you have a model we can load,
and the model is offloaded to the local acceleration if you want. It could be, for example,
something an extension to something like the radio companion where you have this graph,
and we can imagine that instead of loading Python code or C++ code, that would generate
the files that would be executed by the next models. Why not? That works interesting.
But, but, there are some tricks. If you look in details at the documentation in rock chip,
for example, it said RKNN, which is rock chip and N. And if you look even in more details,
there is only a subset of instructions. Means that most of the instructions that you would
love to see are in fact not implemented. So then the trick comes. The Google Coral uses TensorFlow
light, which is basically the same, that is to say you have the specifications that everybody has
agreed, but the operators are too complex. So let's make it simple and let's trash everything
that is complex and just keep very basic operations. So in fact, the available, the list of available
instructions is limited. If you look at the table here, the which is an extract from the,
sorry, the RKNNN tool kit, you see that most of the functions are not implemented. Apps,
not implemented. Our costiness not implemented, etc., etc., many functions are just not available.
And in fact, you will see that RKNN is basically on its version 11. But just for information
today's version 20 something. So there's a big difference between the tools you can use that are
available on GitHub and what runs in the chip. And this is the RKNN info from their website.
But that's basically the kind of the same kind of, it's quite similar to this step, sorry.
To this one but it's on the raw chip source. So basically, what it says is that you have tools
by torch 1 and X, TensorFlow. There toolkit does the translation to the API that is understood by the chip.
So, my understanding was, and I confirmed that, that as long as you are able to generate an
ONNX file that does the DSP, it works directly on the other, almost.
So what do we need in the end?
We need complex number, okay, because we are processing IQ row samples.
So now let's go to the DSP side of the problem.
Now that we think that the chip can be used, now the question that comes is, how does
do I feed my samples in that stuff?
What do we get?
We get IQ samples, complex numbers, stream of complex numbers from the radio.
So we need to do complex number arithmetic, basic.
We need to be able to do the convolutions, because we need to do one, we want to filter,
so low pass filter, band pass filter, rejection and so on.
We may need some trigonometry, trig functions, like cost, sign, tangents, let's say we
want to demodulate FM, at some point we will need, we will need some trigonometry functions.
Do we have this ready?
No.
Those chips have no clue of what the complex number is, okay?
But so what?
We learn that school what the complex number is, so let's do it, why not?
So let's go to the basics, and that's where the fund starts, because in fact, you realize
that you have to do everything, it's not so difficult, but it took me some hours.
So first of all, how do you represent in the chip the complex numbers?
There's two approaches, either you could say the classical one, which is this one, is IQ
Interleaf, real park imaginary park, real park imaginary.
That's the classical one.
You could say, but these devices were designed to process images with color plans, so
you had one block for the red, one block for the blue, green and one block for the blue.
So the first idea I had was, yeah, let's split the complex number into the real
park is money match, and the imaginary park is the second image.
In some cases, that makes sense, because some operations in the GPU in the processor have
been designed to work like that, but in fact, in most of the cases, it's a bad idea.
It doesn't work.
So I finally tested the two and selected the classical one, that is to say the Interleaf
IQ sample.
As we do, as it's done in, I would say 99% of the existing platform.
So if we want to multiply complex vector A, a series of complex numbers, and with a second
one, point wise multiplication, that is this one.
We want, so we have A, which is, for example, A vector A is the stream of samples you get
from the radio, and D is some, some taps from filters, because you want to filter, you
want to make a low pass filter.
So at some point, you will need to compute for each of them the multiplication point
by turn, that's the point wise multiplication.
So I used colors, and this was for me to debug to be honest.
So you want at the end this multiplication, and that's where the problem comes, is that
the architecture is not designed at all to swap pairs, and it doesn't work well.
So you end with something which looks like this, it's honest, it's not difficult, it's
just that you need to do this step by step.
So the yellow box is gather elements, is one of the instructions provided in the
or in an experiment.
So it means that you take vector one, you take vector two, you have a table that indexes
of indices, that tells you which number you have to take.
That selects Q, IQ, that just swaps real path and imaginary path, one by one.
I see that you want checking, and then you make a multiplication and so on and so on.
And in the end, you end with a nice graph like that, which just does all the multiplication.
So to do this, I used in fact Python code, because there is an API in Python that helps
you to generate this one and x5.
I will tell you where at the end of the talk, but all these files are on my GitHub, you can
take them, of course.
So you generate a graph and at the end, you put vector A, your input, your first one, that
will be the second one, and magic at the end, you have a multiplication.
So it means that one of the issues you may have seen is that you need to know how many
complex numbers you will process.
Because the problem is that if you process images, you need to know the size of the image.
That's exactly the same.
It means that you generate an x5 based on a known number of samples you want to process.
So it means that if you want to generate something for, I don't know, 500 samples.
You have 500 samples.
So if you have 1000, you need to call it twice or generate the second model.
And that's, so I ended with Python files where I have put variables, saying how many
samples I'm processing.
And then it generates all the tables and all the indexes based on this input.
It's quick and dirty code.
I'm sorry for that, but it was just a test, a proof of concept.
So it works.
And then I have made, so the complexity teacher, the IQ oscillator, and just to show
this noise stuff here.
So this generates the local oscillatory signal you would need for the down or up conversions.
And this was just an FFT plot to see the quality of the signal.
But to see if it works first, second, to have an idea of the roundings and issues in
the sine and cost generators of I use in the chip.
And honestly, this depends on the chip.
I suspect that the sine and cost are not so nice.
So we may have some, I have not finished the test to be honest with you.
I have also made the filter, so sorry, I'm going to first.
So the complex multiplication is not very difficult, it's just swapping the stuff and making
them be in a line in the good way, so that's the multiplication work.
That's easy.
The local oscillator is basically very stupid, you generate the phase.
And then you apply cost and sine to the phase and you're happy.
So you have on the left, you generate the phase, then you generate cost and sine.
And then you just reorganize sine and cost to have a real part imaginary part on the
good set.
That's easy.
The filter is a bit more tricky.
In fact, a convolution that is done on an image is a filter.
So basically, you don't have to reinvent the wheel.
The problem is that those stupid stuff are expecting multidimensional tensors.
So they expect multiplane image with structure organized by colors.
So you cannot send a one dimension, at least in raw chip, it doesn't work.
So you have to take a tensor, so you have to add a lot of annotations, so that's the CPU
believes it's an image.
So you have specific blocks, sorry, this here, this reshape stuff that I'll just use to make
the CPU happy.
It's used this, but it works.
The good thing is, so this, this filter only works for real tubs, not for complex
tubs, because it avoids the complex multiplication, well, you'll see an implementation
it's already there.
I also tested the A and the modulation, that works.
I wanted to try the FM, because on VHF and above, we most of the time I FM, sadly I couldn't
find the octangent operation, and I didn't want to spend, I had no time to implement
it.
By the means, so it's not working yet.
So is this a mature tool, well, I wanted to share with you some of the cryptic messages
I had when I tested that, so you can't wait, basically it's very strange, my error messages
like, this is an invalid model, tensor double of type blah blah, and you see the error messages
like, hey, what the fuck is it?
It's sometimes crazy, I mean, reading of the messages you have from the API, and I realized
that there is one very stupid thing is that if you use the very last on X platform on your
PC, which is operation set 21 or something, and you target a device that is a version 11,
like the Rochip, then sometimes it just doesn't work.
So you may have to download old packages from, so some references, so I put on my slides
all the stuff I found testing this, there is a very good introduction in Python, the last
link here gives you a lot of hints on how to do that, a kind of conclusion, a kind of,
I think it can, I haven't shared with you any figures at this stage, does this work?
It does, is it efficient, I don't know yet, to be honest, I don't know, but I think it's
interesting, because it may open, it may enable you to use very low cost hardware to do
some intensive processing, I mean, I did this, I used that for GPU, but it's not a low
cost device, while these chips are very cheap, and you can find easily some add-ons, some
PCI Express or M2 modules with these devices, means that, even on the very, for example,
the Rochberry Pie, that could add a significant amount of DSP power, but there is a long
way still ahead to make this work correctly, let's be honest.
So, you can find all the code I made, but these are sections of code yet, I just put what
works, in my GitHub, it's called Onix Radio, I put very quickly and dirty the code I
have made to do this, there is in the GitHub, there is the, you will find the Python
codes that are used to generate these basic functions, complex multiplication, filtering,
demodulation blah blah, so that gives you ideas if you want to go one step further, there
is also a C++ example that loads, the Onix file and generates numbers and calls the
complex multiplication, because I tested that it works, but of course, it's far from finished,
it's just a proof of concept at this stage, so where is this, so that's where you can
find it, sorry, did I, yes, I think that's what the two slides, okay, that's all
faults, there is still a long way ahead, if you have questions, I'll be happy to try not to
answer that, thank you, any question, yes, no, no, no, that doesn't avoid that, so yes, for those
who cannot hear the question, the question is, does this simplify somehow, does this removes
or simplifies the bottleneck we usually see in GPU systems where we have to copy the data
from the main memory to the process unit in GPU case, no, that's exactly the same architecture,
and the problem is if you look at the Rochip architecture, I'm going back to the slide where
it is said, you will see that the available memory for transfer is in quite limited
in fact, it is 30 roughly 400 kilobytes, which is very small, if you compare this to a Nvidia GPU,
where you have let's say 8 gigabytes, you see that it's a huge difference, what it means
that you need, by the way, you will need some thread spalling and pushing data, otherwise
it doesn't work, it's not efficient, but at least what is interesting here is that if
you look at the classical architecture where you have USB stack, pulling samples from a USB device,
you could imagine that the samples you get from the USB device, you just directly push them,
assuming the stuff is fast enough, you have chunks coming from the USB, for example, you
would be able, that's what I'm trying to do, is to directly push the USB, come payload, directly
in the queue to the NPU to be processed, so by the way, you have to do the code, at least
this, but for sure, it does not simplify this issue, that's clear, is it more efficient,
I haven't tested yet, and I don't think so, yes, not all of them, yes, the ONNX architecture
has a 4 operation, and a while, for example, in many models, I spend quite a lot of time
opening existing ONNX files and looking at them while they were implemented, and in many cases
you have the ifs and you have 2 branches, typically when you process the audio, you have cases
which is depending on the sampling rates, so instead of having a stage that resembles, they've
made branches, 8k, 16k, and they swap, and when you have streams like that, they have
while loops, so I think it works, but I'm not sure all the ships do it to be honest,
I don't know if there's a like a break, a return stuff like this, I don't know, I come
side, yes, I think it comes also to the point that you somehow need to loop inside the
ship, I don't know, I don't know, no, I mean, no, sorry, I'm saying something long, it means
that you would have to store somewhere the current phase, lock statues or phase counter,
I tried that, I couldn't find any way to have a local viable, so that's why in the oscillator
I had to push the phase, sorry, I couldn't find it, probably it's possible, sorry, it's
the excuse me, I see time is running, where is it, yes, here, the IQ oscillator I made, I had
to store the phase, the initial phase, so in fact I put the initial phase, because if you
want continuity between runs to make sure that you don't have a phase shift, you need to store
the current phase equivalent, and there's no local viable, so probably there is one, but I couldn't
find it from the documentation on that, I see, yes, sorry, yes, yes, they have also floating
points, but what I meant is that the, yeah, and 32 bits also, but it means that the throughput
is divided, the throughput is given for one byte operations, so if you have six terra operations,
it's in five for one byte, means that if you go to floating point 32, you divide by at least
at least four, yes, you have to specify the type, no, no, no, no, no, no, no, no, no, no, no, no, no,
you specify the type at the input, you say this is floating Fp32 or Fp16, but, not in the rock
chip ones, I haven't seen that, yeah, sorry, I just try to keep the track of the question,
Yeah, you had one question, yeah.
Question, thank you very simple, yeah.
I see that you're using the question that you use.
Do you use that or do you use both?
In the past, I just queued only, because it's faster.
Sorry, there was another question, yeah.
Yeah, I think it was kind of easy for all in five to four months,
like, how many maps can this kind of edit like that?
No, I haven't, that's smart here.
No, no, I have to do it for sure, because otherwise,
I mean, first, the first idea was, does it work?
Yeah, it works.
Now, is it efficient?
I, all of a sudden, I don't know.
One last question, I mean, I'll try to live.
Okay, okay.
Yes?
Do you know how it would work to run by some other next models
or did you get this problem?
Ah, okay, that's a pretty good question, in fact.
The one thing that I didn't test, but it's claim to be working,
is that you can chain, chain models.
So, basically, I made blocks with the idea
that you could chain them.
So, for example, you would have the local procedure,
which is inside, and somehow you will connect the output
of the yellow to the inputs of, so basically,
you have, you imagine a new radio photograph,
you have these blocks, and instead of generating a big model,
you would download in the chip, the blocks,
and also the connections.
It looks like, from the specifications, it might be working.
And in fact, if you look at the files, the ones that provide us,
example, in the model zoo, they give, that's what they're doing.
The problem is that it's extremely difficult
to reverse engineer this, because when you open them with a,
there is a photograph application in,
there is, sorry, I mean, find the name on the last slide, I think.
That's net, netrun.
Netrun is the stuff you can, it's an open source software,
that draws the graph of the operations,
the one I used for the slides.
The problem is, if you look, if you open a yellow model,
you have thousands of nodes, and then reverse engineering
this and understand the tricks, and then it's, no.
So, I think it's doable, because I think that's exactly what they're doing.
They have blocks, they work on small blocks, I mean, that makes sense.
And they had, and they pile, and they pile, they had additional processes on it.
So, very probably, there are options to connect the different blocks,
but I haven't been able yet to spend enough time on this to go on.
That was a part-time project for fun, and,
it's only one act of time.
No, if they claim in the other sheet, but you can have up to three.
And that's what I was using in, in QDA.
That's why I was using QDA, basically.
You have multiple streams of the same time.
So, you should have three models working,
and you could have, as far as the industry,
possibilities to connect blocks, but you have to spend time
operating the Chinese documentation.
And the promise, in this case, you fall to the very cheap,
specific solutions, because if you want something to be working
on other platforms, that's what I tried.
I wanted to stick to the OIN and X implementation,
keep standard, and use the minimum version 11.
Otherwise, I would have problems without a cheap,
and I would not be sure it would be working.
Yeah?
I have the question, when you're saying minimum version,
if you target this R&N or R&D.
So, yeah.
So, run on the run times for simulation.
Yeah, I tested this, if that's exactly how I did.
First, I wrote this on Python, generated on Python,
tested on Python.
So, it means that I could generate outputs.
That's the test cases that you can find in the GitHub,
and tested that the numbers were correct, then debugging.
And then, start to target.
So, then I generated the real one and bypassed Python,
and used the CAPI to generate numbers, memory,
and try, but still with the Linux PC-based emulation.
That works.
And then, tested on the Roch chip.
The only type is only used, or you need it for testing on the Roch chip.
Yes, the problem is, if you don't pay attention,
yeah, that's very good.
Because it means that you write once, run anywhere, almost.
That's the AD.
That's a very cool thing.
The issue is that you have some,
the very problem is that they don't implement all the operations.
Means that if you don't pay attention from the beginning
to which obstacles you use at the end,
your model doesn't work, it crashes.
Because the chip doesn't recognize the operation,
and there is an exception.
So, I learned that from pain, I would say,
but basically trying to understand why it doesn't work.
And then, realizing that I was not using the good version.
And it's, sometimes, on the messages, you see that,
if you read that this, the operator, schema,
and other functionality may change before blah blah blah.
Run time will not down see back what compatibility.
What the, why?
Parent official support for the main,
is only upset 21.
If to understand the scripting message,
you need to read the documentation,
and we are like, ah, that's not the good chip,
that's not the good up code.
For those of you who have already programmed
in a assembly language, that's exactly the same.
It's the same problem.
One last question?
Yes?
In your in seal, the sign of course documentation,
because in the beginning, you said that
the ungradated operation or implemented,
the sign of course, sign of course, sign of course,
and then yourself?
No, no, no, no, no, I tested the, no, no, no, no,
I were using here, I used the existing ones.
And on the raw chip, it should not work.
That's a good point.
I have to use tables, generations.
Yes, yes, yes, yes, no, no, I mean,
what I've used here, sorry, to be clear on your question.
That's this one.
Here, I have used the sign and course functions,
provided by the API, okay?
So it means that in certain versions of chips,
it will not work.
It, what is not implemented in the raw chip,
is the act course and arcs in.
But course and sign are implemented, yes.
So course and sign work.
But the promise, if you want to use octangents
to do FM-double-ish F-s-state emigration,
then you are a problem, because you cannot use
octangents, no octangents, no octangents, no octangents.
So you need to have, you need to implement a look at the table,
or another trick to generate that.
No, but for sign and course, sorry, I'm not on the good one.
For sign and course, they are available in the arcane and end
when an x version 11, which is the very basic one.
So that works.
At least that works.
Okay?
Maybe you've seen that in 2015 or 2016,
I remember well, in this room, there was a presentation
about any installation in the last one in the arcane.
Yes, yes, yes, yes, yes, yes, yes, yes.
The thing is that I was lazy, and I wanted to test
all the things in this, I confused octangents.
So I have to be clever, and I am not.
So I did not know your right.
That's all. For me, unless there is a very last question,
otherwise, I will pack my things.
That's all. Okay? Thank you, folks.
Thank you very much for your time.