In video with its GPUs and its CUDA software ecosystem has between 70 and 95% of the world AI
chip market.
If AI is going to thrive, we need a wider ecosystem of both hardware and software.
And the question I give to you today is how?
I'm Jeremy Bennett.
Today we're going to give you a step on the answer using open source.
I hope that you've come away from this with an understanding of how you can have a new chip
design for AI and you can bring up the software ecosystem you need so that you can run
all your favorite AI systems.
I've joined by my colleague William Jones, who will take you through the practical real world
of this.
I'm going to give you an overview to start.
We're focusing on neural networks and I am fully aware that neural networks is not
the whole of machine learning and AI, but it's the big one.
The way all systems work is the neural networks represented as a graph and the software,
whether it's PyTorch, TensorFlow or whatever you're using, may do a bit of graph level
transformation to make the graph a bit more efficient, but fundamentally sits there
walking over that graph, looking at the nodes and the nodes tell it what the arguments
are, the sensors, the glorified matrices, which are the data, and what the operation
to perform is, whether it's an ad or a matrix multiplication or a convolution.
They sit in a world that is a host and accelerator-based system.
The host may be an x86 today, the accelerator almost certainly, unfortunately, is Nvidia,
not because Nvidia is bad because they've got 90% of the market.
The dispatcher sits in there and works out which of those to run on.
Do I run on the host?
Do I run on the accelerator?
Or do I run across both of them?
And we're focused, particularly, today, on the dispatcher and how it decides and makes
that decision, pushes software onto the accelerator.
And that works for microcontrollers and standard software, PyTorch, TensorFlow, you've
got executor, you've got LightRT, and it's probably handling single nodes off to be accelerated
by a special accelerator, and it's probably only doing that with some operations.
But it goes right up to huge co-processors.
And we've worked on both these scenarios.
We work on the executors.
We work on one of these, which is a risk-five chip, which has more than 1,000 cores on
it.
And in this case, the dispatcher's got to work out how to get across all those cores.
It's probably trying to handle multiple operations at one time, postponing delivery of
results, and so forth.
So Williams, now, going to talk you through a real open source example, William is my head
of AI.
He is also responsible for the UK's national guidelines on best practice in AI for the
electronic systems industry.
William, we're going to now be a brief pause while we change over the microphone.
I'm going to find which point I have, but I think you can.
Cool.
It's the earlier, OK.
Yes, those the project I want to talk you through is a student project that we've
been doing for the last four or five months, I think.
So every year, we do a student project with Southampton University in the UK, and we host
a set of them as the students to do some sort of relevant industrial project to help
them along their degrees.
And this year, we had six students for ten weeks, and we asked them to go away and integrate
well, go through the process and create a demonstrated project of integrating a new accelerated
into the PyTorch framework.
And in particular, by the way, we were hoping had pictures of the students up here, but we
didn't get that sorted in time, so we just have their names.
And there's a lot of credit because this was an incredibly good project.
So yeah, so particularly what we wanted to do with this project is we asked the students
to bring up a risk-vive core as the accelerator, MNFBGA of their choice.
We asked them to go into PyTorch and create a new device in PyTorch and modify the dispatched
to dispatch to this device.
But we also, most importantly, asked them to go away and create a tool chain between
the two that would let this dispatched to any hardware.
And sort of work in a hardware-agnostic way.
And this is in many ways the tricky bits.
And in terms of the slides that Jeremy puts up earlier, we're sort of looking at this
bottom bit here.
The students have to go in.
They have to modify the dispatched to in PyTorch and create a tool chain to connect it
to a, well, it's an accelerator, but it's obviously not much of an accelerator, because
it's just a risk-vive core, because the ultimate goal of this is just a demonstrated.
And the sort of end goal of this for the students or their stretch goal at the end was
to try and get resonates in working on this risk-vive core as an accelerator.
Now, we asked the students to do this with the one ABI ecosystem and the one ABI construction
kit.
So if you haven't met this, this is a construction kit that's largely based on the efforts
of sickle and opens the L, that influence a model of heterogeneous computing.
And the way that this would work and the way that we were expecting was units to solve
this project.
It did something a little bit different, is that they would go to PyTorch.
They would modify the dispatched to this new piece of hardware for the operations that
they cared about.
The reason they're 18 ones, they would provide sickle implementations.
I think this was going to be add, match norm, and the convolution 2D.
And then with the sickle compiler, which is the DC++ compiler in the one ABI construction
kit, they would produce a multi-architecture.
The binary, they'd have the code on the host that is driving what is happening and the code
on the accelerator, which is obviously, like, actually what is doing things.
And this multi-architecture binary would call out to the OpenCL API, which would implement
this heterogeneous computing.
And in the one ABI construction kit, there is an extremely generic and simple implementation
of this OpenCL API, which calls out to a really basic low-level hardware abstraction layer.
Which, just sort of, defines, like, I think, six functions, which sort of defined writing
to the device, reading from the device, and things like this.
And the scope of the student project was essentially, they'd have to go in, they'd have
to do a little bit of work on the producing the multi-architecture binary, and probably
a bit more work on the hardware abstraction layer.
And then they'd be able to go through this, demonstrate it could all work, stitch everything
together, be a nice project.
And ultimately, the goal of this, if the students had time, or if we were doing this
for a real project, because this is how, you know, this is in the family way for us to
approach a project like this.
Sickle is a very mature tool chain for doing this type of thing.
You start with this generic, simple implementation of OpenCL, and you develop something
more target specific and rich over time.
Now our students actually ended up doing something a little bit different to this, about
four weeks into the project, so the students came to us.
And I think they felt they were running out of time a little bit.
It's a tough thing to get done in 60-engineer weeks when you're, when you're still
a young student, so you haven't done this type of thing before.
And they basically said, look, we think we can do this in an even quicker and simpler way,
which gives us even better chance of success, and we were very proud of them for doing this.
It's not easy to have conversations like this with your sort of friendly industrial customers
at the best of times.
And they did a really good job of not just saying, not just explaining what they needed,
but what they're supposed to do to the problem.
And what the students ended up doing is they ended up sort of writing a micro-harder
abstraction layer that sort of sidesteped quite a lot of what we'd originally expected
them to have to do.
So the flow that I described before, how we were expecting them to solve the problem,
was that there would be implementations of stuff to happen in sickle.
The DC++ compiler would take the OpenCL library and compile this now into this multi-architecture
binary, where things would happen.
And this multi-architecture binary can take into these calls that the OpenCL API would
drive how things happened.
But what the students did is they just wrote a sort of, instead of going through this whole process
and implementing this whole hardware abstraction layer and way of interfacing with the hardware
through the ecosystem, they just wrote a little interposet in the OpenCL library that
captured the sort of two calls that they were actually interested in dealing with, which
are the set arguments for an operation and doing operation calls.
And they just captured those and then went away, dispatched to the hardware on their own
and got the results back.
And this was like, it was a really good solution to what we'd asked them to do.
And in many ways it's sort of, in many ways it was better than what we'd asked them to do
or it was a better demonstrated because it's even more minimal than what we'd originally
expected and this, and that is what this was supposed to be, a sort of minimum viable
administrator.
In a bit more detail, because their solution was interesting sort of, in more detail
than the students actually ended up using TCP to implement the communication with their
FPGA accelerator.
And it was quite a nice little system.
And the students ended up using this Zilings Zink Board as their FPGA host of the RIS5
Core.
And this actually is one of these FPGA boards that comes with a little processing system
on it, had a few arm cores and it had a few peripherals and it meant that the students were
able to sort of just, like, almost overnight set up a TCP communication with it and use
that to offload things to the core.
Now, obviously, this isn't a realistic thing that you do with sort of a real trip.
You could probably try and use PCIe or something, but it was a good demonstrated.
It meant that the flow for the students' work was that they have these sickle implementations
of, as net operations, their OpenCL interposer would capture the arguments and the data
from these.
It would send through TCP to the FPGA board, FPGA boards processing system.
We'll put this into shared memory.
The shared memory would be operated on by the S5 Core.
There was output back into shared memory and everything sent all the way back.
And yeah, it's a really nice elegant solution that the students came up with for sort of in this problem.
And it actually, yeah, it sidesteps one of the issues with sort of doing things the way we've
originally proposed a few slides back.
In this way of doing things, the building up things via a sort of simple,
very basic hardware abstraction layer that's provided in the construction kit,
you actually end up compiling your whole program for your GUSC Live Core down,
sending that to the ExcelO extra executing it there with the solution of the students,
but both of you don't have to do that.
You can just send the data, which is at my improvement.
So in terms of the overall success of the project, to sort of sum up,
we set the students with the task of creating a minimum viable product of basically integrating
a new piece of hardware and a tool chain with an accelerator framework, which was quite
arch, and we asked them to see if they could get resident 18 working.
So obviously, they achieved what we'd asked them to do in getting a demo to work.
That was fantastic.
All of this is available for you at what?
If not, now we'll be shortly available for your open source after their work has been marked by their examiner.
So they've achieved that.
They very nearly got resident 18 working as well.
We initially wanted to get the sort of three dominant operations in resident 18 working,
which were at Anne Bach's norm and two de-convolution.
The students got Anne and Bach's norm done very well and didn't quite finish off convolution to the by
basically by the time they had to stop working right up.
They got a very long way, a very long way, a very long way along finishing that.
So that was very good.
And I spoke earlier about making a hardware agnostic solution as well,
so that they'd be able to basically substitute out any piece of hardware for any other piece of hardware
that was supported by the single content that we designed.
And that worked as well.
The students initially started working with, I think, the Zylinks micro-place 5 core for their FPGA,
a risk 5 core that Zylinks provide that does actually work particularly well with FPGA.
And they were able to demonstrate that by sort of just substituting out for a new piece of hardware.
And swapping out for even something with a completely different instruction set.
I think there's still basically worked out the box, and yeah, it was very good.
And the students deserve a lot of praise for the work they've done.
Anyway, I think I'm now handing back to Jeremy to finish off.
So Williams just described to you that we're not talking about theory.
This is practical.
We're able to talk about this one, it was a student project.
We do this commercially for real customers as well.
And that work is all freely available for students to use as a starting project.
And I think it says something that six students in ten weeks,
who had no previous exposure to AI software and infrastructure were able to bring that up successfully.
So how do you do it?
And let's look at what we mean by AI.
And there is a pyramid problem here.
We've all played with chatGPT millions of people having the world or these days deep seek.
There's a lot of professional engineers.
This is actually the big revolution in AI.
It's people using standard models to make businesses run better.
Get rid of the drudgery.
Helping the lawyers find all their cases automatically.
Helping people buying homes find all the legal documents they need automatically.
Automating all the paperwork you need if you're in England and want to take a
Laurie load of goods into urine.
That's where the big revolution is.
Actually, the number of people developing models much smaller people developing
resin that 18 and all the new models, that's a smaller.
And sitting on the top are people like us who actually develop the AI tools.
And it's not just people.
Some of that development is itself done by AI's.
So how do you get involved?
Executive torch is part of pie torch.
Light RT.
What used to be called TensorFlow Light for Micros is part of TensorFlow.
And they've got their official tutorials.
These slides will all be on the file stem site.
So you'll get the links.
Cycle and OpenCL is already in pie torch 2.4.
So you don't have to write the implementations.
You've got implementations already available.
The one API construction kit, it's freely available on code plays GitHub.
That work has done in Southampton and other work we're doing elsewhere.
We're turning into some more of our how-to.
And I know many of you will have used our how-to before.
They're coming this year.
And ultimately it's what we do for our day job.
So if you'd like to do more and you want some help,
come and ask us.
We'll be here, William and I here all day.
And we're at the AI Plummers conference tomorrow as well.
So thank you all very much.
I asked a question that beginning.
How do we get AI working on new hardware?
And I hope we've given you a bit of insight
into how that can be done.
And I think we have a minute or two for a few questions.
Thank you.