All right, my name is Mohammed Galu.
I'm a embedded software consultant out in New York City.
My company's map lab embedded solutions, and this is my talk.
Essentially, you kind of follow up to last your stock and we'll go over what that means in a little bit.
So this is just about me.
Like I said, I'm an embedded software consultant in New York City in America.
Mostly due design work focused on medical devices, scientific instruments,
a light art project that I've been working on recently that involves FPGAs,
and also this past year worked on writing embedded software
implementing Zephyr on a custom ASIC that also had a risk, five core on it.
So obviously my experience in expertise is in Zephyr embedded Linux,
and also some gooey-based applications that are based on cute.
You can sign up for my newsletter.
I host a number of webinars and workshops throughout the year.
This year, going to go into some more training events surrounding embedded Linux,
Yaco, and Zephyr.
So this is a quick rundown of the agenda for today.
We're going to just go over some background.
We're going to briefly go under the hood.
And part of I just want to set the stage for this talk,
because of my vicinity to FPGA engineers,
like I pretend to be an FPGA engineer.
So I like to do things that are with it,
that are at the nexus of FPGA development,
and embedded software engineers,
but embedded software, but what you'll see is that eventually I stumble,
and I need help.
And so part of the motivation behind this talk is to reach out
to hopefully the right community and get help,
and we'll see where that goes.
So under the hood, the plan, what I had envisioned when working on this project,
how it went, it didn't go well, and what are the next steps, right?
So understanding how to debug, like basically,
teaching an embedded software engineer,
like how do you debug FPGA code using open source tools?
So like I said, some background last year,
I gave the precursor to this talk.
Which was essentially using an open source risk five,
32 implementation on an FPGA.
It was based on a really lattice ice 25 PK, like FPGA.
Really inexpensive, so you know,
didn't have to spend too much money getting the board.
I was able to load hello world from Zeffer onto the FPGA,
and you know, the exciting thing was it worked
to a certain degree out of the box.
I used not open source tools.
I used the vendor like lattice tools,
which you know, was fine for the moment.
It was fine for the time.
So the obvious next step,
if you look at last year's presentation,
was I wanted to kind of investigate the CFU,
because the cool thing about the new RV 32 project
is that it has the function.
It's easy for me as not an FPGA engineer
to enable CFU support,
and they have software examples on how to actually exercise it.
They have their example code actually demonstrates
the performance metrics that has a small function
that you can execute to understand what the performance
difference is between a simple kind of custom function
and implementation in hardware,
versus the equivalent implementation in software.
But when I tried to do that,
so here on this thing,
it's basically the summary of the screenshot
that you have is essentially what was the culmination
of last year's presentation.
We get hello world running on new RV 32.
But the problem was when I tried to enable CFU last year,
it didn't fit on the FPGA.
The lattice tools basically complete,
that oh, your FPGA doesn't have enough resources
to be able to support the design that enables CFU.
So that was the motivation behind this,
this talk in this project.
So just some precursor,
again, maybe 90% of you are like,
you know, why are you talking about this?
This is, you know, we already know this,
but you know, for the 10% like me,
it's useful to understand some of these
some of this background material.
So what is the custom function unit in risk five?
Essentially it allows us to offload functionality
to hardware that you can implement outside of the extension
outside of the risk five.
I say, right, that's your own custom implementation,
but there's infrastructure surrounding it
to allow essentially software to talk to the hardware
to be able to execute certain functions in hardware
and you can retrieve those results.
There are useful for operations that are not,
that it's not optimal to implement in hardware.
So if you're concerned about performance,
if you're concerned about latency,
energy consumption and program memory,
it may be useful to investigate
your implementation of your function in hardware
and then leverage the CFU extension
to be able to allow software to call into hardware
to implement to actually exercise that function that you want.
So some examples of why you would want to use
as CFU AI, right?
So now we're seeing a lot of days.
There's a lot of advancements that are enhancement
that you can make in terms of your implementation in hardware
that are relevant to AI.
So you may be beneficial to leverage the CFU extension
to implement some of these function calls
or in hardware and then essentially leverage that extension
to make that call in from the software side.
Crypto obviously communications arithmetic and image processing.
So one of the restrictions that I learned about the CFU is
that it requires CPU dependency.
So it's not something that you can just tell the hardware
okay go and the CPU kind of comes back and says,
okay, fetch me the result.
It's something that the CPU needs to be involved
in terms of producing that result.
So the CFU from mind your standing is that it requires
the CPU to be a part of the operation when you're doing it.
So the CFU actually leverages these custom kind of
op codes in the ISA.
So like I said, the CFU is a specific implementation
of an extension of the ISA.
So that's the CFU extension and that is specific.
This particular extension is specific to the NIORV32 implementation.
And the cool thing like I said is they have a sample implementation
of a block cipher called XDA in the RTL, right,
which is a really simple implementation
or of that's useful for embedded systems.
And so like right here you can see the bottom table
essentially shows how the op codes are mapped
and that you can leverage those to give you a particular kind of
CFU implementation.
So this is basically kind of showing again the mapping
between the actual RTL in the NIORV implementation
and then from the software side how their demo code kind of
implements.
So essentially what they're doing is they're using the
extrinsic inside the compiler to call functions
in the CFU, right.
So essentially they have a particular set of op codes
that are extended using the CFU extension and then in
the software they essentially have these extrinsic
that are defined that essentially just call the the
compiler to basically implement a particular op code and then
the compiler essentially sets up the appropriate
instructions to make calls into the RTL.
And then here's you can see essentially like more
more macros that essentially convert
calling into this CFU particular instruction.
And then we can see here like if you map what's on the
the last set of bits in the bottom macro they map
essentially to like the op codes for a particular kind of CFU
in the table down here.
And then if you write if you actually again like
just just go into macros again you can see that this is
finally the assembler call that actually makes that that
is invoked by essentially setting the appropriate
op codes and then making the calls into the RTL essentially
and then the compiler basically handles the magic of
you know making the calls into the RTL for the
particular implementation that you want.
So just just kind of like an overview of the background
on you know what what the CFU means understanding how
the software essentially ties into the hardware and how
we can kind of leverage those calls.
So now the plan right.
So last time when we try to run the design with the CFU
enabled the tools complain that okay you know your your
FPJ doesn't support doesn't have enough room to
essentially support the functionality or support your
design.
So step one is get an FPJ that does fit this design.
Enable to see a few in the top level board file in the
RTL.
Build in flash design.
You know let's start simple let's just load
Zefer, Blinky or hello world.
Make sure that's working.
And then okay now we need to add what the
macros that you saw in their demo example we need to add
those into Zefer compile those in our demo example
that runs essentially the same XCA blocks
I for and then do the same comparison in the reference
design that they have in their software repo.
And you see if we can do the same thing on Zefer right.
And and essentially like we have a reference
implementation of the non Zefer application right
and we have a reference and we have an implementation of
the Zefer XDA just confirmed that actually the
operations in terms of performance they actually match
the reference design.
Right and so for the first step right this is the the
report that I just decided to use because it was
reference in their repository.
It's an orange crab FPGA.
It's also relatively cheap.
I think it costs like $30 or so.
And you know it was just small.
The problem was that the orange crab that exist today
is actually a much bigger.
It's both good in that right it's good because it's a
bigger FPGA than the board I was using last year.
So I was expecting that would support CFU support.
The CFU extension.
So that shouldn't be a problem.
The problem is again like there's as we're going to see
there's a lot of infrastructure when you're using FPGA tools
that are open source that you need to set up to make sure
that it actually builds for the right thing.
So so it boots right.
And yeah so like I said like the current orange crab
implementation the setup that they're using uses the
smaller FPGA.
So it wasn't going to work obviously in the board that I purchased.
So the first step forget forget even just you know building the
RTL loading it onto the FPGA and just seeing if it works.
I couldn't even just get the tools to install correctly right.
So what I did was you know again software engineer.
Okay let me just see the tools that they're using and just install
it on my local machine.
Nope that one just terribly wrong all sorts of ours.
No idea what's going on.
So I took an easier route.
I said well they have to build the somehow right.
So let me see what they did and just copy whatever they're doing.
So I noticed that in their GitHub actions their workflow.
They're essentially just using a container that they pulled from
somewhere that has a necessary tools has a necessary set up.
So I did the same thing right.
So I just copy to essentially their container their doctor file.
And I stripped it down to essentially whatever I need it.
And you know just essentially ran that as a container on my PC.
So this was exactly the doctor file that I have that I use on my machine
to build these things.
And then on the bottom here you can essentially see the container
and vacation that I have which essentially just runs make
specific to this particular board and then runs a target that essentially just
creates an image which is a bootloader.
Which is the only thing I care about.
So once I had the tools running right did a basic compilation of the
actual design.
So like okay the next step is to actually enable CFU support.
So that was simple enough.
I just looked at the top level design file.
And I just set the flag for CFU to true.
So simple enough.
The other problem is again.
Because the existing design assumes that the orange crab board
supports up the older smaller FPGA.
I needed to modify it to be able to support the newer one right.
And so I learned this the hard way.
I essentially didn't know about this.
I just built it and built fine flashed it like nothing happened right.
I didn't get any bootloader.
I didn't get anything.
Look on the forums they're like yeah.
Yeah because you didn't actually tell the tools to build for an 85 series FPGA.
You still told it to build for 25 series FPGA.
So okay the easiest thing for me to do after kind of digging around the
help files for the tools that I was using.
Let me just replace this 25 with the 85 and then see how it goes.
Ideally the orange crab like initially what I did you know the proper way
of doing things was okay.
Let me just change this variable that they're using in their make files to an 85F instead of
using 25F.
That just was disastrous like that didn't work.
It's all because all sorts of issues so I said okay let me just try the easy way out
and just I know this flag is used for the FP that's used to the that's specified
to the FPGA tools to indicate which FPGA they want to use.
So I'll just change that for now and then leave all the rest for later.
So after I did all those changes got my container running with the tools.
Made the changes of the make files to use the larger FPGA and able CFU support
successfully built the FPGA.
They used the suggest using the FPGA to flash the FPGA essentially just hold down the power button
while you're plugging it in and it's just going to enter program mode.
And I needed to do some do some mechanics with DFU to basically attach the FPGA information
or the hardware information to this DFU file and then once that was up and running
I was able to just get bootloader output from the UR.
So so that's great right at least I confirm that I can get something running on my FPGA
that is based on risk five.
It kind of you know at least something is working right.
So the next step was okay let me just take the upstream Zephyr repo,
latest LTS for.0 and see what happened later certainly it's not LTS.
So built it, create, try to you know just load the application using the bootloader.
They have a script that allows you to like just essentially do the same thing
more scripted in a more scripted way.
The problem ended up being that when I tried to actually upload it the bootloader complaint right
through this error.
Instead of reading the source code for the bootloader their documentation for the repository is really great.
So it's essentially said that the application image you're trying to flash is too large.
So you know you need to essentially increase the particular parameter in your RTL
and then that should work hopefully.
Based on their recommendation I increase this instruction memory size from 16k to 32k.
Recompiled the design again.
This time the bootloader didn't complain but I still wasn't getting any hello world right.
No blanky, no hello world from the basic kind of Zephyr build.
And so I said okay let me take a step back.
Let me see whether it's a problem with maybe it's a Zephyr problem right.
Let me just go back.
Let me go to the reference implementation on the software side and see like okay if I can I just load a blanky example.
I tried to write building it still same issue.
I couldn't get their application to run one loading through the bootloader.
There's no indication of any error.
The bootloader just is there.
So it seems to me that the bootloader is actually jumping to the application.
But when it goes to when the application goes to load there's clearly something wrong.
So the question is right and you know this is hopefully this is why I'm here right asking for help.
So I need to understand like what tools.
What resources that I can use to understand like okay once I'm in the bootloader you know like what's going on right.
So clearly something is working in the design right I can get you right output.
The bootloader is actually doing something right it's actually receiving the image and it's saying okay right you know the image looks good.
But then when I try to execute that image right that's where the bootloader is complaining so the question is is it.
Is it an FPGA problem or is it still a software problem right do I need to debug the bootloader to understand.
What the problem is or do I need to debug the FPGA size to understand what problem is now one thought that did occur to me today was that one of the features of.
This implementation is that you can actually generate VHDL of the application that you're trying to build and actually embed it into your actual design so that you know if that works.
So I don't know like if that's a better way to debug something or a worse way does that would that tell me anything.
But you know again you know from from an embedded software perspective point of view like what are the types of tools that I can use to and it's not just.
The tools right so in this in this project I used like truly open source tools because again the the platform that they supported was like you know somebody had used open source tools.
To be able to like build a design and and load them.
But what can I do in terms of debugging right so you know as an embedded software engineer I know J tag I know SWD I know GDB I know how can I know what you do in those cases right how to debug software.
But what are the equivalent tools that I can use from an FPGA point of view right I've done the Ziling's ILA and I've used an opposite identifying micrachip.
Platform so what are the equivalent of of open source tools right like what software tools do I need what hardware tools do I need.
To be able to kind of perform these troubleshooting steps so.
So what are the next steps right so.
Obviously right the first step is just to get the application to load.
And then just do what I'd originally planned out to do right get these comparison metrics and zephyr.
Hopefully you know if it's successful I can present it next year.
And then again integrate the see a few this is the whole plan of this into zephyr right add those necessary calls and try to upstream the libraries or the modifications that I make disaster.
I try to get it in zephyr as part of the architecture implementation and then the next thing that I want to investigate is.
What I a new feature that I learned of the risk five is is and it's it's an extension or enhancement of CFU which is a custom function subsystem and it allows from for my understanding it allows essentially.
The hardware to and perform operations that are completely independent of the CPU and that doesn't need any CPU dependencies.
So hopefully those are the next steps and hopefully you know somebody out there can help me figure out what I did wrong so far.
Yep, so thank you.
All right, well time for questions.
Any questions?
Yeah, so what I understand is that you enable something first in digital design right and everything rest of the software let's say was same and now it's not booting in application right.
So first thing would be if you change anything in digital design right you would have to run it against some basic verification right with very later or something.
That's some messages in the system where you log in see what happens.
That's the basic thing.
Again, and if you come on the software side then so you have let's say directly accessing physical memory right to load the loads.
So there are other software also right not just the boot loader that you can run on a physical application and try to access that but for the address that you have added in the digital design.
There are things like mayor box or something else.
By saying that you can use to access that but the thing that you have added in the design.
Are you able to access it or are you pretty.
So you're saying even instead of the boot loader or remove the boot loader what does boot loader do is that it writes some.
This is some basic physical address.
You can do it yourself jump there and try to do yourself.
I see right not don't try to jump to application land and try to go from.
And you're also just suggesting to remove the like just remove the enablement of the see see a few and then just.
I would first enable the see a few and do some verification before I go on the software side.
And and for verf like would and can you elaborate more on the verification.
I use very later but who have money they use model same.
Got it.
So many things right.
Got it.
So it depends how much you know.
Can you spend on that right.
Yeah, maybe also to expand on the like available back in features.
I just looked at it and the new RV32 also supports like on ship debugging.
So another feature you can enable and then you get jetac access to it.
You'll probably have to add some pin definitions because right you need for wires for jetac.
You need to tell the FPGA better out those.
Got it.
And then you just have jetac for the CPU course.
So that's maybe also another avenue you can look at.
I see.
You ask for the debugging tool for which let is providing you this is called reveal.
You can just check with.
But then you have to go through those with a tool flow and right.
To localize or you have to go in and stuff to get.
At the instrumentation, but this is working.
As you are used to everything.
Do I do I need to use to I need to use their.
Kind of debugging tool with their entire work because this design was implemented.
I'm not aware of any open source tool yet to do the same things that will be identified with open source.
I have not seen that.
I would be very glad to see that.
There was a really really really all to providing some kind of that features.
But it's so all that it was using them parallel port to.
Collect to the GUI and things.
So this is a missing piece.
And as I saw you are using.
Your reference implementation is in VH.
So you are using G.
G.
Right.
Yeah.
So.
Try to to simulate your whole design in G.
Go get it working there without the hardware.
Get the inside.
To each year and get gt.
Keywave.
And you can also think about using more elaborate tools around ght to get your serial port running.
As it would.
Oh I see.
Interesting.
From my head I don't know if co code TB supports actually you are.
Communication things like that.
co-co-TV is wrapping around your simulator so you can put stuff inside your simulator
and getting things out and to cause emulation or additional test cases or you do not describe
your test cases in VHD or wherever you lock anymore but you do the whole control of your inputs
and outputs in pipes and so this would be a modern way for verification of this kind of
if let's say I try lattices tool flow right to build the design with let's say ccfu enabled
and it works and let's say with the open source tools it doesn't like well I'm just I'm just
kind of speculating if that if that were to happen then like would you would you have any suggestions
on okay like on the open source workflow it's not working right do do I still go down the route of
verification and you know doing test benches to make sure what's going on if that happens
open a bug report on cmbflow okay because this should not happen it's the cpu is described in
plane vh here so it should not matter okay cool cool god thank you your problem you see somewhere else god
from my copy god thank you yeah
actually if you have something that is working with latice tool not working with the open source
tool it could be also maybe some timing issue sorry was a last part timing issue timing
god but then wouldn't if it's but wouldn't I see like a failure in the implementation
if it's failing timing god because I thought like timing timing validation is like a
is binary if it passes then your design should work if it fails and like there's no gray area
if it fails you get the binary but it won't work god okay and the open source tools like
would you expect them to still generate a binary even if timing fails yeah that does oh they do okay
like so we're already once yeah god okay thanks because because in the latest tools
it didn't even let me build a binary like the the the bitstream wasn't generated
if timing fails essentially so I just assume that if timing fails like it just doesn't
generate a bitstream for it god it okay but but but the blue loader still boots though right so
I don't know
Thank you