So, I will be there presenting how at television we are using a differ for embedded video
system.
So, bit background at me, I am a contractor mostly working with television which is a design
service company and we do we build embedded hardware for anyone who has got projects to
build but not all the skills from PCB to firmware to FPGA programming to do that and
the FPVitas and we have them building the real products hopefully to put them on a market
someday you say, if you go this far.
It's a good illustration of what I typically do and they are picking video system, various
conditions with the same laptop on a different packet but as you hear, I am curious about
a lot of the means and I scratch the surface of everything, probably a lot of you will have
a very good experience in one domain which will be absolutely enabled to understand and one
of my goal is somehow to be able to interface the people who are experts in the domain with
their actual art they have to do but on smaller system, on embedded system which are more
challenging than self because you have to do firmware was there is an application programming
and we will see how we can start to approach this kind of challenging system to build
video out of it.
So that is because very familiar examples that everybody might have in mind, kind of promising
my system, very typical but that's not a complete video system you have the parts that
you see here but the images coming from somewhere also and if you have to pick the whole
system and to them from the images produced to when it is arrived, that is a lot of steps
if you try to catch them in categories, you will have a professional video camera like
the one that is filming me and the video editing software sometimes that happens in a
well time when the video stream is coming on, a distribution network that brings the image
to where it is used or watched and finally the home cinema that we are that we are so relatively
to embedded system, we could say that the budget is rather limited for the kind of equipment,
for instance the camera budget if you can make better camera road to make better movies,
the movie company will buy them so we can go out almost as far as we want in the quality
and we have very nice image sensor in this device if you do it in software I guess the
pixel that a center is for rendering, express the lack of risk to tell the budget constraint
for editing software resources, in some case of course not every system is like that but we can
also go very far in a distribution, there is gigabits, internet networks, as high resolution
video transmitted on demand, as a scale of embedded system that's not the kind of low
constraint resource that we are used for and the home cinema is kind of a comfortable system
to watch video but if you walk on every video system you might have to shrink all that
into something like a baby phone that's not the exact same kind of a budget unlimited budget
constraint that you have, you will have to put a price on it, you will have to get people to buy
it and if it was costing 10 times the price which is only a certain cheap for high quality
camera people the product will not exist altogether and you find all of these elements inside
you have a huge cost budget processing budget because you cannot buy expensive part into it
also a huge power budget so if you get powerful parts, powerful CPU side you will run
out of power for battery device and you have time budget because sometimes things need to be
very low latency and reactive so if you integrate some processing in the middle this will
affect the performance of the system that suggests we can only walk at low resolution but
that's not entirely true because if we look at the professional video camera there is a system
running inside the camera as well as there we need some software inside the camera itself
which might be a complete full stage line system or depending on the use case it might be
a smaller lower power system because it can arise sometimes even higher quality one
battery operated and everywhere you can shrink the system to consume fewer power then it's a good
thing to take so it's possible to have a smaller system in the video device system so it's not always
low end so further embedded devices we think about some very cheap things that we distribute
on the market for sense but all sort of system high and low end are using our prevent embedded
world that's mostly a question of which scale we build things if you build something that's
start a powerful system it's a powerful device but you still have an embedded system somewhere
and you look very close at the resources and that's probably what defines the
MLED systems close to the hardware close to the phenomenon that you use
so what if you want to implement a video camera we could see go with something simple because
Raspberry Pi computer and plug USB camera to it and bang we have a video system and embedded one
low power too that's true that works with embedded system the idea that we are looking
inside the USB camera for instance what is inside the device where it doesn't
using a device if you look what implements the device this kind of operating system we are
observing and one possible operating system for doing that is differ which is
a very familiar to Linux API but it's a completely trimmed down system and it works on
much smaller chips that's an example of a product we actually build USB camera so that's
a very small module that's used in terms of this three camera and we do not find a lot of
Linux simple good work computer of this size you can actually see by because it's scaled up but
it's very tiny so we have to find video systems that can implement a complete video pipeline
but smaller but on that one of smaller chips on what can around Linux and then we have to
reinvent the complete video system for such smaller chips that's an example of how you can build
camera device out of such a small board you can connect it to a USB connector and you have
camera connector and that's just to be a lot of a small USB camera for instance
so you could even to click Raspberry Pi Team to not use B camera that will work
or a small embedded camera and they'll actually company who do this you will have a
poor budget that's already by concentrating quite a bit of power it's if you want to have
battery operated device you start to really want to shrink shrink shrink the power operation
the power performance is good but that depends what you mean by my good that you will always
have some device that is a little bit more the one typical of the shift device requires
and the cost latency will also be a limitation in some case depending on what you do
but it has so many to three possible that's how do come is that do the camera model that's
Raspberry Pi and you can play camera to it and that's very good what camera is if you wish to
it's not using a Linux system that works so every video system is not about the size of the device
itself it would say shrink it shrinks the constraints but there's also some very large video system
which has a whole thing called very large to this cup and chill it and it's in this very large
it's also a good opportunity to be able to jump inside of an embedded device embedded because
the computer system is part of the building or device that just an opportunity to to picture yourself
what can be done in the device kind of a crazy machinery all of which we could be even to be
shrink in the time of camera I have no idea how the revolutionary scope works but
just good illustration because we can imagine how this all different aspects of video system come
together you will need some kind of networking for instance in different studies net you have some
motor control for controlling the aperture, the auto focus, etc we have flaps in the embedded
camera aperture to controls the amount of light that come into the system you have all sort of
indicate all sense of the building interface connectivity for instance with canvas or all the kind of
system of course on the camera you not have such a large computer system which I have no idea how
works but you will have some idea of for instance instead of a large computer work for motor
or for opening doors you will have a voice call motor for autofocus as dedicated just to
a quickly reacting to focus change and you will probably have a small tip that controls the motor
and I have ice policy communication with that chip for instance in differ you can use
ice question library that's dedicated for this kind of use case and you will want to
pour off the device as much as possible when you're not using it I'm quickly pour it back on
so this poor management integrated into the function of the whole device for instance in different
there's a poor management system that's dedicated to turn seem on and off so we are that's a
view of how to implement a video system that you will typically run with V42 which is excellent
API for being a super hardware support but unfortunately in some cases some video device cannot
be built with a complete powerful device and you have to use an autos and one such autos which
you try to use for video is differ so what are currently supported on differ if you can currently
just load them a different autos and use them for video devices just all of the people what
that might look like physical if what depth can take are there around the way that the goal
is always we suffer larger devices and we try to shrink them and shrink and see whole small
we can get with an in time of size or cost or poor constraints so I am ex artis kind of the bigger one
it's a gigahertz CPU that cannot run Linux not so many models that kind if good to have it's
good to have fast video CPU in device because when you walk this buffer if you flush the buffer
more frequently that means that you can get the same that that's what put as you have large
buffers that you flush once in a while so if you can go faster you can have smaller buffers
and in turn it means that you can have fewer RAM and you can eventually omits the RAM chip
so you can save power by removing parts of your system also you can use a built in RAM in some
in some cases it's not so that's kind of kind of large but that's just because there's a lot of
different things on it and it's used for all the debugging VPN image sensors for constantly in
easier since NXP team working on it for instance and you can have you can have internet on all
sort of a of a city on board and together all the 92 of the resources you can see that there's
lot of trouble coming in maybe you can have a display out for instance if you have a preview display
on the camera what you have something directly in the hardware that avoids you to between the
Nazis with software which will be too much for even for your health system you have an external
interface such as USB 2 or ethernet you can kind of have the same range of properties for the
attack that comes in and the data that comes out all of my data the same and very fast CPU
course typically when you have two CPU course one will be for control one will be for the data
number crunching so that you can use as much as the number crunching and get all the control
all of the powerhouse of the board so that's the system we build and it's going to be a
particular system to work with because it's FTA so you customize the contents of the chip
you choose how the CPU looks like you should have the video accelerators look like and it's
up to you to provide this call this is why television is also designed to save this part
now because it's kind of challenging to work with FGA so it takes more time so it's a resources
with a with a company to help anyone who wants to build system also for FGA and one thing that's
kind of strange is the use with earlier I mean that's super nice but what 18-gahead CPU
for so many MP lines and so much throughput that doesn't make sense you cannot process the video
so that's a good example of embedded system that can be totally disbalanced they can be very
specialized in transmitting from one side to the other and if there is other acceleration to go from
one side to two the other well you do not have to pay so much about the CPU because it's not
involved in the actual operation of the device so in the end the CPU is only there in the corner
to keep the control operation going you do not need a large complex system like legacy if you just
want that you will have to make the system much larger and what are the results would be used for
a small control algorithm you do not need such a small at all that's why we are that's the
reason for which we are using a different hour hour ecosystem and in other example we start to
shrink a little bit as the other boss from NXP different as a slightly smaller CPU and you will find
some kind of interesting peripherals that you usually find on larger board like ATMX ATM plus
and like a neural processing and trying to do AI inferencing if you want to detect something
on the image on the device directly and react to it very fast for instance so you will have
this kind of wall which is relatively compact also much more than the one we saw earlier which is
one internet interface for instance only gets like the result gets shrinking a bit smaller the
video device less capable but also much lower power and you have interesting poticle special
purpose course which I have a neural inference core insights if you want to collect the image frames
pass it to a detection and giant and tell you that with someone on the system for instance
you will want to detect if someone is walking in front of a device to wake up the device
and save power where there is no one in front you can you have to detect it there is something that
looks like a person and to do that you cannot communicate with internet ask get the result back you
have to do it on the device really embedded and if it's a feature that is to reduce the amount of
power the system itself cannot consume a lot of power and you have to try to find ways to shrink
the system and one way is to use fewer RAM and then switch to embedded operating system that
brings you to reduce the power footprint also board this time very physically small you will
have a very compact board with a GPO pin for controlling things around it with a camera inside
with a Wi-Fi with an IMU for detecting the orientation of the board so you start to have a lot of
hardware things how to gadget that permits to associate with the video for instance you can
know the direction which you look if you use the orientation sensor the IMU and you have
some kind of series available on the processing power to do you can do a few video processing some
kind of converting the video trying to convert the pixel formats or things like that that would be
required for instance with USB communication and you have a GPO compression core which is nice because
it permits to reduce the amount of data is transferred to its body and still equivalent but for Wi-Fi
so you have very small inside this time this is a Wi-Fi slightly high power but you can really get
compact device that will communicate well as using the first and this one is also supported board
you can get some kind of interfaces this time no USB but alternative interface and you
still have this kind of same order of magnitude of between what the image can do and what the data
interface can do so if you have Wi-Fi could eventually transmit full frame of this depending on
how which FPS you like always a kind of matching between the resources of chiplighter so on this
time we want to treat everything away from a board which is not optimized for size but of
you might more for cost you have a chip with almost nothing on it there's a camera connector
than the display connector and that it and you can also customize with the less expensive chips
if you have high volume device you want to with a video system into every watt into every door
it has a lot of durer run you will need something very cost optimized so that you can
actually have a foldable system all the work is it will not be a usable at all and still some
reasonable processing side that's the same as the Nikla so how with fewer processing that's nice
but you leave the comfort of line if programming environment and all of these software stacks
you will have to be reinvented they will have to be redundant different ways so let's look at
what we have now the ever has video API that's fairly low level and that's most of the work
for the application side which also permits to port libraries for vision processing
in particular algorithm you wish to do for instance the QR code detection
well that's one particular thing cool try to pause in one of the available language that you
can run on top of the zipper and so that a video API was introduced by Luke Pula at 2019 and he took
all of the constraint and inspiration from all the others video API put it around and try to adapt
them to the zipper according to what will be pleasant for the existing zipper developers and that
brought something a little bit like this like to picture you have an image sensor that provide
video data to the chip and on the chip you will have a MEP driver that receives a video signals
as I kind of Brian them it to fill them into memory and you have all the memory buffers that are
loaded in with NQ and that taken out full of pixels and you can cycle the buffers continuously
and you like that you like in the conveyor belt that will be filled by some pixels as the buffer
pass in front you load buffers and takes them out that's the NQ and the QR operation of the API
so note that here you have an image sensor every camera every video device need a driver
for the particular image sensor that collects the pixels and it's kind of really hard to get
your hand over the data sheets for the image sensor that's very protected, very private,
for the company who do them as a lot of competitions and it's very good to have access to this
image sensors so if we consider that only the IO interface for the MEP driver not for the
image sensor itself the image sensor is turning data to the hardware as a software only operator
that's a data for the MEP driver that's an example or for you will actually look at the code
you start the device you allocate the buffer and then while true NQ they queue and queue the queue
and then you add to it when you they queue buffer you can do something with a data out of it
for instance process with application so that's the model if we plug it we can use it by first
to plug it to a difference for instance you had dipping in code as you can pass the buffer to a
different device and the device will operate on the data and you'll have to provide new
buffers so that they can receive the converted data and cycle them again like you you will
do in a normal like like you did for the first device and recycle the buffers so that you
don't have to call the allocator every time so that's the NQ and DQ operation of the API
we saw how it's implemented just before and that's that works you can do the exact same way
as before and queue for the first device at the queue and then and queue again to onto the next
device and the queue that works one by one that's that was very parallel you have to wait
at the full image process before you can start to the 7 so you have you can do something similar
with when you unlock all the buffers first you thank you them all first and then on several
thread you'll thank you on the queue on the separate like separate conveyor belts that convey
the data into the devices for instance the first time in thank you the queue and queue the
queue continuously and the salt of pins will also device to load the data in and out
and that will be a second trade for the second profile and answer for every kind of conveyor belt
to have a separate NQ NQ to operate and at the last step you have you can actually use your
JPEG buffer that is ready that is processed you can use policies with a network stack to
send over network or USB for instance so that's the same kind of API but using Paul there's also
pulling API which you can use in video devices so that you do not have so many thread at once which is a
bit a heavy way to use I will let you review that on the slide which will be on the website
so that good parallelism and that's that's a good bedwork for making things a bit more automated
so that you do not have to put your hand onto the device and tell us because it's start to get
low level as this point here every block you can also have controls if you want to tune the
exposure you want to tune the brightness you want to control the JPEG compression level and you
can control the Alexa format etc there's also a video format for Deeper just like in Deeper
to Deeper API are really inspired from Linux API. There is to get some of the same environments
for people who are used to one or one of them and the drivers they also communicate directly
for the controls that you don't have so much manual work to do for every detail so let's
start to be a little bit more automation for all the drivers together so that it's not so bad or low
level to bear it with here is how you permit to you can point the driver at each other that's the
device to it that's a configuration language at Linux and Deeper used for configuring the device
you declare a port you declare an endpoint and each endpoint is connected to the next device endpoint
that's the way the drivers can know they can look up themselves what I am sending that are
too and where do I get that from and then you can connect them to configure them with fewer manual
walks this way that's the full chain from for such you can review that later with M2N with
for the example area so we keep in mind that we still try to improve these APIs with
more comfortable and easier to integrate with actual live software libraries that do
image processing it's continuous evolution of these APIs so now let's look at what if you
are example of what we can implement a video system like real real use case for for these APIs
on how they are actually using in devices over time the domain of photography is giving birth to
imaging and on top this which is a domain where you do not just want to watch the photo
but extract information usually a human extract information from it will only also having in
the last decade the last few years the rise of a computer vision when instead of human
extracting information you don't need a computer that integrates a video function into this
external ecosystem for example robotics or a lot of other use case which will be over game
if we start from the route the phenomenon for the premise to sense lies from the computer system
is photo diode it's permits to generate a very small voltage out of the other flight
and if we just use a single photo diode if we put it on a on a mast on a tube and we rotate the
tube and we can scan position one by one we get an image and that a single pixel in
end one at a time scroll for it a bit and we get the full frame like that of course that
it's possible to implement it with deeper that's API for it but usually you will want to
get into the next general you also realize that work is just one footing so there's a
CO2 sensors like the actual CO2 measurements of the Earth is done with the
single pixel like this which is such kind of infrastructure so it's used in instrument
it's scientific industry not that also genetic one thing which is done with photonics with
emission source where it's only have a number to produce low emissions for for genetic
concern and we have chemical process and as the end you will study all put into the chemical
process with an image sensor on the anything control actually primitive is that but you will
require to have a sensing the voltage out of the image sensor that's very kind of different
from actual camera references so that's not what he's a line sensor a single line of pixel we have
one pixel and we have a line of pixel that you will use that with ADC's and you will have to
sense a voltage very fast one pixel at a time you will need kind of a specialize system
on top of that and for instance that analog device will provide ADC for for deeper that you can
use that you can actually implement a line sensor for instance for doing a scientific instrument
using the deeper all that process using the tiny plants that we are building you can implement
most of this logic using a deeper to right right today so now with multiple lines you actually
get an image sensor the difficulty of embedded very small devices is to get access to a camera
port and camera port I kind of not present on every device you can use adaptor chips so you can
if you need imaging on a very very small device we look for my Raspberry Pi
conference as you can always get video out of it so let's quickly look at what kind of
processing you require to get an actual image out of real to the image sensor because what
you get is that that the image you get out of image sensor or low resolution is normal but the
color is completely weird you have to make several step of processing before you can actually
get an image and you are more familiar with this kind of image for with normal looking color
good contrast and feature so it means that every camera device has to do some processing in order
to not look like that as well as that's really what you get out of the works that's one thing
to implement so what why will you want to do all this step if you just do robotics you might
does not have to look good it's okay if the computer looks at something green but it can have
a few artifacts that are going to trick the robotic imaging system so you also have to do some
kind of image processing for robotics so that you have a clean image without without artifacts
so now let's look at actual different devices for making a better photo material you will have
that's a sense of wavelength of light so that you can see the color spectrum of light
and you will have to use a light like the one we saw earlier and so you need the very fast
ADC and you really have to to as fast as possible wall along with it which we can
implement that with deferral already that's the kind of image you get out and that's the kind of
that you extract from the images to get the wavelength according to the light and for every compound
chemical compound it's going to be different spectrum like that so if you want to implement a drone
that's drone system available in deferred that's a robotics and motor control could call
Kali pilots for auto pilot drone and it's deployed an independent on top of deferred and the wall
that's the deferret so it's actually there on deferral as well as the kind of thing you could
want to do for imaging along with a drone for instance that's Kali imaging with infrared
that was a color not looks not green but fake color here and you can use open software to extract
the images that's kind of use case you want to use very small embedded imaging imaging for
you will for instance you want to use monitoring with taking a photo of your
process you do not need a large resolution photo but you need to integrate a lot of all the sense
for like CO2 the blink and LED ah thanks up well I'm I'm very sorry I just planned too much slides
so it's yeah do I have five minutes okay thank you questions okay so that's a lot of
example let's jump quickly let's just let's just continue to be able to review it
ah that's a yeah too that's I just planned too many examples that's easy to cut all the content is there
it's just to give you the idea of what it would represent to build videos devices with
easy for today that's another example of how you could build a Bluetooth based my classes which
you can really want to show you the video device physically and also for power with only one
emissions of the chip and now I do not have to look at what's next coming in different videos
sorry but this is coming next that means you will also be having news of it coming
coming later but maybe something even more interesting is to ask to you what are your expectation
regarding embedded video devices where you are after building small compact video devices where
you're facing situation are you wanted to have something small as an Linux and did you ever try
see for a PI for a video and do you have back for that or if you also have any sort of questions
I will be very happy to hear about that
so if you do not have questions we can never ask to look at the rest of the slides but
yeah
these these I and exposure in the environment.
Yes, of course.
So do you mean exposure is up?
Yes.
Can you are asking if what happens about all the emissions of controls?
Is that the question?
Yes, no.
We said that the remote sensor would have to adapt.
Oh, here.
Yes.
So from this point of view, the data will make it easier.
All right.
Yes, absolutely.
Good point.
If you have image coming out like this, the first step.
So you are asking, when you have some control on the image sensor itself,
so that you can tune the image.
For instance, auto exposure.
That's a very good point.
And auto exposure is one of the only feature that is only integrated in the image sensor on
not processing on the chip.
So the first two pictures are exactly the same.
One with auto exposure on one without.
It just has a default exposure is correct, but you have to constantly tune the image sensor.
And that will be this kind.
This cannot API.
The control API.
We have the software.
Send command to the image sensor to adjust.
I want more brightness.
And you have to study the image.
It is totally dark.
Increase exposure.
Or as the positive.
It's way too bright.
And you have to read this exposure.
You can do the control with software.
And you can send ice per se command to the image sensor.
And this is why you need a lot of drivers so that you can know which command to send to the image sensor,
which ice class here.
There are other questions.
Is that data sensor?
Yeah.
Oh.
What did I get?
That makes it.