I think you just need to prep the power and say a hum or something.
Just get started, people will quite soon.
Okay, 13 hundred layers.
Top begins.
Greetings, salutations.
My self and hands will be giving you a talk on a live camera,
and specifically a software ISP in the camera.
This is Hans.
I'm Brian.
As you may know.
So, headways say your name exactly.
Hans the corner.
The corner.
But it's just Hans the good also worked.
I think the good the better the ugly.
Actually I saw your full name.
It's much bigger than that, actually.
Very loud.
You say my name just like Brian.
We both worked in the camera domain.
I'm doing some stuff with the welcome camera at Iris the Venus.
I do pull a request now for various people.
And I've had that experience for the last few kernels,
where I've done side off by a hundred and fifty patches on which are mine.
So, be careful what you wish for when you wish for it to be a kernel container.
This is what the job really is.
You know that.
That feeling as well.
So, our talk today will be basically on software ISP.
What an ISP is.
Why you need an ISP.
And what you do when you don't have an ISP.
Up to the mind says.
So, first of all, I go to assume that you're all very intelligent people who may know absolutely
nothing about cameras or images or image processing or anything like that.
So, you're going to learn very quickly because I was trying to estimate the average IQ in this room.
And I would say it's probably not less than one-third.
So, we'll all be very quick in the uptake here.
Bear encoding.
You may have heard of it.
It was created by mine called Bruce Bear.
If I remember his name correctly, in 1974, according to this document right here.
And the theory of operation here is you may notice this mosaic pattern works.
Green blue, green blue, green blue, red green, red green, red green, red green, red green, red green.
You might say to yourself, why does green appear so many times?
And the answer is actually because if you look at this graphic here, your AI is most sensitive to the
colors spectrum and green whether or not you know that.
So, if you think about how a light sensor actually works, it's just an ADC, right?
It goes from zero to some value.
So, it's just a number.
What you have to do is you have to make that sensor sensitive to one particular color.
So, you put a filter on it.
It's basically a black and white sensor.
You put a filter on it for green or red or green or blue.
And you're recovering the blue part of the spectrum at the blue sensor.
But immediately you already have a problem because, okay, I've captured green right there.
But it didn't really capture blue.
Yeah, actually when you capture green, you still capture elements of blue and you capture elements of red.
What you have to do is you have to infer the value of the red at this green based on the adjacent red.
And infer the value of the blue at this green for the adjacent blue.
That's just a simple process of, let's call, linear interpolation.
Just looking at the adjacent pixels and making it a fairly good educated gas based on the value that you sampled at the pixel of your ad.
And the adjacent values, what the RGB for each one of those pixels was.
And of course, again, because you're always more sensitive to green.
You want to capture as much information in green as you possibly can.
And so, basically what comes out of image sensors isn't an RGB picture.
It's, it's usually this bayering called a picture.
For exactly that reason to capture as much information as possible,
but you can't capture all information in green.
Because obviously then, the image will be green.
Is this bit you, see you, see you, see you, see you, see you, see you, see you, see you, see you, see you, see you, see you, see you, see you, see you.
So, one of the reasons why I got involved in cameras or re-envolved.
I also bought involved in cameras in 2008-ish.
It's because X86 laptops, I work for Qualcomm now,
but it's originally a work for Red Hat and a work for X86 laptops.
X86 laptops used to have USB camera modules.
And USB camera modules use a standard protocol called UVC.
And there are things like V-Mose and E-King,
but also picking the right gain and exposure value
adjusting the white balance.
We're all done by a little micro controller
inside the camera module.
So it pretty much just worked.
You could just ask the USB camera module
give me a picture of this resolution
and it would give you ready to do pictures.
But recently, also X86 laptops
and this Qualcomm laptop here also have moved to the smartphone
camera model, which is you just have a roll sensor.
And you actually get a roll sensor or images.
So roll buyer encoded images.
Just like the roll images, if you have a professional camera
and you decide to pick a roll images in Linux,
and then we need to do something with those roll images
and make them usable for a video conferencing or whatever.
This is where our actually lip camera comes in,
because this is a complex problem.
There are a bunch of building blocks in hardware,
which you can help to translate a roll image into a proper image,
which together we usually call an ISP.
A hardware image signal processor.
ISP is tend to be shrouded in sequencing,
that it's secrecy, that's slowly changing.
But the reason why the soft ISP was created
is to just have a generic software implementation,
which would work everywhere,
why we were working on figuring out
how to actually use hardware ISPs.
So that is where the software ISP comes in.
So first, what does the hardware ISP do?
It's basically dedicated ASIC blocks,
which are specialized in doing the necessary transformations
really efficiently.
So really fast, but also really power efficient.
It does the demo's E-king.
The slide says it's implements the three ASIC.
So in camera, as we often talk about the three ASIC,
which is auto gains like exposure.
Exposure time is very important to get a good picture,
and then sometimes you also need to boost the gain a bit
if it's dark.
Even after the exposure time, auto white balance
and auto focus if you have a non-fixed focus lens.
It says here, it implements the three ASIC,
it's nice for on the slide, it's not correct.
It implements statistics.
It does it gives you a bunch of histograms over the role bear,
and those help can help you to figure out
what's the right exposure value, et cetera, is.
And then there is more secret in there.
There is a more simple ISP example diagram at the top.
You see the role input image that is something
called black level correction, because if you actually
fully cover a sensor, so it doesn't get any light on it at all,
even if you could like a really thick cover,
it will not give you zero pixel values.
It will usually give you something about a value
of around 16 for if the values go up to 25 or 64,
if they go up to 1,000, 24 or whatever.
So there is like this black level, which also has to do
with how the analog electronics works and bias currents,
which you need to basically subtract from all the pixel values
to make black actually be black.
Then that pixel correction, there might be some defective pixels.
You want to sort of interpolate around those,
and ignore them, and then we get the demos
in eking and in white balancing, and some other stuff,
like denoising.
This is from the work chip ISP.
You see that the actual physical hardware ISP is
a lot more complicated, so there are a bunch of other things
which hardware ISP does do.
So we have the first software ISP, which we
presented here two years ago, with this exact same room.
Back then, we just had a demo.
But it was merged a couple of months later,
is the CPU ISP.
I'd use this a little camera, because the camera is quickly
becoming or actually is pretty much an hour
day to de-standard for dealing with cameras in Linux.
We no longer want applications to directly access
the video hardware notes directly, because that works fine
with the UVC camera, where an application could use,
say, give me a picture with a modern complex pipeline.
You need to set up and configure the whole pipeline.
So we don't want each application to have to do that.
So the software ISP, which has been implemented
and that streams does some of the statistics stuff.
And then it does outdoor white balance and outdoor game
in camera correction.
It doesn't support outdoor focus yet.
There are some patches out of three for that.
Let's not dare yet.
So the software ISP works pretty well on powerful hardware.
And it works pretty poorly on, on, say,
a Raspberry Pi 3 level device, like the Arduino Q,
which was recently released.
And it does six FPS there with an IMX-2-18 sensor,
and four resolution, which is not what we want,
because the sensor can do 30 FPS.
So yeah.
So this is where Brian's work from the past one
and a half year of something comes in,
who has been working on GPU acceleration
for the software ISP.
So like Hans mentioned there, on the class of device,
this laptop here in the T14S, running the home OS system,
the X-180, 100, you're more than enough processing
power to do all of the 3A's in the CPU.
However, when you start going down to just slightly lower
down, so there's a type of processor from Qualcomm called Kona,
which is released in 2019, the SM8250.
I have that development board in my desk.
And I cannot get 30 frames per second
with the RGB sensor used in CPU ISP.
Now this is a powerful processor.
So as we go further down into the,
embedded domain there, you simply just can't do,
even the basics, the 3A's in CPU.
However, there's this magic device called the GPU,
and the GPU is actually a massively parallel pixel processing engine.
It's all purpose in life as to process pixels in parallel.
And this has been useful in LLMs, and so on.
Actually, I'm not going to talk about that.
I don't really know how the massively parallel bit of it works,
but definitely the pixel parallel bit of it works really well.
So for our use case, that turns out to be quite interesting
because as I was telling you before,
the first thing we need to do is interpolation of adjacent pixels.
So if you think about it, your fragment shader
is basically a program that gets run for one pixel,
and you can decide stuff to do at that pixel.
Well, if you have to linearly interpolate data,
that's great because I'm at one pixel.
I can look at the next pixel.
I can look at the pixel to the other side,
and then I could decide to perform logic based on that.
And I'm doing it in parallel.
So I'm like a scalar CPU, even the superscaler CPU,
which has loops, the whole purpose of the GPU
is to kind of execute a whole bunch of tasks,
a whole bunch of tile tasks in parallel.
So that's really how GPU ASP comes into its own there.
And that the camera already had some of the upstream
and some of you in the narrow actually, where I have to work.
Obstringed support for vertex fragment shader
to do a simple debaring.
However, it didn't have any of the necessary additional steps,
so black level or contrast or color correction matrices,
or any of that others solve.
So what they spoke to Kirin and Laurent
that over a number of months was let's reuse the existing pixel
and fragment shaders.
Take it out of the QT application where it lives
and brings it into lib camera, proper, headless,
which is what we did.
And you might ask, well, what you use fragment and pixel shader.
And the first size is that is because we already have it.
And we want to just reuse software.
And there's no use throwing out software that works.
It represents development effort that yielded results.
And the second answer is actually, once you get through implementing it,
you find that for most of the stuff that you actually
for one specific thing you want to do, specifically
the stuff Robert Mader from Calabra, worked on Robert's right
here, given you shared it, because Robert did some great work
with this, but to do scaling, actually,
you can do that in Mesa quite well.
You can do it quite handily.
And at this stage, actually, you don't want to compute shader.
You really do want a proper Mesa program for that.
So we're just going to talk a little bit about lens shading later on.
Dead pixel correction, denoising.
I mean, these are the next kind of steps.
So the image is noisy.
You'll see it later on.
It's noisy.
And we're saturating in places.
But anyway, it runs really, really fast in comparison
to CPU ASP for every, so let's actually move on to next slide.
So this is the implementation we have right now.
It was committed to not 0, 6.
So yeah, well, this is upstream and in-depth camera now.
We managed to merge this just before Christmas.
So this is a very simple.
So before we had the rupture, yeah, this really complex
and kind of cool ASP.
So in homage to that very nice graphic we have, this one here.
But this is basically what we're doing right now.
We take the CSI2, Mippy Bayer data in, D-Mosec,
black level correct, anybody who knows anything about this,
which would basically be the lip camera guys
will tell me, this is already wrong.
You should be black level correcting app before you
D-Mosec, but we do it straight after that will be fixed.
We apply a color correction matrix.
We tap the contrast and then we apply the camera.
And what we get is, of course, RGBA, H-A, H-A, H-A,
H-A, because the GPU view of the world,
the end of its pipeline is always to write a D-word.
It always wants to write 32 bits in a fragment shader.
So with this model right here,
it will not be possible to easy color space conversion.
However, multi-pass.
There it is, it's good and she's awesome.
And obviously look best on, is the best French director ever.
So then the next steps from G-P-U-I-S-P will be this.
We will have a multi-passed through.
So we'll go from the Bayer domain to the RGB domain.
This is the second phase here.
We will scale in Sharpen.
At this stage here, you can see we'll go from a not vertex shader
to a non-not vertex shader, and this is where
we're going to scale.
We use Robert's code here.
Oh, sorry, I might have framed.
Oh, it shouldn't go past the black line.
They can't see you past the black line.
So this projection matrix is where you will scale.
And we'll do some sharpening.
And then we will play Gamma.
So if you don't just hear in the previous slide,
we will play Gamma at the end.
Here we will play Gamma in the last vertex fragment shader.
Why does Gemini tell me that's where it should be applied?
And it knows more than me.
And then finally, we'll move on to a compute shader
to do the pixel format conversion.
And you last play.
And the answer is, because here, the vertex fragment shader
view of the world is to write 32 bits at the end of D
words, both a compute shader, and also
to do non-D word.
So the compute shader is the place that we have to do
pixel format conversion.
And watch the fifth element.
It's an awesome movie.
Next steps are going through this really quickly.
So I just call it basically, what the next steps for GPU
is multi-pass.
Probably is few additional steps as possible.
But you still have to have an RGB behavior domain,
a Bayer domain, and RGB domain.
And most likely, a pixel format conversion phase.
That pixel corrections is probably easy to do.
Other focus, I have no idea, either easier
hard, lens-player correction.
I have no idea how difficult or non-difficult that will be.
And we have a demo.
Yeah.
So last six months, I had a student group
from the Fahalchula in Delft, the Hasselg Scho,
a computer science department working on Lenshade
correction.
And these are two example images of what they have written,
and which is available in the mailing list now.
So if you look at this image, you
can look at the top left and top right corner.
You see it's a bit gray there, while the ceiling
is pretty much white.
And if we now switch to the next one, which
has the LSC applied, the corners
are brightened up quite nicely.
So this is actually quite nice to see that it's doing what
it's supposed to do, because I mean, Lenshade's round.
So it captures not a whole lot of light in the corners,
because well, it doesn't have corners.
So we need to apply a boost to the light level in the corners.
So this is what this does.
So CCMs, as Brian mentioned, and as we'll go in much more
depth in the next talk, the GPU ISP now supports
a color correction matrix.
This allows us to do things like, since the color filter
is leaky, so the blue field, the green cells
get a bit of red and blue light leaked in.
You can subtract red and green from the neighboring,
from it to get the pure green value and things like that.
That is what it's just at 3 by 3 matrix multiplication,
really.
Another thing which it gives us, which is actually
already implemented, but I don't
live in the demo, sorry.
Maybe you can show your, I don't have the saturation
thingy.
I think Karen shows us the next.
OK, yeah.
So if you look at the camera pictures versus, like,
commercial phone application, then the color
pop a lot more in the phone camera apps.
That's also just because they're not really a photo-reelistic
representation.
They boost up what's called saturation, so the color
is look more rich.
And I think we probably need to just, since everyone
has used to that, just put the saturation control default
at, say, 1.2 on the scale of 0 to 2.0 with 1.0 being
neutral, so we'll just actually, the default lip camera output
will also have a bit more rich colors.
A lot more about this in the next thingy, which
is interesting, especially, also, again, we're
by Robert Mader, so for Dora has been shipping the soft ISP
for a while now to support Intel-based NIP cameras,
so the non-liffici cameras on various laptops.
Robert has been working on post-markato as in all the mobile
distros, and this is a small list of phones
where they now have the cameras working using the soft ISP,
especially also with the GPU ISP, because that helps
the lot also on the mobile phones.
And then a demo.
So this is actually the GPU ISP with lens fading correction.
So yeah, it works.
And the color cell aren't actually too bad here in this
light.
So yeah, we have a couple of minutes left for questions,
I hope, it helps if I switch to the, yeah.
Ah, yeah, go ahead.
So you're working in the graphics pipeline,
which is why you're the multi-pass.
Why don't you do everything in a single compute shader?
The question I get all the time, and the answer
is because somebody else already did the really hard work
of decoding, because we don't get a simple blank stream in.
We get a CSI-2, maybe encoded stream in,
which is 10 bits.
So you have to track inside of the 10 bits,
and you're not sampling other specific place.
You're sampling in a thing that slides.
And I don't want to have to understand
that and reimplemented the computer.
And the other answer is, when we started doing it,
we didn't know what generation of GPU
would necessarily try to support.
So we tried to go for the oldest one possible.
And computers came in in the bet 2009, I think.
So I mean, we're casting it that fairly wide,
but the thought when we started doing it was,
we weren't exactly sure how far back in time we could go.
And we want to be able to go as far back in time as possible.
And if you are using a class of hardware that doesn't
support computer shaders, I'm happy to say,
we support you, but maybe get something a bit newer.
Yeah, so basically, we wanted to have like,
open class 2.0 as bottom level,
like anything which opened class 2.0,
to support at least a current single pass, GPU ISP.
Take a little time for one minute and 42 seconds,
more questions.
First up, best rest.
This is really quick.
So it should be less than 30 seconds.
Great to see CSI2 been integrated with the lip camera.
Is there a plan for CSI2 as well going forward?
Security extensions.
That might be a better question for the next talk
because I have no clue.
So when can we expect GPU ISP shipping and post-market OS?
Oh, it's in 0.70, just got tax and release.
So we'd see it up to you.
Aren't you a post-market committer?
It's a question for Robert.
Is shipping?
Today.
So first, thank you very much for your work.
You're like the delighted end of the tunnel for my new X1.
So Fedora and I have a working open source stack
for work with what are Intel webcams.
But do you know, is there any work going on for Debian,
a channel or arch-based history as well?
I guess some people are looking into this.
Sometimes I do get questions because I did the Fedora work
from people for say, Nick, so as that they like,
like, how did you solve this or this in Fedora?
So do you want to make a remark, or?
OK, thanks.
I mean, they come and I think that you don't have a mobile.
OK, so mobile and already has the GPU ISP
even, so I mean, I'm welcome to help any destroyer.
If they have questions, they hit a roadblocks,
but I cannot do all the disrupts.
And my time is up, so thank you all for your time.