Good afternoon everyone.
Thanks for joining us on a big room.
I'm here on Bingham.
I work at I design board and we want to make every camera work on every platform.
Simple easy challenge.
I've been working in bed in Linux for 20 something years and with cameras for somehow.
I don't know how but the last decade.
It's far more than just me but I've become the release maintainer for live camera.
So my name gets thrown in there just for that.
That doesn't mean I do all the work.
I actually have very few commits in live camera lately but I just merge everyone's work.
I used to be more of a kernel developer but now I just become a manager.
So today I want to talk about these color charts really.
So anyone who's just been in the talk before which I hope is quite a few of you.
I've just learned that we've now got hardware acceleration for the software's P.
This is really cool for me.
This is amazing because now on all the platforms that we have difficulty processing images and raw bare images.
We can do it in real time even when we don't have the open source drivers for the ISPs.
So that brings us on to what do we do next?
What do we need to do to improve this?
So I'm going to go through what problems are currently having want to solve.
How you tune a camera?
I've got a proposal of what I would like to see happen next.
And a big shout out for help because like I say I just merge other people's code now.
So great.
We've got pictures.
I can turn my laptop on and I can join a web call and I look like that which is stunning.
We are very green but we can fix this.
We've got live camera shipping in products where we have full tuning.
That's easy on other products but we have to do this per sensor per camera per device.
And that's tricky.
So we've got images that come through.
Brian did a great job of showing how we've got all the bare patterns and the big focuses.
The fact that we've got far more green elements that we'll try to come into the next.
And when you zoom in on the picture.
We see all the bare pattern and we've got.
Two greens are red and a blue.
We need to do something about that.
I always had this misconception that it's the fact that you've got two greens that makes it green.
And it's not because we're going to interpolate those and turn the two greens into a single green anyway.
What I've sort of started to understand more is that it's more about this thing called cross talk or the bleeding of the colors.
And the balance of what the sensor really sees the world as.
It doesn't see green as just green.
These are blend and that's with this graph taken from a date sheet.
You can see that there's a lot of overlap.
So the blue pixels are actually picking up a bit of the green sensitivity.
The red is picking up a bit of green.
So they have a bit of bleed.
But the green picks up all of the green.
Some of the red and some of the blue.
So there's more signal there.
So we need to do something about it.
And that's where these come in.
What's real?
What's the ground truth of what is the real color that the sensor sees?
And we need to convert what the sensor sees is what we want to see.
What's the real color space or SIGB that we want to present on our screens digitally?
So to do that, we need to take a lot of pictures.
A lot of pictures in various conditions.
This is in my light box at home.
I've got a big set up for this with like six lights that are calibrated.
And we know exactly what color temperature they are.
And I can measure how much lumens.
How much light photons are coming out of the device.
So key things on the note here.
You'll see I've got a bit of paper in there with the TL83.
So that tells me exactly what lights setting up.
Which lights I've turned on.
And I've got a little, it's called the Snackle.
A little light meter.
I've got one if anyone wants to see it.
It can measure the color temperature.
The color of the light being illuminated.
And also the brightness.
So these are the parameters we want to know.
So that we can say of all these colors that we now know very precisely.
These are calibrated.
So we know exactly what specific pixel values they are.
How do we turn that into what we want to present?
Now, if you look through my photo album.
It's interesting.
It's lots of these pictures.
So I'm going to take what I'm taking pictures as much as I can.
Of any device that people send me.
Of all the color temperatures.
And that I have in my light box.
And this is going to give me not just for one condition.
But many light conditions.
And then we can put that into the tuning process.
So as I said, we've got like say six lights in my bottle.
Or many in their world.
Each light color temperature is going to make that color.
And make that color chart look different to the sensor.
And we need to figure out how do we turn that back into what we want to see.
So we can, that's actually, we can linear interpolate this.
So we don't need to have every single possible combination.
You want something really low, 3000 or 2000 something.
And then something high, 64K is basically sunlight.
If you can get a couple of ranges in there.
You can go crazy and have a lot more.
But here's sort of like a bare minimum.
We can capture images and work out the correction we need to apply for those conditions.
Then in a camera can identify what condition you're in.
Apply the relevant correction.
And we can stop having green images.
So each of those, the calibration tools that we've got will read the chart.
Users open CV to find out where all the grids are.
What color they are.
We know the layout.
So we can turn that into a color correction matrix.
And I let more than I realize on this soon, this talk.
There's interesting properties of this.
Like the diagonal that I've highlighted in orange down the side.
That's basically the strong component of each channel.
You want to take the red to red green screen blue to blue.
It's really important that the rows sum to one.
If we were to apply a correction that was greater than one,
you're going to be adding digital gain to the image.
So trying to visualize these has been really interesting for me.
I also want to build up the tools to evaluate the tuning process.
So we can see what the data looks like.
At the moment our tuning files are just big bunch of numbers.
I'm readable.
If someone says to me a tuning file, I just go merge.
I don't know what it is.
So I'm trying to build up graphs to see what they all look like.
And we'll see a more example in a minute.
Luka sent me some FFM5 and someone else actually sent me another sort of real devices,
which is great.
So I can start capturing these conversions on real devices.
I've got about eight phones in my home now.
And this is great.
If people say me stuff, I can put it through the box in about an hour and a half to take these pictures.
And then we can start building up these magic numbers that are going to help us
correct the color on devices.
So we take a lot of pictures.
We've got some numbers.
We can put it into the pipeline that Brian and Hans has just talked about that they're creating.
So this is also as Brian said, probably not the right sequence,
but I've highlighted the LSC, the AWS and the CCM.
For the most part, those are the components that we need tuning for.
The rest we need to identify from the sensor or something, but these are the ones where we need to know
exactly how the light coming in is going to impact the capture of that.
I really have to see Lenshading's got posted last week, so I'm looking forward to seeing that progress.
AWS kind of comes out of the captures that I'm already taking.
It's one of the things that we can measure in the chart we can see why.
So we can work out.
How do we make that white white, which is the white balance?
And then the one I'm focusing on today is the CCM.
It's kind of sometimes called crosstalk because it's this crossing over of the channels.
There's lots more we can put into these blocks later as previous touch on.
But there's not so much tuning there.
Well, the Dean always and sharpening is going to need a lot of tuning.
We can do more advanced color correction with the 3D look-up table instead of the CCM.
So there's lots more we can do, but my focus at the moment is how do we make the colors look better.
Coming on from the Lenshading topic earlier, it's another thing that we need to calibrate.
As Hans said, it's this fall off to the edges.
You can get strong vignetting and the Lens itself is not square.
It's a circle so you can really see that circle to shape there.
And as the light traveling through the glass, I'm not going to say I know the physics,
but it basically goes slower.
The more glass it has and we get how to apply again because it sort of has a high resistance.
And once we put what we need to do is capture white pictures that are flat.
And so we can measure an average out where this happens.
And you get some really cool graphs that you can actually see the shape of the glass or the shape of the signal traveling.
So what we're doing is in the centre we only apply a low gain or no gain.
We're going to have to amplify that signal to bring it out and balance it.
What I find is interesting if I switch to the next one.
So that's two temperatures.
We've got a 3000 Kelvin and a 5000 Kelvin.
The shape is identical.
But you just see that the scale is different.
Now we have to as part of the calibration measure for many temperatures.
But also each color component.
So the green, the red and blue, they all are affected differently by the physics of traveling through the glass.
So again, it's all just about capturing that data.
We capture the RGB and we can produce these tables.
Next one.
So now we come back to what we do with all this data.
Lib camera has something called the CT, the camera tuning tools in the sources.
So it's all in the repository.
So you're going to capture a bunch of pictures like my lovely gallery and store them in a directory.
The tricky parts are you have to make the file name say what the color temperature.
Of the light was to capture that file and the light's level.
So you have to literally put the metadata into the file.
Maybe we want to handle that differently later.
But it means that you've got to every time I take a picture, I have to manually go through and rename the files based on what I capture there.
That's a bit of a tricky process at the moment.
I'd love to see all this automated with some sort of help at all with our camera application,
where we can just take the picture and give it the data.
But it should be easy.
There's not much to it other than give it a directory of pictures and run the tool.
And that's going to produce you a tuning folder ammo, which you can then put on your device and get better pictures.
For me, the Gospel, the Bible, to read is written by David Plamen at Raspberry Pi.
I don't think any of the work we could have been possible without his expertise.
And he's got an amazing write-up, which really goes into detail about how to understand what's going on here.
My colleague Steffen is working on extending the Lip Camera documentation to platforms other than Raspberry Pi.
And of course, we're looking at the problem from how do we make this generic for all platforms, not just the one.
So Steffen hosts it at the moment itself, but I see that move to the Lip Camera all soon.
Okay, come back to the problem segments and issues.
We start with bare images, we've heard how they don't look how we want them to see, so we need to make some corrections.
Tuning these devices can be quite hard.
There is a bit of built up knowledge that you get just by doing it.
I'll say later, but even the power line flicker, when I take some of the shots for the lens shading,
you can see the 60Hz signal oscillates in the images.
I hope we'll build up as a community the knowledge of how we handle all those situations.
And probably the most awkward part is that you need equipment.
You need one of these, you need light, you need to be able to measure things.
But I'm not worried about any of that, I'm worried about this.
You guys are doing a great job in enabling an enormous amount of devices.
There's more mobile phones, cameras, laptops, tablets, and every other thing that can now capture from a camera.
And we need to calibrate them all.
And there's not enough of me.
I need minions.
So down the left, do not treat these numbers as real, but I'm trying to say that there's a scale here.
As a community, on the device we've got, we don't need 100% quality image shading.
What we need is to get away from zero percent that we have right now.
And the bare minimum that I think is needed is a color emitter.
Something to measure the lights that you know where you are and the color chart.
So that comes in about 130 pounds, which is even itself is a bit awkward.
No one wants to buy a phone and they say, oh, by the way, you've never spend 130 pounds to make the camera work.
But we don't need everyone to have one of these.
We just need some people to have them.
The snuggle is on an express.
Now, it has an accuracy of plus or minus 5%.
Which you've got a 10% range here.
It's 50 pounds.
The real ones of these per like 2000 pounds.
It's the cheaper version.
That's what I'm trying to solve here.
I want this to be easily obtainable equipment.
I don't think anyone should cheap out on the color chart.
There are variants of this with different colors.
Then we need to change the tools to know the different actual colors.
But that's just so pervasive and easy to get hold of.
It's just easy.
If you want to go beyond just capturing a few pictures in various scenes.
What we're doing in our team is building our own light boxes.
So because we need to have a bit more control over the environment.
And we buy box from IKEA.
That gives us a space.
Then we use stage lights because you can accurately control the lights.
And again, when you've got something that can measure what the lights are putting.
Give us a good starting point.
A flat panel LED, almost like the ones in the ceiling here,
are great for the lens shading.
You need to be able to have something bright, flat.
Ideally, you can change the color temperature.
And again, on Amazon, they'll express you to get one that can change the color temperature.
They're usually quite cheap.
At home, I've got a box that costs me 800 pounds.
That's if you want to go further.
It's big, it's bulky.
No one wants to buy that.
And of course, if you're really going for full commercial here,
then you're going to be spending hundreds of thousand pounds.
That's not what we're trying to achieve here.
This is kind of my next problem.
As I capture more pictures and more data, it grows.
And when I finally achieve finding lots of minions to capture all these pictures for me as well,
we're going to have 500 megabytes per sensor-ish.
So if there's three cameras on a phone, we're going to have up to two gigabytes per device.
And I'm already panicking that there's 100 devices or more that we're going to be supporting.
So we're going to have lots of data.
And we need to store that somewhere, and I want to open data,
and I want it shareable so that anyone can work with it.
And...
Did I miss bit? Sorry.
Yeah, so more than that, there's one of me or very small team with lots of devices.
And if you were Qualcomm making a real phone,
you're going to have 100 people per device,
whereas we're in the opposite spectrum.
We want to support every device with low resources.
So I really want to make as much of this as open.
I want all the infrastructure to be open.
I want all the data to be reusable.
And actually, that's really important to me because I don't want...
What do we capture this data?
We're still working on the tuning tools.
So in six months, we might have the tuning process improved.
We can't go out and capture the images again on 100 devices.
We need to store that, save it and reuse it.
The data in the picture is not going to change.
We need to just make it reusable.
And ideally as possible, I want it to be easy.
So, yeah, there's some tricky parts.
I did want to highlight that if you've got a calibrated light box,
the snarkle is going to disagree.
I use both at the moment.
I've got the calibration from my box,
but I am actually currently using what the snarkle replies.
What it tells me.
I want to evaluate how that will work for the devices.
It can be fiddly, but that's just the mechanics.
We can learn that, we can build on that.
And as long as that cold shots visible,
even the environment shouldn't matter.
We want lots of scene and environment in there to have realistic environments.
Realistic shots.
I'm going to not dwell on that.
One important thing I wrote here,
don't capture the light in your shots.
You don't want to have the illuminant in the picture you take.
That's really important.
You only want the reflection.
If you get light in the shot, it's going to saturate.
And that's not what we want.
I actually do most of my tuning at night.
I go into the office with all lights off.
And then I've only got one light.
I don't worry about the outside sunlight or anything like that.
So,
discussing this with a few people,
it was like, how do we get started, where do we put the data?
As a developer, I'm used to get.
I've got control of the camera namespace on free desktop.
So, I've made a get repository.
So, the tunings I've got are up there.
And the tuning repository.
Which is a starting point.
It's easy for me.
I can just commit photos to it on a branch.
I'm using GetLFS, which means that if you clone the repository,
you get main and it's empty.
So, you're not going to clone all this data.
But if you check out onto a branch,
like the Fairphone 5, GetLFS will only copy and download the artifacts
that are used on that branch.
So, that's point of a good starting point.
But it still doesn't scale to 100 devices and hundreds of gigabytes of data.
I can't imagine wanting to fork that to make a pull request.
And I can't see how it works.
So, I'm kind of at the point of trying to work out how to do it next.
And this is where I want help.
I'm not saying this is the answer.
But if we have something like Google photos,
where you put data into a bucket and a web interface,
maybe that's a solution.
So, I've come across an open source Google photos solution called Image.
I'm not saying this is what we need to do,
but it's like this is possibly a way we could do it.
If we can take existing tools, even, and extend that.
That's what we're working with.
We're working with photo albums.
We want to take a collection of photos,
as grouped together, but then be able to choose which ones we like
and add metadata to these images.
We're already handling the back end of how to turn these images into
the tuning data.
But as the live camera team,
we don't have the expertise to do these bits.
I could imagine something that in that
friend's hand might let you change the parameters in real time.
Now that we've got the software SP and GPU,
I could imagine that could even be built in.
That's low.
I could imagine a system where once you open this album,
you choose the picture you want to investigate.
It might load up a page and then give you some sliders and parameters,
so that you can fix things or adjust it,
and then feed that back into that album's metadata
to fix the tuning for that device.
What I'm looking for here is how can we get other people
able to say, I've got this device.
It's not looking right.
Let me fix it.
Not have to.
Sorry, let the person fix it.
Not me.
So that we can open up that process of making the pictures look good.
One thing I think would be really important there is that
there's this magic term for delta E.
It's the difference of what we expect that color to look like on the chart.
It's numerically what it actually is and you sum them up and take an average.
We want that number to be less than 5.
If it's 10, it's visible if it's 5.
It's not so perceivable by the human eye.
So we've got an actual scientific metric that we can say,
these parameters are better in these conditions.
So we've got something we can build into the system to say,
this is better, this is worse.
And if that's if you have people to help,
now that we've got these devices,
take pictures.
If we can get people to get hold of the
the color charts and the color remissor,
not everyone has to have those.
Maybe you've got a group of friends and you share
one set or move them around or we might be able to post those
and get captures that way.
But ideally, it has to be a raw picture.
We've got to have that raw data out.
And what I've been doing so far is taking a raw
and a jpeg on an Android device.
So that kind of actually I'm cheating.
I'm saying, what's this image look like?
What does the real stack think it should look like?
So I can kind of have like a target.
And I think if we can do that as well, that would be helpful.
If anyone wants to contribute, as I said,
there's lots of things that can be done here.
I need help with how to restore it, how to share it,
how to build this interface.
Everything here is like, I don't know the answers.
Yeah, I'm looking for support.
I'm very out of time.
I want to give a shout.
Slide out.
This stack is open source for slides and I've loved it.
It's brilliant.
My travel is sponsored by Adios and Board.
And this is all about the lib camera project.
If you want to get involved, come to the IRC or
Matrix channels, mail the list or find us.
Hopefully there's enough of us that stand out in her
days now.
Thank you very much.
Thank you.