Okay, can you hit me?
Okay, thank you.
So hello, I'm Paval Mahek and I'm here to talk about phones and cameras.
For the things that something about me, you have an email address up there, you have my
favorite account.
When I'm not hacking phones, I'm usually playing with horses and I have good smartwatch, which
shows time, which is not usual for a watch.
So if you want a watch that shows time, then you can take a look.
I'm playing with phones.
I like small ones, so this one actually runs post-marketers and I'm trying to use
it as a daily driver that will bite me, but whatever.
So I can not hit him when I try to make money, and that's about it, I guess.
So phones usually have dump sensors, which is a good thing.
Smart sensors are in NARPC, but we won't talk about this, because we don't have too much time.
And we usually don't have useful ISP image signal processor, because we don't have drivers.
Sometimes you usually do have ISP, but we don't can't use it.
And camera is basically mandatory for all users, everybody needs camera.
So what do we need to do?
First thing we need to do is auto exposure, because if you don't do auto exposure, you have
completely break frame, completely white frame, not useful.
Auto exposure is for translate white easy.
You can get something quite simple, like if it's too wide, you just turn down the time,
and you will get something working.
Auto focus is harder, because most cameras need auto focus, and auto focus is actually
very tight and it's quite hard to do well.
And I'm currently not aware of any good auto focus implementation.
I try to write few, maybe you can try them, and make a picture so that needs more work.
Next stuff.
So camera vendors like to like to like to you a lot.
So if they speak about pixels, they mean stop pixels basically.
And if they speak about bits, they mean something else, then display means.
So you need conversion between this, and this is called debiring, and then
cork, you multiply with color matrix, and do bunch of other steps.
I won't go into too much details there, but it's computationally quite intensive.
And then they are some like optional steps, like lens shading compensation,
because the camera optics is quite bad too in the phones.
And if you compensate for that badness, you get better picture.
Then if you want, you can do denoising, sensor, yeah.
If you have sensor on your screen, if you had like 100 bit pixels, you would be mad.
If you have 100 bit pixels on your phone, sensor, that's pretty much normal.
So basically they are all bad, they are like there's different levels of bad, and software is supposed to deliver that.
They don't need to do, well, you've conversion.
Basically RGB is bad for JPX and for video encoding, moving encoding.
So you want to get to other cork space, and then you need to encoding.
You often have hardware blocks to do that, but you don't have drivers, so you want to use it to do it on CPU.
Working design.
If you want something working, you want to keep the sensor in mid to high resolution,
because that's what enables you to do phase detection autofocus, for example,
and you get all the data.
In the mid resolution, you can still get good enough frame rate, and you can debyre and downscaled on GPU.
GPUs is fast enough to do this.
That's an advantage because if you want to take a photo, you just the preview is running.
GPU is doing conversion from the robot data to your display,
but if you want to take a photo, you just copy the buffer, and then process the slider.
So you just save raw.
On the movie, you really want your GPU to convert to a UI format, and you can pass the 2G streamer.
G streamer is great, quite fast, actually.
It seems to be faster than the beginning itself, and if this, you will be able to get, like,
go to 8 megapixels at 30FPS from the dream 5, and the dream 5 is basically the worst case phone.
Your phone probably is better than that.
The dream 5 is a break, it's a well supported break, but it's so, but if we design for that one, everything else will probably just work too.
So, I discovered implementation and here it is, I call it click machine.
I'm not the only one who worked on that, I just basically asked for help on social media.
Hey, I need to do this, it should be easy, but it's not easy for me.
Help me, and people did, and the robots did, and if you can see the results as a good lap,
so you wish to be able to search for that, it's part of the 2G project.
I edit phase detection of the focus, which is nice for robustness and so on, it needs more tuning.
Design if click machine, if you have a 3D project, you wouldn't know if click machines is movies first,
and movies first is actually good design because still images are easier and so on, so that's what we did.
It's quite liberant for specific, unfortunately.
So, big sense goes to Dos, mentioned here, and his tourism employee, and he helped a lot of the work,
but in the end, the single liberant for specific, but it's good demonstration that this thing is possible.
I tried to port it to something else, I wasn't too successful, if I knew more GPU programming,
I should probably be able to do it, help will come.
It's 8 bits specific, because liberant for currently only has 8 bit drivers for the camera.
You basically won't turn out 12 bit for good images.
There are patches or dead, but they hit the kernel to, so there's more work to be done.
Signal processing that we do debire and downscale at the same time, which is great,
because it means that we have better colors, because we don't have to interpolate,
and it actually makes stuff easier because we can take the photos at the same time and so on, so that's good thing.
It's like a few hundred coats, lines of coats, it's not like crazy complex, and it works, so I'm happy with that.
It can do certain FPS in, like, don't hit megapixels, so video recording with sound via this streamer.
So, that's something I believe is good.
One second or not, liberant five has a hardware back, and it's known if you overload the memory while the camera is working, it will just hang.
So, I don't know solution, I don't believe anybody knows the solution, solution may be getting you hardware,
and that is the solution long term, but if you want to see, like, pretty complete implementation that means tuning and a lot of tuning,
that's click machines, and it's available on GitLab.
Next project I should mention is megapixels, megapixels is way more nature than click machine,
but it's not designed for movies, so without a design, it won't be able to record long movies.
I don't think it's going to be fixed.
It's GTK based, so it has, like, quite nice interface and so on, but there's a triple GTK,
and GTK new GTK requires new hardware, but we only have all hardware on phones.
So, Samsung will eventually need it to be done with that.
Probably not by me, but hopefully someone could ever think of Samsung.
And then there's a future, right? Future is the camera. Everybody knows future is in the camera,
because that's where the money are, it's same for the PC, sent phones and so on,
but it's also a lot of work, and it's currently missing some quite important pieces.
So on GitLab, in the camera repository, I have a modifth fork of the camera,
which contains autofocus from NekoCVD, like, Samsung's to hand,
that autofocus needs a lot of work, but it's important to have the component first.
Lip camera needs a lot of work, as a whole, it's moving, thinking.
It's supposed to be suitable for any hardware, so work is progressing more slowly than it should.
Then you need some kind of client, we've been coming up to take photos, right?
And you can take some kind of cheese, but cheese is not the real camera application suitable for phone.
You would like to, for example, take the high resolution picture pictures,
and you would like to select time for the exposure and so on, manually and so on.
And I'm not missing the cheese, it's suitable for, like, this kind of debugging, so I started my own.
It's called, I called it Millicum, and it's in the repository,
set for the phone name and come.CPT.
It uses SDL because SDL is nice, and SDL doesn't bite unlike GTK,
which means the user interface won't be that pretty, but this is still prototyping, right?
Lip camera made great progress in last, I believe, 14 days, because GPU ISP processing was finally managed,
after a few months of, like, maybe half a year or more of efforts,
thanks a lot to people that made it happen.
And that's basically means that the camera is starting to be usable.
I'm not saying it's usable for photos now, but it's moving in the red direction,
and at this moment most of the code is there.
What we are still missing, it can do the buyer on the GPU,
and it can do color matrix multiplication on the GPU,
which are two heavy steps.
I don't believe it can do one shading at the moment,
and I don't believe it can do I-O-E.
It's just like, I'm thinking maybe just for all the shading.
Okay, so as you see, the camera is moving quickly, so we have patch on the mailing list,
but let camera is so out of focus was on the mailing list too,
year ago, and it's still on there, so it's hard.
And as a single, I would really like to see, is buyer the processing,
which means if I already have my photo in the NG,
I still need to want to pass it really ISP one more time
to get it into the high resolution, nice photo.
That is for that are also there from a very, very people,
but that's another major task waiting for us.
So a little camera will be great in future.
It starts to be useful, but it's like not yet, not yet there.
So, while developing all this,
I realized that I need a way to see the images basically.
And after some research, I realized that we're like,
current data formats are not suitable, so I created my own.
This is supposed to be basically what the NG should be,
but the NG is very complex, stiff-based format,
and it can't restore everything we would like to store,
so the NG is non-starter.
Basically what this format does is there's fixed size header,
header contains width, width, image width, image height,
and for CC code.
And timestamp and stride and so on,
but the idea is that if you have the data from the sensor,
you don't say with raw, you just append this easy header,
and it means I have the other tools that can work with this.
For example, I can divide it, I can display it.
I can take a bunch of four CC images
and to move it for testing purposes and so on.
I'm going to happy about this.
I believe it helped my development a lot,
so if you find that you need a simple format
for your global camera application,
four CC might be for you.
Of course, if you are doing professional photography
with the graphics, that needs DNG output,
but there's quite a way from raw,
raw picture up to the DNG or JPEC.
And it can move conversion to PNM,
portable network map.
It can do conversion to JPEC and so on.
I'm starting to write simple tools,
maybe some are Python,
some are an instant error on C,
but it's important to see at least some sync.
Okay, about hardware.
There are two phones that should be mentioned.
One of them is the brain five.
The brain five is the board I was talking about.
If you want to take photos with your phone,
this is the hardware to work with.
It's not cheap, it's not nice, it's brick.
It doesn't have a good battery life,
but it's graded for us and created this,
and it's quite a major.
The camera work there two years ago,
and it still works there.
If you want something closer to real phone,
then maybe one plus six is for you.
With very lightest code,
we have camera sensors working in post-marketers,
and it has autofocus tips and so on.
It has, it's like, not this tick, it's thinner.
But the general advantage is that the support
is getting there and people are working hard on it.
Okay,
so I believe this is for it.
I don't think we could get something like
camera showing on this thing.
So if you want to see,
I have all three applications running here hopefully.
I can show you after the talk ends.
And this is thank you and time for questions.
Thank you.
Or comments or anything else.
So if there are no questions, then I get one.
I don't know.
I don't know if there's a phone from the previous talk.
Okay.
Yeah, so question was why not the telephone?
The telephone is different generation, basically.
So Libre and five is like three for it.
So at the moment, and the support is there.
The telephone is more like one plus six.
It's Qualcomm's thing.
And basically, what one plus six and the telephone,
I believe they share the SOC,
which means that we're done on the telephone.
It's applicable to one plus six and so on.
This is the generation.
We will get there.
But if you want easy start, it's still within five.
And I mark questions.
Okay, so thank you by the way, health wanted.
Yes, so question was, if my projects utilize hardware acceleration,
the utilized GPU for that.
Because basically the drivers for ISP, I'm not AI.
And GPU is fast enough to do that.
So modern phones have ISP block and JPEC and coding block and movie and coding
block.
But I'm not aware of any phone that the hardware drivers would be available.
And that means that I'm using GPU.
Okay, ping-pong problem is, so I didn't mention ping-pong problem.
Maybe I should have.
It's a phone I have had good things about.
But I don't have, like, I don't have one.
If I found one on my doorsteps, I could probably play with that.
But it should be mentioned.
One more thing, there's ping-pong pro.
It's the dump camera and useful camera development.
And then there's good old ping-pong, no pro.
And it has a smart camera, which basically means it's like PC and everything works.
So if you have a phone which can take pictures that come quality,
you might take a liquid fine ping-pong.
Maybe for some kind of application development.
But it's really, really different hardware from everything else.
But ping-pong pro would be very suitable to play with.
Okay, it's black magic, basically.
It's low-level black magic.
I have some of the function names are the character's long.
And I don't know what is it.
I believe it's HGL towards some sync.
It was major blocking point because, apparently, if we have a GL version 3,
we would be able to work without crazy hex.
But we are limited to a GL2, which means the GPU can do it.
But we are doing some quite crazy hex.
Take a look at the product machines.
It's white black magic at places.
Okay, so if there are no other questions, then thank you.
And if you want to see it in work.