Yes, I started this library that's called the Libs of Cura to support cameras on Linux
because cameras are difficult and they should not be difficult, they should be rather easy.
And yeah, why am I here? Because I want to do to continue talking about this.
Well, the first presentation I gave was at 38C3 at the Congress. How many of you have been at the Congress?
Hands up and up? Well, there are some people. Yeah, so I'm going to go a bit deeper than that.
But since you haven't been a doesn't really matter, but the story started with the Librim 5.
Oh, I forgot mine, but how many of you have lived in 5? Oh, I'm so proud.
When I was working in Librim 5, we had to support the cameras at some point.
And this is where the story started. I had to write some kind of a library for the cameras.
And it turns out that it's kind of difficult because of not using USB.
Like, you have a webcam, you have a webcam in your laptop.
We have a different kind of a camera on your mobile phone and like,
webcam's run over USB. Typically, most cameras in most laptops run over USB.
And USB is quite amazing because it does everything for you on camera and cameras with the inside the phone.
You want to have more control because the person who is using the camera on the phone,
like, probably is the first private camera that they are using.
And they want really good quality. They want everything to be done properly and powerful.
And like, not just built into the tiny controller in the camera,
but actually running on the main CPU with like all the computational photography staff and so on.
And this has become a problem in 2025 for Linux because Linux phones are becoming popular.
So it's a problem that's a sign of actually making it.
And yeah, USB cameras, it's all due to the air quite nice.
Everything is inside. You see all those, all those elements.
There's like a sensor scalers that sticks wide, but as the buyer so on.
So on those are not optional, except for maybe this JPEG encoding.
So they're really nice. You just ask for a picture and you get a picture.
Even get a native JPEG for a classic. This is amazing.
Meanwhile, on the Libre5, you don't have that.
You don't have the components which are required to build a camera.
Because they are not part of the standard, which is not used to be standard.
Like, there is no standard actually that much.
There is like a statistics unit, but it is supported in hardware.
It is not supported in the driver. I was able to make it work.
It's kind of, and yeah.
So is USB really that different from like,
where people not using USB on the phones instead?
Maybe like, there is enough support to provide drop pictures, not just JPEGs.
There is enough support to create your noise, reduction algorithm.
And maybe there is some place for computational photography.
But why are they not using it?
I think it might have something to do with complexity of the USB protocol itself.
Or maybe the USB protocol that controls the camera is called UVC.
Maybe it's just not powerful enough regardless.
It can do everything, but maybe it's not powerful enough.
As you can see, this is like the device diagram of USB device.
And the frames come from the top from the sensor all the way down to the bottom
through all the processing notes.
But this is quite simple. It just goes, but put, put, put, put, put, put, put, done.
And the images down and the end. But on an Android phone, the diagram can look like this.
In USB, I have not found a way to actually do some extra routing to the side and back.
And so this might be one of the reasons why USB didn't really catch up in more advanced photography.
There are also problems where you produce several pictures at the same time.
Basically, this is one of the few laptops that doesn't use USB.
And this is a complicated pipeline that you have to support in any library.
Only by 5, it looks kind of like this again. The problem is, for all those different diagrams,
you have to, if you bring a camera library, that's just does the right thing, which makes it simple.
The only, if you want to make it simple, it has to do the right thing basically.
For this set of different devices, you have like one device here, another device here, one more device here.
You have to configure them all in such a way that the frame from the top is in a format compatible
with the output from the top, the input on the next level.
And then it gets transformed. And then the output on the bottom has to be the same as the input from the top of the next device.
So, yeah, this is the pipeline. And let's take it a bit complicated.
This pipeline creates an image in the raw format on the top.
Then there's some kind of a processing device. And this processing device,
combines the image maybe, and it creates two images. One image for saving as a photo,
when you click the shutter, but another image so you can preview the photo before actually committing to it.
So, let's imagine what that looks like.
So, for example, for the sensor, it's quite easy. You only have output.
Well, it can probably include some scalar inside, whatever. It can have two outputs. One thousand pixels by 500 pixels, another 2000 pixels.
And there, the processing node, which takes an image of any size in a particular format,
and outputs two different images in two different formats. So, okay.
You have two different notations here. And here is a notation with no variables.
And there is like a variable here. Look at that. There's an A. There's a B.
And there's a half an A. And there's half a B. So, quick exercise in outputs two over here,
which possible sizes could we get? Come on. Come on. There are multiple options.
There are multiple answers. Okay. I'm going to tell this for you. If you have an input
of like A by B. And the output is A half B half. So, you can get from this option.
From this output from the sensor, you get either, yeah, you get a 1000. Wait, no.
500 by 250 or from this option, you get 1000 by 500. Right? So, this is like a kind of a pipeline.
And it has a sensor and inputs, a certain outputs. And basically, if you want to
use a camera that kind of passes the frames from top to bottom, your library has to
compute everything. So, that is matching. Yeah. So, oh, welcome to the Sudoku conference.
Have you had previous opinions with Sudoku? Hands up, hands up? Yes. Do you enjoy Sudoku?
Oh, there are people who enjoy Sudoku, even. What is Sudoku? Sudoku is a kind of a game where
you feel in numbers based on other numbers. And those are the three principles of Sudoku.
The first principle is that every row has to have distinct numbers from one to nine.
As you can see, all numbers are distinct. And also, the same applies to the columns.
All numbers and every column in column A and column B and in column C. Every number is distinct.
And the final property is within every square nine by nine. The numbers have to be distinct.
Okay, there's a wrong idea. And does it look like something? I think it looks like something.
It looks like source code. It looks like a solver. It looks like
analog. It is actually, we already, we actually just now created a solver. Okay, well,
I'm simplifying it a little bit. There needs to be some extra supporting code, but it doesn't do anything
magical. We create a solver, a Sudoku solver. Oh, I got. And if we try to use that solver,
it's actually really simple. You can try this procedure yourself. Just stand as QR code
go to the URL. You give it a table. And it's solving the constraints that we presented previously.
Those constraints, there are going to get solved if you just give a table for this kind of a
code. It's going to basically fill in the spaces, which are signified here by underscore.
And this is just a fully declarative way of doing things. What we did when we described
the constraint that the present and the everything else is solved by the computer.
Does this look familiar? We have a kind of a constraint here. The constraint on the sensor
is that it outputs certain resolutions. The constraint on processing units is that it expects certain
resolution, and it produces a certain different resolution. So maybe you can write it like this.
A sensor takes a sensor defines a valid relationship of width, height, and format. And there's
basically the same thing as we already looked at. And the processor creates a different relationship.
It's a relationship between the input frame and output frames. And the relationship is that
one of the input frames, the big one, is the exact same resolution with just a different format.
And for the preview, it's half a resolution and still a different format. So if you create a little
bit of a solver with those constraints, let me explain that. The solver gets a query which defines
for a sensor that behaves like this. It takes some kind of input values. And then we have a processor
that takes the same input values. And returns a frame that is 1,000 pixels width.
What kind of values are there that we expect from the sensor? Basically, kind of a query. You could
do the same thing with SQL, but it wouldn't be so nice. So it wouldn't in a production system look
like that. There's actually, if you remember, there was an entire tree of different nodes in a
real device. So this is a very simplified version of that. But if you traverse the entire tree,
this is probably something that you can do in SQL very nicely. So if you have a solver like that,
you can't support any arbitrary device with any arbitrary configuration.
And that's why the kernel exposes this information easily to the user space so that the libraries
can actually do this. And now it doesn't. So this is like used to me, I kind of knew that,
but I realized that last week again, when I was trying to add support to this laptop so that I could
make a presentation demo. So there's one problem. There are calls which actually, whenever you set
an output or an input on a device and please use this resolution and then the device would tell you
which resolution. Actually, no, everything depends on everything. If you set a resolution here,
there are possible resolutions somewhere else changes. The format changes. You can't make sense
out of it. But even not that. I wanted to go into detail on this. There is not even a consensus
in the kernel spaces. Whether some resolutions should tell you which devices should tell you
which resolution is the support. So like this has strong consequences. You can't write a driver
in user space without knowing everything about your devices. So the driver might work because
like, yeah, the devices have some commonalities, but the driver will not use the devices fully.
And that also means that if you have a new design, a new set of devices, you must upgrade the user
space software. It will otherwise not work perfectly. You might get like it because like I'm
not going to get into details. It kind of kind of, to sort of work. But in the end, you basically
have to write this kind of constraints that we already wrote before. You have to write those constraints
for every device that you want to support in order to support it properly. So every library has to
have something like that. But one of the principles that I want to follow would live off
purest is to make things simple. And one of the things that are not simple is finding memory
problems. Debugging them is especially not simple. So I am using grass for this library.
And grass is this nice properly that's called borrowing. This is like the prototypical interface
for taking frames. Let's have like a stream of frames coming from the camera and ask the
stream for a next frame. This is like the basic loop for doing anything you want. And then you process
the buffer, you take the next frame, the next frame, back and loop. So let's try to break this.
Let's try to take a frame out of this without putting it back. And then try to take the next frame.
This is actually clever. This is the reason that I'm using grass. If you try to take this frame away,
without if you try to take a new frame, without putting the old frame back, you're going to get an
error. Look at that. This example basically shows you, you take one frame over here.
And then you're trying to get clever. You're trying to lose this memory here. You're trying to lose
this buffer. What happened here? You're trying to lose this buffer and process it somewhere else.
Like maybe you save this memory and you're trying to be sneaky, send it to another process or something.
And then you're trying to take the next frame. But, but, oh gosh, what happened here?
No! Come on.
But, well, I swear, computers. Computers are mistake. I sincerely believe computers are mistake.
So when you try to be clever and lose the first frame in order to get the next frame,
last you're going to tell you, no, you are not going to do that because you still haven't returned
the first frame. And, yeah, that's one of the reasons that I chose Rust, but there's another
interface that is kind of better and better and not if you really want to go crazy. So, that is the
past of the Live Obscura. In the future, I'm going to try to support more devices and make better
documentation because the documentation is also kind of part of simplicity and user friendliness.
And, yeah, the scripting engine that I showed you before the one in Prologue, I want to continue working
at it. Also, I want to thank to give thanks to prototype fund, which are funding my plan to take
over the world. So, thanks, questions.
Do you feel like from what you're learning by being a little obscure? Like, it's actually
a little bit better. Is it becoming a better library than the camera? I think it's like,
by the physical that it's not supporting as much hardware is definitely better. I think
I am trying to make it an experiment and figure out how things could be done to make it perfect,
or like, basically an experiment means I can fail. So, without like, bothering other people.
So, I'm just trying to go crazy and see maybe some of this craziness is actually really good.
So, was it tested on the Live Obscura or other phone? Actually, this library wasn't tested.
I started working on something else with Live Obscura and that works on the Live Obscura,
that's what we shipped. But this library is like a not quite a continuation. This one is a
a new project, compared to that. So, it has, I've been trying to make it work on the Live
Revive for the past two weeks or something, and I haven't done it yet. Not finished.
Okay, now the questions, let's move on from the internet. Will Live Obscura take any ISP colours?
ISP colours? Yes, in its signal. Will Live Obscura take on any ISP colour stuff? I'm not quite sure
what is meant by that. But there is like a library. ISP is image signal processor. So, I created
library that is called crispy and it does GPU image processing. So, it kind of does some of the
colour stuff and it does some of the format conversion stuff. So, maybe that answers a question.
GPU work is hard. Actually, on the Live Revive there's a specific problem where it only supports
OpenGLES20. And I think the last time this was mainstream was like iPhone 4 or something.
It's really difficult to find any resources to work with it.
Thanks for watching, and I'll see you in the next video.