Yeah, alright. So, welcome everybody. Right. Let's get down to it. Thank you all for letting
me open this deaf room for giving me the chance to present this interesting topic. I just
realized that this year is going to be my 20th anniversary of using Linux. I'm full-time
kernel developer since 2020. I've joined Kalabra in 25. And I mostly hang around in Linux,
media, and Linux, Rockship, and in that capacity I am a charge of bringing the video capture
and camera support mainline. So, this is the goal. Right. And this is for recent Rockship
SOCs. So, by that I mean something RK 35, something something. And if it comes to camera and video
capture we can group those into two roughly because they somewhat share the same architecture. And that's
the 66 and the 68. And then there's the 88 and the 76 and the 62 and whatnot in the other group.
So, just as a bit of an info there. Now, we do capture on camera. First of all, it's amazing. Right.
Then it's pretty much about capturing video frames from some companion chip to memory.
However, a camera does require more. So, the companionship could be some
HDMI to mepices, I convert it, but also it could be an image sensor of course, and it delivers
raw frames in the sense that you need to do debaireing and then it's hard to green. So, you need to
do corrections and all the valve balance and all those things. At some point you'll need to
eliminate it. You may have some optics and yeah please like this. So, I won't be covering this too
much because they're interesting talks tomorrow, so make sure not to miss those.
I don't feel it's required to say why we want to use upstream here or why we want to go main line.
Rockship SOCs are already well supported in main line Linux.
Yeah, woo for that one. So, my colleague Nikola will be giving a kind of an overview of what goes
and what happened last year, also tomorrow. So, if you're into Rockship SOCs in general, make sure
to drop by. Yeah, and we do capture on camera is a bit of a gap here and the game here is to close
this gap right. I've given a similar talk recently in Amsterdam last August, so we'll be covering
what changed since then and so that the goal here is just to give you a bit of an update. Still,
first we're going to look at the hardware roughly and then give the overview of the current state of
the art and then let's see what comes next. So, I said, well, okay, there's roughly two groups and
the first group here, there's the video capture you didn't and the iSpeed image signal processor
and they work mostly parallel, right? So, they share the the phi, this phi is special because
you can share it between the two or use it for one of them. So, this is configurable.
And then, of course, you've got a ton of DMA engines and converters and ESpeed iSpeed cores just
well, all the processing blocks and lots of things in it. So, again, the spit mode is something
I'm going to mention quite often because this is well, something we need to take care of
soonish because it blocks things and yeah, okay, the iSpees have different versions I'm not going to
into. Keep in mind that the iSpeed has different operating modes so you could use them in
in line where you take the image from the sensor and pass it to the iSpeed or you can buffer it into
memory and then use it like in memory to memory operating mode and yeah, there are use cases with
raw capture and HDR and but yeah, this is a bit beyond this code right now. Of course, there is a
receiver that like me PCSI is used for camera sensors often and it features some virtual channels
so you can have different streams over the same CSI link. So, this is also some interesting bit.
The other group features an architecture like this where there is first the recap that is the
interface and then the iSpees are sub well at the next stage of it and you can do raw capture
with the recap itself and it features a huge crossbar max and then you can select certain streams
to go to the iSpeed and some to memory and so well whole lot of possibilities.
Again, we have different numbers of different files and different iSpees over all those variants
the iP cores are roughly mostly the same but not exactly so sometimes it's easy to port
them from one SOC to the other and sometimes it's more difficult so there's a bit of a variation there.
So the recap again this max 2 iSpeed block this is something we'll need to take care of
because there's enables the usage of the iSpeed in line mode and this should be one of the
most frequent use cases i'd say. Right so in comparison to Amsterdam in August
this CSI defy was merged after a bit of well some i think it went up to reform or something
this split mode is something i mentioned and so this is new because i didn't realize that we
really need to take care of it it has some implications on how things are exposed to user space
how they're exposed to the device tree so how do the bindings look like and things so this
is what something we should be looking at but there is initial work going on i heard from
i user that he was at least able to to bring up say the other pair of lanes and so
yeah things will look good not just say not promising anything but i indicate that.
Right um yeah the dc5 is on the more modern rpc bases this needs some loving care
the receiver was renamed a few times but now finally accepted woohoo and with that we can go forward
and this should be easy to port on say if you have 75-76 or 62 machine so this should be easy
i hope um then the the mp and the dvp capture so the dvp means digital video port it's just a
parallel interface so this went mainline recently so it's just the wall initial driver comes with
all that and was a bit of a relief that this was finally accepted yeah and basing on this there are
first efforts towards mp capture on the 3588 and also on say the towards this max to i's bbit
and well i can show you a picture of the mp capture thing so this is just a rock 5b plus a
raxa camera and it's just capturing to memory and then using g streamer by air to rgb
to convert to do the demosic and it's awfully green and noisy and whatnot but well you can
recognize me so first success you can right well anyway so okay um however well we kind of stop here right
so um yeah and here's nothing really easy so um i did start so i was here two years ago and promised
that i'll be mainlining patches for the isp on the 66 and 68 i never managed to do so so sorry about that at this
point sorry say laurel sorry for me anyway right now there is focus on the 3588 because
this is like the flagship right and so um and the good news is that there are different
parties working on the isp support so there's us there is ideas on board of course and there is also
rock chip doing some bits and i hope they really so this really takes off and we've got the
contributing here so in terms when it comes to SOC camera SOC vendors rock chip
is somewhat one of the best right so it's still not good but there's somewhat the best so just the
others are way more worse anyway it's just about this closing having their openness right
so rock chip is some some company you can actually approach and you get the information at least
the register contents to some degree it's not easy but it's at least possible right so
and yeah so i'm really looking forward how this develops so right so isp support is somewhat
in the making but it will be some churnian till we get there and then of course we'll need
flip camera support right and looking at you guys see
hi in which SOC rock chip is contributing there's a way like so there's an initial
driver for the RP 3588 isp from them but it's just basically throwing something over the fence
and then let's see what comes around but we'll try to get them i think most work will be done
on the 3588 and then we can see what it of course we'll try to make sure that the 76
in the 62 can be supported at some point the ice piece different but the architecture is the same
overall so i think we can share some code there and yeah right so what's next
yeah there are certain things to discuss mostly about how to expose hardware using
video for Linux to the user space i mentioned the split mode quite a bit it's about okay
how does the media graph look like should there be two of them or one or whatever
it's about how to expose memory to memory mode and inline mode can use which between the two
in the 88 there is the chance of two ISB's working in parallel in unison and how can this be
activated if need be um we definitely looks already offers the streams API which is a good match for
the mep virtual channels i mentioned and then there is multi context for the time division
multiplex because you can have like up to six or seven sensors on the 88 but you have only
have two ISP so you need to do some multiplexing here and this leads then to the question again
together with memory to memory operation do we need a scheduler here and so there are certain
things to discuss and well this may take a while for now i'm going to continue my work
regarding the 88 weakup in particular the max unit and then yeah a preliminary
initial ISB driver that does the inline processing right so still a green picture but at least
debaired this will be the first goal here what you can do is just test all this as far as it
possible and well with different setups different use cases feel free to reach out and just in case
well you want to have an ISP there is a software ISB lip camera now lip camera 07 0 right
it's called Brussels for a reason and right and there will be an interesting talk about that one too
so make sure not to miss this if you're looking for a new challenge we are hiring and
for the time being let me thank you all for listening and i'm open for your questions now
ah sure yet go ahead
yes um so it's the question is about capturing HDMI
yeah um okay about capturing HDMI audio as far as i'm or as far as i know it's
something there's a companionship and in the this driver you will register some also device
as a sock something so this is at least the capture i see's i know
if this but let's have a chat afterwards so uh you mentioned that you are the fairing
basically support for statistics and parameters which usually is not the hardest part of supporting
now it's been because that's just basically about exposure actually i was going that for
question of time you don't have time for you missing the presentation for that
okay the question was about kind of short parameters statistics is it a time issue or a
documentation issue um pretty sure the documentation is there in the sense of code is documentation
we've got downstream uh the biggest downstream BSP and so we'll we'll extract it from there i'd say
i reckon i'm not too experienced when it really comes to parameters and stats i see this from an
abstract point of view and my job is to get a stream running and this is why i'd just accept that
debaring yes and then just some stream and the the what say to make it nice is the next step
done there was some what yeah for a question we're going to be uh downstream and rocks is
what so how much I guess in production of these accounts uh it was better if it was
how much I get it from the downstream is it's so how how how
how feature which is the downstream channel code so it was basically so that you can enable and
can most of all the ice cream okay so the question was about the downstream BSP in terms of features right
or or uh you can get more from the raw chip themselves as adults uh so the downstream BSP i must have
met i tend to not look at it so i i hear yeah of course they implemented it on their basis so
it's pretty much every feature should be there right but i only use it just a reference and so yeah
what is the step to sort of a fixed 30 in main line for the video in the end of the video code
so the the question was the px 30 uh and how the status is uh i during this work i brought the
video input process of for the px 30 main line but i never really had this board on my desk or this
chip on my desk and it was just a continuation work so i wouldn't know but there is the rock
chip maintainer here somewhere i think two rows behind you and uh
all of them are not high at least just like what you put three nine nine and if you put it
or like like you can just put the camera on the whole thing and you will get the real
nice and much other thank you so just for the record uh after the lecture way
for the recording so high co claims that is covered well by the RKI SP1 driver and Kira
mentions it should be feasible with a bit of tuning uh filling around don't expect
perfect images out of box but you can definitely get there okay not perfect but some images
and this is a good start at say right okay and thanks again y'all for listening