This is the other end of the story. This is looking at an application that is consuming
a camera and the troubles we had with it. Have I got this switched on? Maybe.
Lights green. Yeah, so the troubles we had interfacing to the APIs and yeah, I have
a sub-confusion, which was that plan 9 had the right idea.
So yeah, I'm Tim Panton, I'm the CTO at PI.pgmbh pipe and we license a WebOTC stack for
small cameras and some of them move, some of them don't, that's a baby monitor, but some
of them move quite quickly. This is actually the subject of the talk, I don't know if you
can see it up in the top corner there, but it's a camera that's mounted in a race car over
the driver's shoulder so that the driver can be helped by the picker, so it's a real-time
video from behind the driver's shoulder looking out over the track, but also what the driver
was doing back to the picker. So we're interested in low latency, high quality video
over 5 or 10 kilometers, so we're not talking thousands of miles, but a distance, so we use
5g for that, but we're not, I'm not really going to talk about much of that, except that
that's the context of it, oh and by the way, that's what it looks like. Yeah, so I'm not really
going to talk about a lot of that, what I'm interested in talking about is the bit between
the camera and our software, how about interface works together. I mean, just to give you
the full picture, we have an arm, SPC sitting there, running Linux with our software, sitting there
doing the robot. You see piece and some smart stuff about bandwidth management and various other
security things and whatever else, monitoring some sensors and stuff like that, but it needs to
get the key thing is it needs to get the camera data. So it talks kind of about how we did that
and the history of how we've walked through that. So the first things are like a lab prototype
to see whether we could make this work at all and what we did, which is what you do with any video
projects like this, is you start off with Gstreamer. You put a Gstreamer pipeline together
and you kind of get something to vaguely work. And that's what we did. And so basically what we did
is we took the VFRL to Camp Source and encoded it to H264, Packadized it to RTP, all in Gstreamer,
and then sent that to those RTP packets to local hosts, which we then picked up in our web
HTC stack, did stuff, magic stuff with them and then sent them out over the wire. And this
actually worked, I mean it's simple, it's a really simple set up. And one of the nice things
about it is that the video bit is well isolated from our stuff in the web HTC stack. It works really
well in the lab, but the moment you get it out onto a real network with packet loss and variable
bit rates and stuff like that, it failed miserably. And basically what you want to be able
and in order to get anything you have to send keepsending keyframes, which reduces what you get
in terms of the performance. So I want to say a little bit about like isolation, what I mean
about the separation. So there's kind of essentially three sorts of isolation talking about there,
we're talking about process isolation. So is it running a separate process? Does my, in fact,
Java process? Does it have to care about what's happening in the video, low-level video APIs?
And it's really nice if they don't because then the threading gets a lot easier and stuff like that.
The second piece of isolation is about memory. Are they sharing memory? Is there a security
risk between like sending data between these two things? Is it something I need to worry about?
And then the third piece is a, is license isolation. And this is actually, in our particular case,
we have potentially have some proprietary algorithms sitting in the Java, doing stuff that
maybe the race manufacturers have told us about. And we can't leak that out into open source,
although we want as much of the stack as possible to be open source. So we need to be able to
understand what the license isolation is. And that's not always possible. So all of these things
are really desirable. So next step is, okay, so let's generate full frames, let's be able to control
generating full frames and control the bit rate. So we can manage the bit rate on the encoder.
So we went off and found a nice open source project for the Raspberry Pi, which managed the
camera and did all that. And it was, it's another G-streamer note. So we did that. We wrapped it up in
a pile of C so that we exposed those two pads up into a standard IO. And then we had the Java
exact that. And we got the ability to control the bit rate and to force a generation of full frame
directly from Java, which was, but keeping the isolation, which was lovely. And with a really
simple API, a really elegant API, which is just basically asking. And so we got, still got good
isolation. So the only downside is, it's a G-streamer pipeline. Now, I love G-streamer,
it's really useful, but it is a pipeline. So you end up with some latency due to the fact
you've got a bunch of frames in the different processes in the queue, in the different stages of
the pipeline. And the other downside of this design is that every packet has to loop up and down
through local host, through the kernel. And that's unnecessary and costly. So we wanted to
get the latency down, because these cars move pretty quickly, so it kind of matters. So we went
for what turns out, in my view, to be absolutely the pinnacle of what we've done with this,
which is to read H264 frames, direct from Davideo. So you can, with some trickery on a particular
version of Raspberry, and you can ask it to generate H264 from Davideo. And when you read a block,
do a read from it. You've got an entire frame from it, and you could just packetize that and send
it out. So we did that, and you got very nice low latency. You've got direct VFRL2 access to
changing the bit rate and forcing full frames. So this turned out to be a really easy way to do this.
And very, I mean, nicely low latency, and good isolation, because the definition is,
is like, it's fastest to max, that's basically, it's fastest to min exec. So we're not like,
we've got a really nice thing. However, it only works with the Broadcom blob, because that
tight integration between the encoder and the camera is in the Broadcom blob, and it's nowhere else.
And then the Raspberry Pi folks deprecate it in the next release. So that was out, which is a sad thing.
So now we get to the, like, okay, so we can't really use the Raspberry Pi, because they've got
rid of the hardware encoder in the five, they don't. And just various reasons, it becomes on
practical. So we ended up moving to sadly not a rocket, as it turns out, let's say, aim logic.
And this hardware, this cadres film for, nice hardware, great. The only problem is that the
supported G-stream camera interface doesn't, well, sorry, encoder interface doesn't support
forcing full frames and doesn't support changing the bit, right? And I could not find the source code
that would compile for that AML encoder module that would work with the camera. So like, I did
a, I spent a week looking for it, and I just couldn't find anything to compile. Well, I did find
was the door, I saw that it loaded to, to talk to the encoder. So I found the source code to that,
and did what was either an elegant or ugly depending on your taste. Hack on that was just
shares a two-by-memory segment between that and the Java. And so you write into that, what bitrate
you want, and whether you want a full frame. And for every encoded frame, it looks at that,
and says, okay, so I need a full frame in general, so it changes the bitrate as necessary.
That turned out, and it was nice about that, as you can say, it's ugly in some respects,
but it's nice in that you can say to the, you can mark this shared memory segment as effectively
read only in the encoder software, and you can have it right only in Java. So this is like,
you know which direction the data is traveling. So although you're sharing memory and you're
losing isolation from that point of view, it's still sort of controllable from that point of view,
which is nice. And we're back at the highlight and see because we've got a pipeline,
and it, you know, in some sense, it's ugly. So this is what we're doing at the moment,
which is the next step, which is to say, the thing I'd been avoiding, trying to avoid doing
for a long time, that we, I didn't want to talk direct to the APIs in Java. I wanted a nice
clean interface, turned out to be impossible to do that, or it turned out, I've avoided doing
J and I, which is just really ugly API for that. But the Java people have done produced a
much cleaner API for accessing.ISOs, that I started to use called FFM, and I'm not going to talk
anything about it, except that it, it gives you at least some sort of control over the memory
access and our, and allocation, and you can invoke, it invoke methods on the DLL. So, we basically
access both the encoder and VFRL2. The downside with the VFRL2 is we couldn't just read from
dev video. We had to use the, the, um, icons for doing the memory mapping in that segment, but that's
turned out to be manageable in the FFM, so that was okay. And we ended up with really nice latency,
we'd add 270 milliseconds glass to glass, which I was quite pleased with, and that's with, with a
private 5G between them. It seemed pure Java, so I don't have to maintain any C, which is really
nice. Um, and I don't have to change what ships in the, by the, by the cada stuff,
guys into the operating system. I don't have to modernist.so, which I'm much happier with.
The downside is the isolation is obviously a little bit limited, and there's more code. I've got
some code to maintain. However, terrible thing happened, and they have now updated the hardware,
so that it doesn't support that interface anymore. Um, the, the VFRL2, uh, the video has gone,
and it's now dev media, and it has a completely different API, um, which basically VFRL2 kind of
pretty much doesn't do anything that you expect it to. Um, but there is a dot, so which is compiled
in C++, so I have to mangle a bunch of names in order to try and even access these methods.
But it, uh, and I have to, because of the C++ API, I have to mangle, um, mangle and I have to do
LD preload in order to get this thing to work. But it does actually work. Um, I guess it's because
it's Android compatible or something. I don't, right? Don't know why this changes happened,
but it's kind of fairly unsatisfactory from my personal point of view. Um, and I'm working with
Cardus to try and make the VFRL2 stuff at least, uh, work so that I can control the brightness and the
hue and, and, and, and region of interest and all of those things. So I kind of want to say, the summary
of this is that plan nine was right, like the file system interface, the cleanness, the simplicity
of a file system interface, makes people who are trying to use, uh, advocate right applications
that aren't in C++, then their lives massively easier. If you're trying to use a memory safe
development environment, then the C++ dot SO is the wrong answer, like it just is unsatisfactory
and in, in, in every respect. And it, it would be really nice to, to, to get back to the kind of plan nine.
It's in the, it's, everything is a file, um, aspect. There's a talk nine o'clock tomorrow morning,
if anyone's awake on the basics of plan nine, or recommend you go to it. It's a real eye opener,
how they think about things, and how much you can get done with a really simple clean interface.
And I'd love us to get back at least close to that. Now I want to say, like, open source is great,
like I've been grumbling, but like the open source, I remember long ago when it was cheaper to
read to write a whole new TCP IP stack than to license one from Intel. So we've, we've come an
awful long way since then, and I, I do, I appreciate the fact that any of this was possible at all,
but it would be nice if we got back to something that was simpler, um, for the developers to use.
So, um, that's kind of the, the point, and come and talk to me afterwards, ask questions,
but shout, because I'm a bit deaf. Um, and like there's a bunch of open source, we, we do
consulting on and some of which we've written, and some of which we haven't. Um, and, uh, yeah,
I can contact me on any of those, or just find me a bit around today. So yeah, thank you.
Anyone got a question? Uh, I'm kind of talking to a little more later, but the one thing I'm
really curious about, I see really like read from the relatively, yeah, instead of memory mapping the
for your use case, you, I would expect you won't remember my purpose, because then there's those
so we do, we do, so why do we, why do I want to read rather than memory map? Um, just, uh,
so yeah, the reason we want to read rather than memory map is that it's just simpler, right?
It's a simpler, until we got to the, um, ff, um, ffm interface of Java,
memory mapping was like, I had to write a bunch of C that would then I'd have to link in to the
Java, in order to do the memory mapping, whereas, um, with ffm, I'm allowed to do that without
having to write a new C, so it's essentially about not having to write C code.
Yeah, at the cost of, but I mean, the latency is, I find down below a couple of hundred
minute seconds, I'm happy. Any other questions?
Because it's just like you asked up with a three-day shift in library from the vendor,
who's done a school at the school. So, I love to respond to you.
I mean, um, this is kind of why I'm here today to have that conversation. Why am I not using
mainline? Um, I should be looking at it. It didn't appear to be possible, so I don't know whether
it is or it isn't. Yeah, we'll, we'll, we'll find out. Anyone else? No? Good. Thank you.
Thank you.