I'm back 20 years now, I think 10 years ago, for a long time, I've already, so what
it's for those of you who don't know about it, it's a web engine that provides a I level
API that allows native applications to embed the web in native main manner, and if
that you can build web browsers, of course, but any kind of application where you want
to load a web site on, and what things we do quite a bit is a set-up of UI as well,
and use readers, you can need not many new things, and you have various layers, and
at the platform level, we have ports, so we're like companies build the specific integration
with their platform, so they can ship webkit in their port, in their hardware, so we have
Apple projects, the Sony Pistition, so as a port, and then on the Linux, the stuff we have
a GTK port, and then for embedded platforms we have, I think, on WP, and then there are other
things for Windows and JavaScript only, if you only need JavaScript.
So I want to address it here in the room, if you think about it, many browsers nowadays are
most, even the ones that based on Chrome use the web site, so it's good, I think, to wonder
how sustainable that is, is it really a situation we want to stay in, and yeah, something
to think about.
Of course, you might not know it, but you can already use the webbath to see in WP and
what GTK, it's just not enabled by default, you have to be let yourself, and there are various
reasons why we don't do enable it by default, licensing reasons with borrowing this cell,
which cannot really be shipped for GP applications, the foot pit in tabozy is quite
high, that's not really a big issue, but it's one, and then as we use GTK, we have to
integrate with the code as an encoders, with its own integration for the code as, but for
encoders, it's not supported.
Right now, we come to the other way, I said, I did encoding with the webbathc, and so as I said,
we have already used a multimedia framework, the webbathc itself is kind of a multimedia
framework already, so it's kind of duplicating one, where we have one already, so I think
it makes sense to keep the one we have already, and so yeah, let's talk about the streamer
quick introduction, and the multimedia framework, actually it's more like a data processing
framework, where you build graphs, pipelines, we call them, while data flows, and you
have filter elements, and you can like do processing with that, it's quite cool, and it's
deployed in many different kinds of situations, this space station is one, there's a
robot there, using the streamer, it was also used to, in the legal project, to detect
graphic signal waves, like to do the data processing of that information, and yeah, many,
many plugins that are available for many platforms, and check out tomorrow there's a
talk about it in the state of the union, in the open media development, so a quick example
is a digital pipeline on the left, you have the file source, which we read a file, and
then you can like DMACC it, if it's a case of ODG file, and then the code and render with
the things on the right side, so data flows from the left to the right, and then just
see what's the two things, it's the elements, like the ones I showed before, and it's also
library, so the element can be added in pipelines, and then you can have, you can, you have
signals and properties to use, to interact with a robot C, so for instance to do the
offerings or stuff, to add ice candidates and so on, and then there's a library, or
LiBGST WebRTC, which defines classes and objects API that you need in your application
to interact with, with robot C, and then for the transport, we use a library called
LiBNICE, that's also in charge of doing the connection establishments, so this
constatus of the integration of DST WebRTC in what kids, so we have many pipelines, one of
them is for capture devices, and another one is for the streaming, the encoding and streaming,
and also the coding, if you have incoming tracks, and also, we demonstrated that we can
do zero-copy encoding from directly from a webcam to RTP payloader, which is quite cool,
and then when you receive streams, they are like, to a different pipeline, we have for
playback for the video element, for instance, so we have at least three different pipelines
for single-packed connection, so for string capture, we have one pipeline, as I said, and
then you have to handle the permissions, for string capture, you have also to handle permissions,
because most of the time you're on the Linux desktop, you have to interact with things
we call the desktop portal, so we have one permission to work it, and one at the desktop
level, and eventually we plan to recover it on that, and then we interact with pipelines
really on Linux, and it's quite nice because we get DMBs, and we can like pass that directly
to the encoder, the media streamer link, we have a custom element for that call, it's
a source element, so the thing you put on the last side of a pipeline, and it can be used
to interact with capture pipelines, the canvas video elements, and also incoming streams,
like when you want to play an incoming stream, incoming video or use stream, and then if
you think of a media stream, like it has one of multiple tracks, and at the GSM level,
those correspond to the paths, the things we use to connect elements together, and that's
also our width pass data between elements, so yeah, but really simple example is stream
simplified, because the pipeline can be really big, I wouldn't fit on one slide, so I
should simplify a few things, on the last side I have a source element, which is doing
capture on the capture device, then you can encode and then RTP payload, and then you pass
that to our WebATCB, which does a lot of things, and it's sending data over the network,
and then if you receive an incoming track, you have a source pad that was created by WebATCB,
and then we can be notified of that pad and connect to a media player, so yeah, the
per connection WebATCB was designed to follow the spec, so if you know WebATCB, you can
quite easily use WebATCB already, you have signals corresponding to all the different steps
of the spec, and yeah, it was quite natural to interact into WebKit, we have some issues,
but in general it's quite easy to have a basic prototype, so it's just a precise interface
very, very, quite well, if WebKit's promised asynchronous API, so for instance, on the
JavaScript side, if you want to connect to a video element, you would say videorelement.source,
and then in WebKit what it does is that you would create a media player, and then use the
media stream source element that I introduced a couple of slides before, and connect
that into the pipeline, and the data would flow naturally to the decodes, and to the
sync. Then more things like statistics, data, and all this is already supported by
a GST WebATC, but the statistics, since WebATCB normally knows about the streaming, and
doesn't know about the encoder or decoder itself, we have to augment the statistics
with information we have in our pipeline, so there's a few a bit more work to do on that,
and then the data challenge was so quite easy to interact, because the APIs like one-to-one
mapping, and then it was really nice to interact with. There are more things, those two things
are not implemented by GST WebATC, we have to do it in WebKit, we play strike, so we're
like swish between one media stream track to another, that's basically like just using
the pipeline manipulation, basically, and then we have to add support for all legacy offer
options, because we had some application that needed it, I will talk about that in a few
slides.
So yeah, the use case is obvious ones, video conferencing, we are a little bit there yet,
we have some basic code types for our couple platforms, but they are not fully functional
yet.
So we did some work on GST on LifeKit, yeah, still work to do there in that, in the
phones.
But I want to talk about a couple more things, so CloudGaining is one really interesting
use case, we've been working on, we got Amazon Luna to work on setto boxes, it's related
with our web decoders, and you can also, if you use Amazon Luna, you can stream to Twitch,
your work can, it's not really supported yet by our reports, but eventually we will get
that working.
I had a demo, but maybe I don't think I will have time, maybe I will add in a real
short.
Another use case we worked on is pixel streaming, we had a contract with a company called
Zoo, they have an open source code application called Kitikad, and we made it working
in Linux with our Kitikad, we fixed a copy of things in work kit for that, and it was
really interesting, I also demo, but maybe later, it was quite interesting because I used
data channel for the pointer events, for the integration with the model, and I thought
it was an interesting use case for web artists.
I'm going work with quite a few things, we want to focus a lot on DevTools, because as you
might know, in work kit and in Safari, the DevTools are not as good as in Chrome, so people
have a bit of time that, maybe you know, this is a feature you have been expecting on
the web art catalogs, it's really, I don't think it's many people know about it, but
it's there, it's really basic, so we want to build up on that, we have made a back-end,
we call it DevTools back-end, which like when you enable it, it will emit a JSON stream,
you can do live-grafing or post-motaman analysis of your web art selection with that, for
now it's enabled with an environment viable, and I have built a basic, really basic
content for that, you might recognize the graph from Chrome, so I reuse that to make
the graphs, so that's currently like a third-party project, they don't really would like
to have it integrated in WebKit itself using a custom UI protocol, or maybe in Inspector.
Now things we work on currently, network collision simulations, so we want to improve our
packet of story, like fake RTX and these things, so being able to simulate conditions
is really nice, and we added that support recently, we have the basic infrastructure for
similar cast, as we see, like VP8, Templars, Calabity Works, but it's not working yet for VP9
on V1, so some work we've been needed in GSMXF for that, after encoding we made some early
experiments already on Raspberry Pi and on desktop, and we've got it working, but it's not
free of string yet, and then there's other things such as entering encryption, yeah, the story
there, we should implement that because you can encrypt in various ways, yeah, we need to work
out on the implementation details, we're getting that. We want to improve, so I understand
boxing story, because WebKit is splitting various processes for secretary reasons, right now we have
everything working in the web process, which is not really a good for security, like the
catcher device success is done for the web process, yeah, eventually we want to improve on that,
and then the platform we will have to continue with that, but on the desktop, we have to use the
we could use a portal called the camera desktop, camera portal sorry, yeah, and again we got
really guarding sandboxing, right now we also have to like disable the sandbox in the web process
to be able to set up a peer connection, because we don't have the UDP, the RTP traffic rooted to the
network process, so yeah, it's still some work we plan to do with a new ice implementation called
device, which is written in Rust, and it's quite cool projects, and eventually we plan to
integrate the new project into a kit as a ice transport backend, yeah, that's it, that's it, that's
thank you, thank you.
sorry
So the question is about RTCP feedback, yeah, if we support that or not, yeah, I think we, yeah, it's
supported in about Sibinia, yeah, the RTP stack in just to me is quite complete, it's really awesome,
there's many things there supported and it works quite well as well as so, yeah, yes.
I do, I do explore perhaps using the higher level elements with RTCS or Sibinia from the
rocks plug-ins, because that came after you started the project, yeah.