It's my second presentation about this project.
I was there last year.
I thought, no, if some of you have seen
the previous presentations.
Last year, we talked about the co-conset of our project.
This presentation will be still about the same project,
but we have made a lot of advance on it.
The topic is announcing web media supports
who web SME,
driven open source frameworks.
We will focus on this presentation about the SDK
that we have produced,
and we want to foster the reuse of filter
to enhance the contribution to this project.
So, let's start by presenting
who am, so I'm John Graham,
a university lecturer and researcher at
Yuli Nguyenia Sal Amir.
This work is the fruit of research I predicted
many years ago during my PhD,
which was about building a universal
virtual machine for Reco-figurable video-connect.
So, I mean, the co-concept is still there,
but we have integrated, like many, many technologies,
to ease the development,
to foster the deployment of staff.
Meanwhile, I've been working for JPEC,
so I was involved in the project of re-architecting
JPEC, you all know about JPEC,
and I have also on a complete niche over the topics.
I have made a plantation yesterday
about completely different topics,
so it was about cookies,
whether you like it or you like it.
You don't like it, watch this plantation,
because I'm a member of the support pool of experts,
so I have made a project
that I've never software to spot cookies.
This work is also supported by Maya Bistrom,
which is the CEO of Barat Technology,
and she also is the co-creator of the platform one
codec, which is a video sharing platform,
so she couldn't be there today,
she's living in Massachusetts,
so it's not like next door,
and I mean, this is not like a sustainable travel
to get to Brussels.
Anyway, so the state of play
I mean, what we discuss in our last plantation
is that browser or gatekeepers
when dealing with formats.
I mean, they can promote or discourage
the supports of format,
so this gives them so a lot of power,
great power to decide the left life or death of format.
So this happens for instance to og and fura
from the exeve foundation,
and also the GPGXL,
for which the support has been dropped recently,
and this induce like a lot of popularity
despite their respective advantage.
So what we propose?
So by only using WFCCWFCC standards
and open source technologies,
we want to fix this problem of gatekeeping.
So this should help everyone to innovate
and use their own standards.
This also will create the ability
to support the gasi-fundars format,
and this also will help people
to deliver new format,
to experiment with new formats.
Okay, so now how it works.
So instead of creating complex script,
creating new tag, having to integrate
like a new framework,
we currently extends the common tag
by using web components.
So we put new attribute to the extract
that will extend that will add some logic
to this commonly used tag.
So this has several advantage
because you don't have, you have fuel impact
on your web page, if you have CSS or JavaScript,
which is attached to this tag,
then you don't have to modify it.
And that sits.
I mean, and it's this work with video, audio, images,
and stuff like this.
And then for the web assembly,
and the media feature parts,
what we do is that we create new attributes to this tag,
which is the using tag and the web tag.
And this will add,
the this will add some clue to this internal logic.
So this attribute will point to external dependencies,
and the idea is to enhance the support
of the browser itself.
So for this example,
we take the medical demo.
The template, which has been uncoded using the MP1 format.
And this MP1 format, as clearly a port is supported.
Now by browser and the tag itself,
I mean, the internal logic is going to test,
which format is currently supported by the browser itself.
So this is where the pack is used.
So the server, where the rule of the server,
will be to resolve the pipeline and to execute efficient
the pipeline.
And by using some library, which are pointed with the web attributes.
So on this example, we have the lib, MP1,
which contains a set of filter related to MP1.
So on this one, we have a MP1,
PS, and TX, the mixer.
We have a reframor.
And then we have a decoder,
an MP1 decoder, which is taken from a subset of MP1.
And same for lib x2, 6,4.
We have an encoder,
taken also from a subset of MP1.
We have a reframor, and we have a mixer,
that will send the result to the browser itself.
So yeah, to summarize,
we have the universal video that will extend the tag,
the solver, which is like an optimised version of jpg,
and then you point to a set of library that do the decoding.
So each filter has specified input, specified output.
It has code inside it, and the idea is to use the solver
to find the right combination of itter to resolve the problem.
So if you just take a step back on this example,
then you might think that transcoding each video inside the browser
to have the normal format, which is supported by the browser,
could be like an overkill process,
because you have to do it for all image,
or video that you are putting inside the browser.
Actually, this is the more challenging use case,
because it's video, because inside video,
you can have like many codex in it,
you can have a AC audio,
you can have h2, 6, or hdc video.
So the idea is ready to have something very modular,
that then once it works,
you can optimize into a single filter to reduce the concise,
and to only take the part that you want.
And if you think about it,
there is many optimization that could be added to that.
And this is what is done with the video tag.
First, we can detect is, for instance, web codec is supported inside the browser,
and if it's supported, then you can just cut, retrieve the frame over there,
and then use the GPU on coding to make it faster.
You can also directly use a canvas, for instance.
So the real idea is that you have something very modular,
and you can add a lot of logic in it to make it work.
And the last thing is that, remember that the HTML
title, this kind of issue of supporting issue,
by only providing some track inside video.
So it means that if one track is not supported,
then it will get another track and stuff like this.
So it means that the only alternative provided by the browser yet
is only to provide several sets of the same video.
With that, everything, I mean, the all the logic is inside the tag,
and you can do a lot of things with that.
And this is the modernizing case,
but when it's used with image, with audio,
this process is almost instance,
because WebAssembly is like gasing fast.
So if you need to be convinced,
about this, I have made a demo website,
where you will find some images,
so you can test it with the URL or with the QR code.
And on this page, you will have a G2K image,
a GXL image, you have an AC-3 audio,
and you have MPEG1 video,
and you will see that on most of the browser,
maybe only on Safari, Safari is,
I think it's about MPEG1 and also a GXL.
But in any of our browser,
such as Firefox, such as Chrome,
nothing will be displayed.
So this one is like using the command tag.
On the right, you will have the real web page of ther.com,
using the universal tag.
And if you test it on your phone,
then you will see that everything will be shown quite instantly.
I can show you how to do it.
I have a time.
So that's it.
Is it used?
The answer is yes.
I mean, if you have the filters to do it,
then you only need to modify the HTML itself.
You don't need to include complex scripts.
You don't need to dive into any framework.
You just need to know your old knowledge,
about your good old knowledge,
about the HTML and place.
You know the filter.
To help you about this tag,
how to build your tag.
We have developed a VS code extension
that will help you to just provide to you the right tag.
To do this integration.
So it's called beta access.
And it's available on the Visual Studio code.
And this is all the structure of this extension.
So you click on a file,
and then you will get the tag that make it work.
So you have a preview and you have the tag.
And then you have a library of existing filter
through the framework.
So I have made a video about just how it works.
So it is.
So I'm just click on a gxl file.
So since Visual Studio code is based on Chromium,
then gxl is not supported.
So then I'm going to install the extension,
the directsess extension.
There it is.
Devon.
You click on it.
And then you click on the same file,
which is in support here.
So you could be audio,
which you will see.
And then you will get the preview.
And it tag itself.
And seven.
Function 80 that will help you to the integration.
If you want to export everything in the installation
and HTML template,
it will distribute the result on your browser.
If you want to test in any of your browser,
if you want to optimize this code,
et cetera.
So yes.
So is it to use the adjunctsware as yes?
When you have existing filter,
then you just have to put some attribute
to the wave, some information,
to the wave attribute.
Now, for if you don't have the filters
to to to handle,
I mean file inside the framework,
It is it's another problem.
I mean, is it easy to develop?
The answer is really like, no.
Not very.
Well, because you have to have like many privacy
quit, precrecaise,
to do your development.
First, you need to install to understand
how xPack is working.
Like, what is the, the, the, the, the, the,
the function that you have to include.
I mean, it's not so complicated.
Usually, you have two functions to get used
is how you recognize a mid-stream.
For instance, what is the magic code of this format
and how you process the stream itself.
And then you have a several other functions
that helps to tweak the filter.
But it's not like the others program.
The others program is really to do the conversion
in a web assembly using a web script.
I mean, I don't know if you have ever used
the web assembly completion.
You have a glue in JavaScript.
And this glue, every change of version,
this glue will be changed also.
And I have put many optimization
inside the web assembly.
So we have to redesign at each new version.
I mean, this glue are the difficult code
that we have done.
And then the last one, the last problem
is also, is there any intellectual property
around this format?
Are you able, are you allowed to distribute it?
And so this is quite a complex question.
And I mean, you take your responsibility
when you create your own filter, you know?
But I mean, there is a lot of license-free filters
that are existing that you can use.
And they talk more, I mean, more efficient than PNG,
than GPEG than I don't know, P3 for you.
And that you can free use.
I mean, there is no restriction about this.
I mean, this is really your responsibility.
But for the web assembly part, for the JPEG
part, we have designed a new interface
to help you to build this new filter.
So this is the interface that we developed.
So the idea behind this is that the tool chain is open source.
Of course, it's open source.
The all this work is open source.
The tool chain is open source.
And you can freely integrate it inside container,
inside, I mean, VM.
And what we choose to do is to integrate this tool chain
inside GitHub.
So this interface will be linked to your GitHub accounts.
And then we will use the GitHub action
to compile the color and get it into the right format
to get it used inside a framework.
So here it is how it looks.
You have several action, and we'll see what the action
is for.
And this is just to follow how I mean, the progression
of your completion, if you have any problem,
that it will be reported over there.
So there will be two main actions to compile the color,
which will be to commit and push to your GitHub repository.
You will have everything compiled with the GitHub
action.
And then you will retrieve the result and put it inside your query.
OK?
So I've made also an example for that.
So what I'm going to do is like consulting
a very simple feature, which is the VMP feature itself.
So I have some C code language to the code VMP.
And you can access to this public repository.
And I can show you.
And the idea is really to build your own filters using
VMP, I mean the SDK.
So we have a compressed VMP inside the media pipeline.
And what we're going to do is to decode it and flip it
to show that this is really this one, which is used.
So I will have again a video for that.
So this is how you get into the SDK of your man.
So first, I have used my users to code.
I have BVR, the BVR extension, which is install in it.
I have a click on VMP.
And what this message means is like, even if
Chromium can decode this image, I'm not using my framework
because there is no VMP filter as a candidate
to use this format.
So let's create our own VMP decode.
So first, I'm going to click on add.
You have two options of that.
So the first one is that if you know,
and you exist in repository that provide the VMP decode,
you can just add this repository and it
will be automatically added inside the list.
But for my case, I will start from scratch
for doing this decode.
So I'm going to create a new project, which
is called the VMP demo.
And then I will enter inside my new interface,
which is like this decant interface.
So it is, so you have a G over there, a B.
And if you click on it, then you will have the full access
to do the completion.
So when you create a new project, it
will be a fork of a base filter of a locust
written of a filter.
And then you can just copy past your code inside it
to make it work inside the full media pipeline.
So you have some C code.
You have some JSON description to be
filled to make to give some clue for the selection
and this file.
So this is like the syntax of the pack.
So some of you may know.
And what I've done is clearly to copy past my code.
I have put like the magic code to spot like BMP file.
It could be using BMP code on the strings,
or having like a reading the extension to see if it
matched with the filter.
And then I have only my processing function, which
will be to decode BMP.
OK?
There is.
So this is not a lot of code, actually.
So I've put some metadata on it.
And then what I want to show you is this,
just to make it sure that it's clear for everyone.
When you create a new filter, you will fork the base filter
and you will fork this filter inside your own icons.
Which means that you can trigger a GitHub action
to do the completion.
So it is, so you don't need to go on GitHub.
I mean, everything will be done on the background.
It's just to show you how it works.
So I'm going there.
What I'm going to do now is to commit and push
my modification inside my repository.
Let's go.
And then push in, and then this will trigger the GitHub action.
So this is to convert from C to my framework
of the BBR framework.
You will see all the building step inside the extension
itself.
And once it's done, then you can add the result.
To the list, you can add to this list.
And now you see that the BMP is flipped.
So it means that we are using the BMP remote
that we just built.
OK?
So this is the general ID.
And the general ID is really to based on fork
because every fork that will be created
has to be public.
You know, it's a limitation from GitHub.
So what we really want is to have many features,
which are built.
And the more filter, that will be produced,
the more this framework will be used.
And the more you will be able to do with it.
So it is.
So I mean, I've really worked on the documentation
of this project, just to show you how to do it by yourself.
So on this part, the developer part,
you will have all the steps that I've shown
to create your own decorator.
The BMP decoder is something simple,
but usually what you will have to do is you know a library
and that doing that is doing something, you know?
And you just have to integrate this library.
I mean, do the update session between the process
function and the library itself.
And this is most, if you have like existing secode.
I mean, under the project is not only limited to seeds,
can be used in rust, can be used.
And I think that could be combined was as long,
as you respect, you know, the API of the project.
So this is where you can find us.
Everything is explained on this.
So this is an open source project.
So all the filters that we have currently released.
So there's many filters, which are existing,
but we have limited the release to the one
which are free to use.
Where there is no limitation to distribute your project.
The glue by itself is based on JPEG,
but it could be based on everything else.
It's just that JPEG has many advantage to do this
integration because it's all written in C.
There is very few dependency.
The filter itself is very, very small,
very, very few, I mean, things well contained.
And that's it.
And the tool chain to build your filter is available
on the player repository.
So you can use it locally.
And I mean, this is the same thing,
which is attached to the GitHub action.
So how it is, if you produce new filters, of course,
it will be your filters.
We don't, I mean, we first do the terms of services,
but it's like prior growth, everything is open source.
So you can do everything that you want.
If, I mean, for the filter parts,
if you want us to reference it, I mean, of course,
we will just review just for security reasons.
But I mean, everything is very open.
So I think I've been quite fast on it.
But I mean, it's like all for today.
We really open, we are really open to all contribution.
I mean, if you are working in a research,
if you are like an artist, like for the effect glitch,
if you want to create your own filter,
to have dynamic glitch inside your web page,
you know, it's something that is doable.
In there is a lot of opportunity,
it's based on this framework.
So I think that's all for me.
I'm like, open to question.
If you have ideas, want to contribute,
have many, any question, like, give you the mic, yeah?
Are you convinced?
Yeah.
OK?
Yeah.
We're doing a web page.
All the way in better history.
How did you work in a web page?
Yeah, yeah.
That's a good question.
So if I have created a new page,
how to include this filter in the, yeah,
how to include these filters inside the new page.
So that's it.
You have like the universal tags.js.
And this is the only integration that we will do
on the page.
And then you can extend any of your image tag,
audio tag, video tag with this new attributes, you know?
So you have to have this script inside the page.
And then you have to have your filters available somewhere.
I mean, it will just get the filters where it is.
And you will, like, retrieve it, compile it,
and then execute it, you know?
So you need to have, like, as a material, the JavaScript,
and then all the, the, the, the, the, the wasn't file,
which is the server.
And I mean, the separate wasn't file, you see?
But you can distribute it from any place like from a CDN,
from, even from GitHub, you know, it work from GitHub.