Hi, so my name is Daniel Moy. I'm a senior software developer at Calabra, and I've been working on
Jason Murphy for a while. So at Calabra, I lead the effort working on adding analytics
supports to Jason Murphy. And also, I'm a Jason Murphy developer. So, first let's talk about
Jason Murphy. It's like probably a lot of you already know it. It's a major multimedia framework,
open source. It exists for 26 years now, I think, and it's maintained by thriving and very
welcoming communities. So, if you have any interest, I invite any of you to join. I'm relatively
new to the community. So, I went with it. Yeah. It has extensive plug-in system. It's a proven
architecture, very flexible, and you know, composable, and it's been used for analytics for a long
time. But more recently, with all the analytics, all the major silicon vendor adopted it,
fork it, and I have to say specialize it on their framework. And so, why do we have GST analytics?
Well, because we want the core of this to be available in the upstream Gstreamer, not necessarily
in the vendor fork. So, this way, when you decide to move to a different platform, you don't have
to restart pretty much all what you've done for scratch. And so, based on this, what we want to offer
is a loose coupling with, we want you to have the access to the acceleration, but in a loose
couple of ways, like it is done for a codec and everything else on Gstreamer. And yeah, it's
we, in the way we do this, we really adopt the Gstreamer way, the mechanism, and in some case,
if the mechanism doesn't fit it, then we adapted to the analytics needs and other elements in
Gstreamer can benefit from it. And most importantly, I think that it's a community-driven
effort. We get inputs from very experienced developer and reviews, and we understand the needs
of more people because the, like, they can bring it to the project. Just a little bit about
what is GST analytics? Well, it's a set of elements that are a bit specialized for this.
For example, on X-Inference, TFLite, Burn, those are the elements that encapsulate the
the inference framework, but because the in machine learning, the outputs normally is very
specific to the models. So, we have elements, what we call a tense-ready code, they translate
the output from the models to a more standard way inside Gstreamer that other elements in
use, again, decoupling it from the model. So, we can think about if you have an object detection
or a segmentation or another model dealing with voice. Well, it's not necessarily specific to
that model. You can build an application that's using it without being specific to that model.
If you find later on that you have another one that is more efficient or more suited for your platform,
you're able to change it without, you're able to change the model you're using without impacting
your application. So, yeah, consumers. So, once we've produced those results, we have,
we've built a couple of elements that can use it, like we have overlays on the
serializer, this serializer, and yeah, and the latest is, I bet earlier I was talking about
we are adopting the built-in mechanism. In Gstreamer, for example, you know, it's, we
Gstreamer use negotiation to find the compatible elements that are, like, based on capabilities.
So, for example, in inference elements, we'll produce capabilities that are specific to this model,
and then the tensor decoder have the capabilities that describe the capabilities it can support.
So, this way, they can be negotiated, and so this allows us to create a tensor decoder bin. So,
we can select the right tensor decoder based on the model that is used. So, it's simplified,
again, building an application on top of it. We, based on that. So, in Gstreamer, we have met us,
met us or attached to a buffer, and we extended this to, to, to, to, for analysis, for example,
you know, we can describe object detection, classification, key points, and so on. So, this is,
the, the meta framework is very flexible. I had the, had the, had the basis just a matrix
that define relation between different met us and, and, and container. So, you're able to
create new met as, as you, as you need, but you have already multiple examples to follow
to do this. And, yeah, as of in, one that one TA that was released last week. Now, we have,
like, yeah, we, the, the negotiation is available inside, between tensor decoders and the
inference is available inside Gstreamer. Here's a typical, very simple analytics pipeline.
You normally have a sort of data preprocessing, and analysis, post processing, and analytics.
And here I just mapped it to a simple pipeline. You know, there's a lot of Gstreamer
element that can already do most of the preprocessing we do. Some of it is still, it's indeed
inference, because we didn't have, for example, four floats support. We don't have a, a, a
video type, that support, a floating point, but we are adding this as, like, we are working on this,
it should be available soon. So, yeah, for the next part, I'll switch to a bit demo,
so I'll show, I didn't know, and explain a bit more what is, uh, what that was talking about here.
So, yeah. So, the, the first demo, it's a segmentation. So, yep. So, I have a segmentation
of running inside the Gstreamer pipeline. So, and, oh, yeah, forgot. So, if I have multiple object,
you know, they should be segmented out. So, that, that's just an example of segmentation.
Here, there's a couple of things to know. Like, they're, again, it's, uh, the, the type of object,
or, uh, different MTD, like, analytics meta with the segmentation, and they're put in relation
together. So, it's very, can adapt it to your needs. So, the next one, uh, I'll have a couple
one that are sort of, uh, built to, uh, a final goal that we'll see a bit later. Um, this should be
detecting the, uh, hopefully it will. It seems like it doesn't, uh, shoot. That's sad.
Oh, anyway. I don't know, maybe the, the light is, uh, oh, seems like, oh, yeah, here.
Maybe. Yeah. So, we had, it detect my hand and, with an angle, like, it's a rotated box. So,
it detect the orientation of my hand to you. And, um, so, I'll go to the next one to explain a little
bit how this is, uh, done. Um, so actually, when it's detecting my hand, it's detecting specific
position on, on it to, to, to understand what's the orientation. And that's how the, the box is rotated.
And here, it's actually, to, to be able to show you that I'm using the, the relation meta that are
able to put to, you know, the, say, like, there's, uh, there's a, there's a bounding box that
this is associated with key points. And, uh, uh, that's how the visualization is working.
Uh, so, if I go to the next one. So, um, actually, again, this is, we're building to, like,
now we are just involving one model. We're building toward, um, oh, yeah. So, now you see my hand is,
like, a horizontal, but it appear vertical. That's to simplify, like, that's the requirement for the
next model. Um, so yeah, it's showing upright on the, on the window. So, this, uh, allow to,
that's, now, at this point, the, the, the first inference is the pre-processing of the next model.
And, uh, here. So, uh, uh,
you guys should have, okay. Oh, yeah. Well, maybe if I just position myself a bit like this.
Sorry. Yeah. Yeah. Exactly. So, yeah. So, the pre-processing, like, now I can detect, um,
key points from my hands, uh, and, uh, yeah. So, so, so the first model was the pre-processing,
and then it feeds to a second model that is doing the landmark detection. And, uh, yeah. So, the next step
is actually, so I wanted to, to use, uh, so based on this, I can detect, like, the specific position
of my, my, the key points of my hand and do, uh, I'm flying language recognition. So, with a bit
light, it's going to work. So, uh, oh. Oh. Oh. Oh. Oh. Okay. So, uh, oh,
pushing a key. Oh, that's not the right thing. So, uh, unfortunately, I, like, it doesn't seem
to work here. But, uh, the idea is that, uh, I would feed the key points in, uh, in a second model
that would classify the position of the key point and recognize the, uh, the letter, like,
that I, I'm, I'm describing. And, um, and then I, I, I, I was able to, you know,
the tech on, just on, based on time, which letter and, uh, in words and describe, um, yeah.
So, back to my presentation. Yeah. So, we still can improve, uh, improve this,
and, uh, maybe add also ASL word, which is more than just, uh, there's a, uh, it's, oh, yeah.
It's, uh, there's a video analytics, and here's, it was just, uh, image analytics, but we could
recognize more complex language. And actually, we, one thing we were thinking is actually,
there's a, we have, there's an application called cdwet that is based on pipeline, uh, on pipe wire,
and, uh, uh, uh, which were, the goal is to, uh, do a virtual, uh, virtual camera, where it can
remove background from, from the camera, but independently from the application you're using.
So, if, if they're, if they, they're background removal, whatever video conferencing you're using,
there's a bit crappy, you can use your own, and, uh, also, um, so it could do background removal,
but it's also could do the hand sign recognition. So you could generate a transcribe of what people
are like kind of signing basically and, uh, even going further, we could, uh, send the transcript
to whisper speech to, uh, uh, you know. Uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh,
uh, uh, uh, uh, uh, uh, uh, uh speech synthesis from, uh, from the transcriber. So that's all, it's just
This example of things that we are doing, but I'm mostly working on the infrastructure
to get those models and all the free infrastructure to get analytics working in GSMR.
But I think we've, yeah, we went far, so I'm inviting everyone to look at this.
So in summary, GSMR offers a very highly composable framework for a building analytics pipeline.
It has a powerful metadata framework based on graph.
So there is no assumption on what can be related.
So which helps you to describe better, like the scene, what's based on your intent,
and it has a loose coupling with the acceleration.
So it should facilitate porting to other hardware or, yeah.
So that's all, thank you.
Can you take questions?
Yes?
What is the, what is the, just remember version you're using?
It's 128.
Yes.
So the question was, which version of GSMR we were using for this is, I was using 128.
But depending we exactly which element you're, that was there.
There are available since 124, like for example, on X inference is there since 124, I think.
Yeah, as of 128, all the, the one that I've used are available, yeah.
Yeah, so actually, I just run on the CPU.
So the question is, which hardware I'm using.
So I'm only using a CPU here.
So, yeah, AMD or ARM 7, yeah.
Okay, yeah, the question is, from on X, which packet, what I was using.
So I'm using on X run time.
I could have you, like, if I use the, there's also on X run time GPU,
I didn't have, and didn't have time to set up all the, the hardware to get all the acceleration.
So, but all this is on CPU, it would be faster if I run it on the GPU, of course.
Yeah.
It is possible in just primar to, is possible to put this on GPU in just primar?
Yes.
So, so, so, is it possible to put this on,
to put this on a GPU? Yes.
So, because we use on X run time.
So, there's an abstraction there.
There's, there's different back end to, on X run time.
And so, there's CUDA tensor RT.
There's also, well, a lot of others,
VSI and, like, yeah, most of this is,
I have a back end for on X run.
Well, they, if you, if you just build on X run time with those back end,
okay, if the question is, if it's available with those plugin,
this will be the same plugin as we use here, is just when you build on X run time,
you'll have to build it with the, the specific back end.
So, yeah.
For TF, just a bit related to this.
For TF light is a little bit different,
but we, we have, acceleration, like,
a delegate available for the VSI, which, yeah.
Yes.
Can you?
Okay.
So we try to, like, we don't, so the question is, are we utilizing the vendors board package
for, for, for the, for the inference, but so we try to keep it to the minimum, and so we're
not using the, the, the vendors to look and package directly, other that, like, it would
be, you know, abstracted by the inference elements. So it's, don't use the full framework, if
that was your question. Yes. Exactly. Any other question? Thank you.
Thank you.