I think we can start, and people are still coming in.
So basically this talk is going to be about how can we, as an open source community, be able
to build a manoried and especially general purpose, you manoried, right, that can do kind
of anything.
I'm going to highlight draw, which is a project I'm building, and feel free to test
it out.
You don't even have to test it out.
I hope that this talk inspire you enough to give it a try.
So you may remember me from first-end last year, I came with a small robot that was able
to cut itself, to do stuff.
And the cool thing is that after this talk, there was the school founder of hugging
faith that gave me a call, who was like, that's really impressive, that's really cool.
You should join us to do human rights, and so that's why I'm doing this talk right now.
That he was not really like enthusiastic about the robot coding himself, it was kind
of like, yeah, maybe we should just do end-to-end learning, so that's kind of what we're
going to talk today.
And for me, I'm just a full-time open source developers.
I have my own company, and you can hire me if you want, but yeah.
So maybe just for start, why do we need humanoids, why can't I have all the things like
all the type of robots?
The thing is AI strangers award, if you probably realize with Chargipity, and Robotics
have suffered for years from not being able to generalize, right?
So if you have very good robots in factories, very capable, but they will never be able
to generalize now, if you want him to do pancake, he'll never be able to do.
But the thing is that we kind of know how to generalize with artificial intelligence when
it's about mimicking humans, right?
So Chargipity, you give him a lot of text, and then he starts speaking.
We don't really know why, but he starts doing this.
And so we kind of want to do the same thing with robots.
And the way to do it with robots is we probably need to have a robot in the human shape, so
that we can show him how to do stuff, and then he's going to be able to do so.
It's kind of our hope, and nobody should work, but through this, we don't actually know
how AI works, and so this is kind of like, how it is.
So let's maybe define what's open source general purpose seminars.
There's kind of like three big components that we need to pay attention.
First one is AI, right?
We need AI to be general purpose.
We need hardware, we need open source hardware, hopefully, so that we can really think
around as a hardware, and then we need software.
And if we have those three components that is open source, it means that we have full control
of the human rights, and we can repair it, and maintain it, which is actually the hardest thing,
right?
It's really easy to buy, it's really easy to build, it's very hard to maintain, and it's
true for AI, true for software, it's true for hardware.
Like if you go to this adventure, you will break things, if the AI is a hardware, and being
able to just really point apart, it's going to go a long way.
And if in the future we want to have children to live on Tatooine and be able to build
this 3PO, this is the way to go, there's no way to do it with closed source hardware.
But is it really attributable?
Well, the thing is, there's a huge community of people doing open source humanoids now,
it's kind of like a booming thing.
I work with Hagenphase, they have all kind of a naxumbly kit from scratch, and they will
also teach you how to print 3D parts, build it, connect several motors, and how to connect
it to computers to be able to control it.
And in this way, you can also build your own humanoids, right?
You don't even have to follow someone, you can just take maybe a template and modify it.
So I would deeply, if you into hardware and deeply calling you to try it out, I think
it's there, it could be very fun, we can project there.
And also, like yeah, it's a couple of movements, and then AI is also having a moment,
especially open source AI, from I probably heard about deep seek,
mistrol, Kuan Vial, Hagenphase, there's a lot of open source motors that is popping up,
but also about the training algorithm behind it, because at the end of the,
a huge chunk is going to be about training, and there's even more specialized,
especially for robotics, that is popping up.
So if you into AI, I would highly recommend to check those things out as well.
And so this seems to be kind of a solve issue, or at least there's a huge community around it already.
And so the missing part is kind of the software, from my opinion, right?
And this is what I'm trying to solve with straw.
The idea is that you have hardware on one side, you have AI on the other side,
and you're really trying to make those two connect in a way that is very efficient and very fast.
So if you only have AI, what you typically do is kind of request reply part,
and where you have an API, you send a request to whatever you want,
and then you get a response, it doesn't really work for robotics.
What you want is kind of a pub sub type of system,
where you would be able to publish a data from, for example, a camera or microphone,
and then the AI is going to be able to subscribe to them,
and then get the latest data, and then be able to inference or whatever,
and reply back.
And so that's what we're trying to do.
It's actually not very complicated to do pubs, so if you into software,
what is actually very hard to do is how do you distribute the code from
on different type of hardware, different type of requirement from the software side.
You might have code that is written in Python, if you say,
you might have code written in Rust, if it's for robotics.
And so how do you make it very easy and accessible for anyone to start on the weekend,
and not have to install a million compilers and be able to figure out,
if it's Linux, if it's Nvidia, if it's different type of hardware acceleration.
And so what we try to do is to push everything on tip,
which is Python, the Python package manager, and then make it just peep installable.
So everything from there should be peep installable,
and this is kind of the way we think to source a problem,
people will be able to automatically detect which OS using,
which architecture using, and hopefully we can even push it more with
a which acceleration you're using as well.
And we can also package Rust or CC++ code into Python as well,
and then people can just use it from there.
All right, so maybe I can just show you very quickly how do our works,
what hard work is that you define a YAMO graph.
So you have kind of a YAMO configuration,
what I say, okay, I want a camera, I want a narrow model,
and I want a visualization.
This is the input, this is the output,
so the camera is going to send a flow of images,
and then we will have a narrow model on the vision,
so it's just going to subscribe to it with an input keyword, right?
And then you can have environment variables to specify it.
And so I can show you quickly how it looks like,
if you're using your own computer,
so it's kind of the repository from Gidham,
and in tight-door repository, you have an example space,
where you will have all the examples that we've built,
so one we're going to look at today is a VLM1.
So VLM sent for vision language model,
so one that's able to react to text and image at the same time.
And so I'm going to show you one that is very easy,
that I just show you, and it's just like three nodes,
so three, three separate process, right?
And each node is a separate process.
I don't know if I mentioned this.
And so I say, oh, here's three nodes.
I have one output from the camera, which is an image.
I have a ticker because I want to coin VL model,
so coin VL is an AI model to do inference on the timer,
and then we have a clock, right?
So if I do this, I have to first install everything, right?
My computer maybe he doesn't have to install everything.
I already installed the right, it's very easy.
And when I have to install nodes, I hadn't just say,
do our build, and then he's going to figure out maybe too fast.
So let's say I want to start from scratch.
I want to create a new environment.
Let's say I build a new environment randomly, 3.11.
What it's going to do is I can use UV, which is a Python package manager,
build also in Rust, so it's safe.
And then I say, OK, I want to use this one.
And now I want to install everything from this.
And then we use UV, it's going to pull everything has to pull,
and then it's going to install everything has to install,
and then it's just going to work.
And hopefully, finger crossed, it's just work from scratch.
But the idea that everything from there is to confirm by the developers,
so you kind of put all the specification within
Pyproject, Pypy, until in a way that it should work for you on your computer.
When I want to run it, I just have to say, do our run.
OK, I think I missed the slide, but it's OK.
And then it's going to be able to pop it.
And now it's going to take a bit of time,
because we have to download the model from the internet.
We can also have local models.
You can have adapters.
That's what's been shown later.
But one, OK, I'll see if they're my phone is fine.
And then, say, maybe I just show you what I just showed you before,
which is this.
So Pypy install is just a Pypy install.
There's no compiler.
It's compile of free.
The idea is that a lot of people, especially if you're starting up with the buildings,
there's so many, so many things to learn that you might not want to have
to install compilers and stuff.
And when you run it, it's separate process, linked with chat memory.
So that's a very big thing to have a performance system,
especially for war weeks.
And so now everything is running.
And so now I will have the VR model, like it may be smaller.
Can I zoom in?
No.
All right, I'm sorry.
Just zoom, zoom, maybe?
OK, zoom in, perfect.
So you can see that the model is just describing.
But he's able to have, is it too small, still?
Yeah, maybe.
OK, sorry.
Sorry, sorry.
Zoom in.
OK, is it better?
Is it OK?
Oh, yeah.
So now he's just describing what he's seen.
And if he's seen me, he's going to say, hey, there's a guy looking at him.
So I try to do peace sign.
He's going to say, oh, you're making a peace sign.
And so he's very descriptive, right?
All right, he's carrying, OK, that's fine.
And so it's very capable, right?
Everything is running locally on my MacBook laptop.
And now, like, OK, it's really cool, it's really nice.
But it's actually very able to see stuff, right?
If it is built, it's right with it, it's the built to see stuff.
And so yeah, so that's super easy stuff.
And so you can do cool demos.
And so that's kind of like a very quick getting started with Dora.
And yeah, so I just said general is important.
And we tried it basically.
And we said that, oh, if we show him, it's very fast.
And compared to CCP, or HTTP, if you do API calls, especially when you
have more and more data, right?
So if you have HTTP or you have multiple cameras on the reachy,
currently we're going to do a demo with this three.
So it can very take some times.
And if you really want to have a quick feedback,
it's going to be very important.
And also, we realize that not only is CPU kind of
sharing very important, you also have the V-run,
and you might want to do like Shed V-run between nodes, right?
And this is also possible with Cuda, hopefully in the future,
we can do it with more.
And it's also safe a lot of time because you don't have to
copy from CPU to CPU, CPU to GPU.
So yeah, all right.
And this is all possible because we're using Apache Arrow,
which is a unified memory format.
And basically they say, OK, an array, it's like this.
F-16 arrays like this.
And then you can use it from Python, C-C++, Rust.
And there's already a lot of libraries that can use
this unified memory format from new MPHPANA to PyTorch.
And the good thing is that you can use it on CPU, but not on GPU.
So the latest benchmark on GPU is actually using Arrow GPU.
And so it's kind of not just for Cuda, but it will be able to
scale on, for example, macOS or other types of platform.
And you will be able to share CPU memory from Python to Rust,
hopefully.
So that's kind of the idea, there's a talk, I haven't, like,
develop this, there's a talk in this for them from Arrow.
If you want on Thunder, I think.
Yeah.
And so if you want, OK, so that's very cool.
If you want, we can do a last demo, which is with the richy.
And so what we need to do, if you want to make it even more capable,
it's just to connect the output of the VLM into a robot,
and then we have a moving robot, right?
We just saw that it was very able to describe stuff in the world.
And so yeah.
And yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah.
It's such an amazing part, yeah, okay.
So yeah, so what you do is you take the VLM output,
and then whatever he says, you're going to map it to an action.
And so for example, if I try to say hello richy,
can you do a hand wave?
He's going to be able to hand wave me.
And so basically, he's able to use this VLM description, right?
And to do an action.
And if I give him stuff, he's able to give stuff,
he's able to go left, right?
Depending on if we're sending on the left, on the right side of the camera.
And also what we can do is have a speech to text model.
And so what is going to be able is that now we have a fixed prompt,
but we can also have a modified prompt, okay?
So yeah, this is good.
And so for example, we can get a napkin.
Yeah, okay, that's fine.
And so for example, with the speech to text,
we just add a no, no, the separate process is going to run a whisper model.
And then he's going to be able to translate whatever we say to a text,
and then we're going to put the text into the prompt,
and then you're going to be able to modify his VLM, right?
Instead of just having us, he's going to be able to,
I think he's kind of stuck because it's mine, okay?
And then he's going to, yeah.
Yeah, yeah, yeah, yeah, yeah.
Yeah, that's kind of the idea.
It's a very, very simple stuff, nothing to complicated,
but you can be like, but wait, wait, he used to describe stuff.
Now he's moving this kind of weird, the thing is like we train it, basically.
And how we train it is that we collect a lot of data,
and then we say, okay, so when I do this,
you're going to wait, and you're going to say hand wave.
And when I do this, you're going to take a napkin,
and then he's going to be able, then we're going to have a training pipeline,
that's going to be able to map the VLM output to those very keywords, right?
But this is just one way to do training,
and hopefully there's multiple ways.
We don't want to fit to just one AI model, right?
We don't want to fit to any type of AI model,
you might have different type of stuff you want to do, right?
And so yeah, but this is all possible with storage,
we have a couple of training pipelines, and so there's kind of the idea.
Okay, what's next?
Then basically what we're going to do is add even more models.
So as you can see now, the robot is only doing pretty fine stuff.
And the thing is like, we currently don't know how to do AI infertile action,
in an environment that he never seen.
It's kind of like extremely hard.
If we in the lab, and I know exactly where stuff is,
and the light is always constant, I will be able to have
predefined movement, and he's going to be able to grab stuff
in a way that he's kind of generalistic, but we haven't figured out
this on the completely unseen environment,
so that's why I can't really show it.
But we're working really hard, okay?
And we're getting there, and hopefully next year,
we could have something like this, and we also want more robots.
This one is a rich sheet too.
There's some part of this open source, and some part
is not open source, and I know that we at first
them, so hopefully we can have even more open source stuff.
And we can directly from the paper installation,
it just work on your computer as well, and add even more demos
from speech to text to text to speech.
I said text to speech.
I can maybe show you later, text to speech demo.
And we also have, yeah, latency optimization,
and make draw more mature.
So thanks for listening.
I think I may be early a bit, yeah, okay.
And so yeah, so this is first them 2024.
It really helped us a lot.
So we're very, super happy to be at first them 2025.
And if you want, I can do just one last demo.
And I can show you kind of a speech to speech demo, hopefully.
And so it might, yeah, my screen is sharing, okay.
And so, so the thing is, the thing is, we have kind of
dumb down the robot to be able to move, so he's not able to
to talk to us anymore, but we also trying to fix this out.
If we, basically, if you talk to the road,
it's just going to say wait or like, hand away.
And so that's not really what we want.
But I can show you with Quen Vial, if I say hello, how are you?
And so basically, I can have, is working.
What do you see?
Stan's in front of a chokeboard with two written on it.
A person stands in front of a chokeboard with two hundred and two written on it.
Yeah, so I'm kind of, I wasn't really too prepared.
But basically, the idea is that you can ask him stuff.
And I think there's a lot of noise.
And then he's going to be able to reply to you in a way that it's
like, comprehensive.
And you can put in a speech deck to speech node at the very end,
so that he makes noises.
But like, he's kind of like looping on himself now.
And kind of not really working.
OK.
What is the men doing with his hands?
Man's holding up two hearts with his fingers.
So yes, that's kind of the idea.
All right, that's all I have for today.
Do you guys have any questions?
Yes, we're standing in front of a chokeboard with a number two
right now, right now, right now.
So what is the hardware board that's processing this?
Is it hardware, go?
The board on this computer on.
Also, you have a small computer on this,
but just from the manufacturer itself.
But the idea is that the software and the AI can run on any computers.
Hopefully, if the computer is able to run it.
So if it's Linux, Mac, Apple, or whatever,
we're trying to make it run anywhere.
Yeah, yeah, for the software sign.
Yeah.
That's good for FB.
Your project is trying to stop a similar problem.
FB is trying to keep forward FB.
Yes, so, so, so, so, so, so, so, so, so, so, so, so, so, so, so, so, so, so you
question is about, is am I trying to kind of do the same thing as Andre did for phones
on robots, kind of, right, I wouldn't like say this because, you know,
but Andre is a bit pretty big, but the idea is like, at the very least,
yeah, we we really want to make it, so that it's very easy for anyone
to pipeline stuff and make it run and be be able to people install it.
Hopefully, everything is open, so, so, so, so, so, so, so, so, so, so, so, so, so, so,
If people want to have, like private closest thing, feel free
that's totally fine, but yeah, that's kind of the idea, and everything is
is, but the big thing is, like, you have to have good communication,
everything is communicating with 잘� nochering, yeah.
Should I?
Do you have the list of open source open source, you've�rian projects?
I don't actually, I could—I was, yeah, but I don't…
There's probably, like, a potato for this probably, yeah.
How did you come around with the performance differences with Ross, and especially how do you feel like this project is going to close the gap with something like Ross?
Because I found Ross to be a very frustrating system to use, but I think a lot of people use it just because it has all these simulation tools and visualization tools.
Completely a bit. So actually I put Ross to you here. It's kind of like if you use Ross to.
So Ross to has really been built for C++ and if you use in Python it's very slow.
So this is kind of a graph of like what the difference isn't in Python with like no, like from Vacy, Ross to with Dua.
And so normally Dua should be very fast compared to Ross to in Python on CC++. They also use trial memory.
So like the performance should be similar.
And but they they have some form of like GPU stuff, but it's apparently it's very hard as well.
The idea that we like I think Ross to was built in a way that it's okay for robotics for robotic people.
And what I'm trying to do is just doing robotics for AI people.
And yeah, but I think they have a very good stuff as well on many different fronts that I may never be able to to bridge.
But the idea is like if we just can do a very good work on AI, I think it's good enough for me. So that's very good.
So I was curious about how you found
As curious how you found your interactions with like the rest of hugging face and the level of robot project.
Do they like call people's work?
No, we were we closely the thing is basically there's another guy working with me called Philip, but he's been.
He has a family thing of good family thing, but he's not working at the moment.
And so I'm just trying to to to make Dua progress, but hopefully in the future I can work more and more with hugging face have more and more integration.
There's a lot of things that's very common. I know there's a lot of things that we also can do with Dua that we can't do with like hugging face ecosystem at the moment.
As as it is in the way that we like for me, for example, more and more with move forward what we want to do is thrust on Python, right.
So for example, the visualization that you saw it seemed to rerun visualization is fully.
This is fully rust based. It's just packaged on Python, right.
And what we want is Dua is to be able to bring all the rust ecosystem into AI people, right.
In a way that's very efficient in some way as well.
And so that's why we really want to integrate both, but like hugging face house or AI model or thing.
So how do we make those two? I combine, you know.
That's kind of what we want to do and we work really closely and hopefully we we can have more.
But as a moment, I'm kind of like not enough time to do it or so that's kind of how it is.
Yeah, we're going to reply. Okay.
All right, so thank you very much for listening.
Very happy to tell you all.