And thank you everybody for having us today.
So I'm Steve Fan, Steve Fan Caron.
And with me, it's Anahllo, and I don't have a touchman
diva vacer, who will show you live demo on the robots
in the end.
And so of course, with me, are Michel Scogov, the black robot
you see here, which is a new app key, and cookie, which
is the yellow one, which we can talk about afterwards.
All right, so before talking about these robots
in particular, I want you to share with you,
like one point of view on open source robots today.
I like open source robots.
This is a good place to say that.
I'm trying to curate a list of open source robots.
If you know some that are not on the list,
please feel free to add them there.
And I just wanted to kind of take a look back as to why,
also we can now we are in a better position
to do open source robots than we were in the past.
Here are a quick view of some of the projects.
NASA opens sourced a robots, and they're Apache 2.0,
which is like a rover that has excellent documentation.
They're also educational open source robots,
like the robot soccer kits that are used for education
purposes.
It's actually quite fun for kids in mid schools,
for instance, to control these robots in Python.
And more recently, there was the Open.mini,
not meant to resemble any particular robot,
but it has been heavily reproduced,
like more than 50 times, which in my opinion is amazing.
And this is a bipedal robot, also a bipedal robot.
And everything is open source,
you can train reinforcement learning policies for it,
UV run train, and it will train one for you.
It's also very cute, et cetera.
And so I was wondering, so I've been doing robotics
for a bit over 10 years now, and I wonder,
like, how come we have this kind of a comprehensive
explosion of open source robots now?
Like, how did we get there?
How was it and what changed?
And this is just one point of view,
but looking back at 10 years ago.
So personally, I've been working in academia.
And in academia, we were working with very expensive robots.
So this humanoid, you see here,
costs north of 300,000 euros.
It's expensive, you only have it in the lab.
And when you want to operate it, there is a whole process,
because you don't want to break it.
So you have a crane, et cetera.
There were open source softwares already,
but the hardware was not open source at all.
Then, I think one big change came with Bencats master thesis,
who introduced the QDD, what we call now QDD actuators,
or the appropriate safety of actuators.
So the combination of a brushless motor and planetary drive
train.
And it was also fully documented on Benz blog,
which is a great blog to read.
If you haven't, it's really a pleasure.
And it helped reproduce the actuator heavily.
Reproduced, including by I'm going to cite one company,
because I think they've been doing an amazing job.
It's called MJBOTs, it's an American company,
and they do open source motor drivers,
and also integrated actuators.
And so these robots, basically, were made possible,
because we could buy the electronics and the actuators
from that company and then develop software on top of them.
And MJBOTs does thing open source as well.
So the actuators, the secode of the motor controllers,
you can access their driver board is a hatch.
You can put on the Raspberry Pi, which we do here.
You have the firmware, you have C++ Python libraries,
and the supports is implemented.
The state of the art technology, known as Discord,
where you just pop in and ask a question,
and the government jables, it's just there,
and then it's directly there.
All right, so these took us to upkey wheel bipeds,
which are one way to apply these actuators
and motor driver boards, and we wanted to make it
as a wheels biped.
So it's kind of a in-between the mobile robot
with a fixed base or the car, and a humanoid robot.
So you have, in less actuators,
you have the problems from both worlds.
So you need to balance, you need
to balance, because you have wheels.
So if you don't do anything in your forward direction,
in the sagittal direction, it's kind of fall.
So you have to do active balancing.
So if you want to do research,
you still have the same research questions
that you need to solve for humanoids.
But it's only six joints.
So when it breaks, well, maybe you have to replace a motor.
I mean, if it falls, it can fall on the ground
and stand back up.
We added the bumpers in one iteration,
but even if you don't have the bumpers, it's OK.
If it falls on you, OK, it's going to bruise you,
but that's it.
Happened in the past.
And then, yeah, so it's kind of a small form factor,
but it allowed us to do plenty of things.
So the goals of the project was to be fully open source.
So the hardware and electronics are open source already.
We build on that, and then the software we build on top
is also fully open source.
These hybrid locomotion capabilities, I mentioned,
so worst of both worlds.
We want it to be reproducible and accessible.
So let's parts.
I mean, don't use too many actuators.
Try to make it a bit easier so that people can build it.
And then the two outcomes for us are education and research.
So for education, we can use it as a way
to teach model predictive control, so more on the control side,
and also reinforcement learning, so the newer techniques.
And we build on open source libraries for that.
And then, for the research, also,
it's something that's cheap for a research lab,
but that allows you to run experiments with new technology.
So I'm going to talk about the software first,
and then Vanolstein and Etienne can show you
more about the hardware.
So the software, I have to say, doubt that first,
it's not using Ross.
This is we kind of wanted to make something
that you can pip install, and it
could just run directly on your machine.
So as a matter of fact, if you want to try it out,
you can get Chrome, go there, and you
will run the example.
It's going to fire up a bullet simulation,
and you can apply forces to the robot, et cetera, directly.
So we want it to be something that beginners can directly
access.
I'm not saying this is not possible in Ross.
I don't know how to do it, but it might be possible.
And then we want to use it in Python
to stay in a simple world.
So develop new behaviors in Python,
unchecked with all the mess you can do,
and then the low level that communicates with the actuators,
runs in C++, because the MJBot API was in C++.
There is IPC between the two, and every time there is IPC,
actually, we log everything.
So they exchange dictionaries, and these dictionaries
are fully logged, so that then we can use them
for learning from real robot data, for instance.
And I wanted to mention the gymnasium API
we're going to come back to that.
Interface to different physics simulator.
There are actually some connections with the presentation
we had before on the Zebo.
We also had the questions of what physics simulator
to use, and we're going to come back to that as well.
So if you want to try it out, feel free to get clone,
and run.
This should work.
Gymnasium API, we wanted to use a standard API
from reinforcement learning, also,
because it fits well with the control, standard control.
So if you go to gymnasium, they have a bunch of standard examples,
like the card pool here, and it works basically,
you make an environment to expose your system to a user,
and it should have a reset function and a step function.
So initialize and execute.
And we stick to it.
So if you want to switch from the card pool to an app key robot,
you only change, if you look at the diff,
you change the name of the environment,
and you change the observation instead of being a position
for the card pool is going to be the pitch angle,
for instance, for an app key.
And that's it.
So we hope that this makes it easier for newcomers.
They can just come, look at the examples,
and it's basically the same.
If they had reinforcement learning during their education,
for instance, it's going to sound familiar.
And then, this allows us also to connect
to stable-based lines, for instance,
or reinforcement learning libraries in general.
So reinforcement learning libraries,
I'm going to show you an example with stable-based lines here.
And essentially, what they will want to do,
so they will use the simulator to generate samples.
And then you will use these samples to train your neural network
that will commend your robot, and that's called the policy.
And so the way it works here is that it spins multiple instances
of the robot, as many as your CPU can handle.
And then it's going to apply PPO, which is an RL algorithm here,
to train a policy for the robot.
And this is going to take some time.
So I'm not going to let it fulfill.
This particular training would take about maybe 10 minutes
on my laptop.
If I were not on battery, I would leave
it running in the background, but here.
And it would be really slow here, because I put the GUI
in practice, you'll never show the GUI.
And we interface with RL-based lines to do with Zoom,
which was also provided by the developer's stable-based lines.
And basically, if you have a German variable for your robot,
you can go to this library, connect it to your environment,
and ask it to train policies for you.
And it will do it directly.
So, oops, sorry for that.
In the previous version, we implemented that logic ourselves,
and then we figured out someone did it better than us.
So now we use that.
So stable-based lines is one.
There is also this nice contribution that came
from Magic Crystal, who is someone else in the academic community,
using MJ Lab.
And so this is another physics simulator,
which was mentioned before as well.
And we'll do a local warp, the GPU version.
And this one can run more environments in parallel,
because it runs on the GPU.
And in today's technology, this is faster.
So I will show you not the training this time,
but if you want to try it out, also,
mark made it really simple that here, this is a script,
but the script is just because I'm afraid
I will forget the command line.
You'll be run Python soon, right?
And here, so he prepared this,
where basically here, you have a module
of simulation that's running live.
And then from the keyboard here,
I can command it to go forward or backwards or turn.
And you can see that that policy actually
learns to use not only the wheels,
but also the rest of the legs.
So it's actually much smoother than the policy we are going
to throw in the end, which is going to keep the legs straight.
And this one also is a neural network policy,
so that was trained with a large amount
of simulated data in mujoko.
And so as well, if you have your own robot,
or you have other use cases, so MJ lab app
issues use how to apply it to this robot,
but if you want to use other robots, you can use MJ lab,
make a gymnasium, or basically,
even they don't use gymnasium, I think, actually.
Connected to MJ lab and use it to train policies
for your robots.
That's it, about reinforcement learning.
So I wanted to kind of share some examples of things
that worked and didn't work in the project so far,
so the project has been running since 2022, roughly.
We have been very happy with open sourcing everything.
Again, very easy to say here,
but this means people will just pop up.
Basically, I didn't expect
and I think nobody expected really
in advance what would work or not.
So for instance, some of the things that didn't work,
I expected at first that people would want to reuse
the kind of C++ interface to communicate with MJ
about actuators.
No, and that made the code more complex to build,
because every time we needed to change that,
then you had the cascade of projects to update that was a mess.
So eventually we just merged everything
into a single project as much as we could, and that helps.
So it's simpler for us, and nobody really cared anyway, so far.
We open sourced the hardware, the 3D printed parts.
Some people would design it entirely, so you know.
I mean, we don't know in advance what's going to work
or not for other people, so very happy about open sourcing.
We were very happy about switching to Pixie.
Oh yeah, in the things that didn't work,
I should mention the Wi-Fi in the network.
I mean, this sounds trivial, but actually,
yeah, we had trouble with that.
This robot has drew all like it sets up in access points,
and it tries to connect to the Wi-Fi.
We made it too complicated with the Wi-Fi.
That's my fault.
But then when we switched to Pixie, you had this tool called Pixie Pack,
so it will prepare your whole Python environment for you,
and upload it to the robot, whether it has internet or not.
That was hugely helpful as well.
So Pixie also was cool.
And then the model predictive control, which we are going to show you in the end,
that was cool as well to be able to run model predictive control.
It, the main benefit is that it has two parameters.
So when we switched to a different robot,
even these two robots, they have the same structure,
but they have different masses, different geometry.
The MPC just works as is.
You basically need to update the wheel radius.
That's a biggest parameter.
Everything else, we don't need to adapt.
So that was very convenient to make new robots.
And yeah, and things that didn't work here,
we also tried to reuse the same code for the rear robot,
and for simulation testing, and for reinforcement learning,
that was too greedy.
Now we have the couple.
For reinforcement learning, you live in a different world,
you train a neural network,
and then just take this neural network and run it on the robot.
That was much simpler.
Anyway, so some applications that have been tried with these robots,
adding an articulated head.
So if you look on the top of the robots,
and for the new version, even on the front,
we have a four-screw pattern, so that people can put add-ons.
So basically, you 3D print a part,
and then you can put the camera,
you can put the handle, et cetera.
So one project did an articulated head, two extra motors,
and the application was tracking a narooka marker.
So someone moves around in the room is a marker,
and the robot just follows you.
We've been applying it to estimated contacts.
That's more like a research question,
but this robot does not have a wheel to ground sensor,
so you estimate it indirectly.
We've been using machine learning to try to improve
on that contact detection.
And then Vanhonten, around here, worked on obstacle avoidance,
and so he can tell you if you're interested
about how to use, so Gaussian splatting
to generate images for your simulation.
So they are photorealistic, as far as a robot is concerned,
and reinforcement learning to train a policy,
and then we can use the camera so that if an obstacle
comes in the field of view, the robot will correct the velocity
that the user inputs, but basically, this is something
that with an implicit model,
it doesn't have an explicit geometry of the environment.
We can talk with Vanhonten if you're interested.
Just hand it over to you, and then in the pipeline,
we want to execute some more dynamic motions,
like pre-recorded motions that people can design offline,
and then run on the robot, so like jumping would be cool.
And then also stream line the way to use vision
when you move, not just proprioceptive,
so these two robots here have cameras, for instance,
that will be developed more in the future.
And now I will turn to the hardware,
so maybe I will give like a high level of review,
and then you guys can take over with a demo.
So for the hardware, we try also to make it easier
for people to reproduce.
So we have a bit of manual with pictures,
never enough, but some pictures, a bit of material,
3D printing parts, I mean,
get repository using LFS for the free CAD parts.
Now, thanks to these guys now, the parts are designed
with free CAD, it was initially blender, my bad.
And yeah, there are variants of these robots.
So I will show you, so cookie here is an app key T,
which was designed by Mika and Mathieu.
And then this, that you see on the right,
is actually a complete redesign of the robots hardware
by Martin Krizo, because when he built it,
he was like, oh, I actually have my own preferences,
so I would just redesign everything.
So he re-improves again and shared it.
This is also it's shared on shape,
and you can find it in the readme of Marx project.
So quickly about the electronics, when you build the robots,
so the here is a battery sure, the battery plug.
We connect the standard, like a re-obi battery
to an MJ bot's PowerDisk board,
and that will then supply power to the Raspberry Pi via the hat.
And then this Pi is connected to the motors via twisted cables
for it uses the KNFD protocol to communicate with the actuators.
And then we don't go beyond that.
I mean, the MJ bot's API takes care of us
for us of sending packets to the motors
and handling them.
You send the packet, you get back the sensory measurements,
et cetera.
And yeah, just in pictures.
So we didn't plan anything for the hardware,
so it's not like a linear progression.
Of course, it's a tree of what people did.
So here you have various robots, rookie,
Michel Sorgoff.
This is Nadia Fedor.
This one is a name, Michel Sorgoff B's cookie,
and then this one is called Blueberry Service.
And yeah, we hope it continues.
So cookie here, you will see in the end,
it goes in the direction of adding more features.
So it's a cooler robot to use,
and for instance, adds charging of the robot directly onboard.
So you don't need to do it separately.
It has bumpers, it has a screen on the robot.
So you can interact with it directly without your computer,
but then it's more complexity.
So it's harder to reproduce.
And even when it breaks, I know it will take time
to open things and find the cables again, et cetera.
But when it works, it's a pleasure to use.
And for upkey V2, I will switch over to you guys,
because the two designers of upkey V2 are here.
So.
OK, so very well.
So I'm using a team for these two years now,
and they made it.
And so we had some, like, real-time one things.
They've been, they've been working for upkey V2.
At some point, we wanted to do a computer design,
just great time.
Because the previous part were design blender.
So to iterate on a part and modify it,
it was not easy.
And with Freakhead, we could use
parametric parts to modify it easy.
And I don't know if we want to improve,
like, increase the length of the legs or something.
We can do it.
It was also wanting to make the robot less complicated
in the complete other direction than upkey T, I think,
I believe, it's much simpler.
We had way too many parts for now on the main parts,
or one torso, when the one lower leg and two
because two sides.
And so it's a much simpler robot, much easier
to operate with, because everything is accessible,
even the cables are not internal, but they're now outside.
And yeah, improve the durability of the robot too,
because we had issues of tbs and females breaking.
And so the new redesign, which has a sort of fog design
for the apparent lower leg, makes it much much stiffer.
For instance, it hit a wall at some point.
And we can see here.
At the part, slightly broke, but it's not a complete failure,
and which could not have been possible in the previous design.
I think we can launch the end school.
Yeah.
Let's end with a live demo.
And if you guys have questions, feel free to.
Demo effect might pop in.
So what you are going to see here
is the default MPC controller
that we're also distributing in the robot package.
Yeah.
And we have a leash.
Yeah, we have the little leaf, just in case.
So if we don't have time for the demo now to switch
to the next presentation, you would probably see us around
also today with the robots.
So don't hesitate to, yeah.
Don't hesitate to interact with a happy tool.
Yeah.
And so the fact that it has legs is for,
come and take a visit for crouching,
but hopefully in the future, I hope we can
climb these stairs, for instance, that would be cool.
But we will see.
Maybe I should showcase a bit.
Maybe you can stand here.
If you want to try later on to come and pull it
and see how much it can balance, feel free to.
And so this is the MPC, it's a QP that solves the real time
on the Raspberry Pi with a reverse gradient matrix
to have the legs be able to bend and go lower.
And you can feel the perturbate and integration.
Exactly.
And it's still open source software.
So the QP solver is called ProxQP.
It's open source.
And yeah, the inverse kinematics and MPC
are also open source libraries.
If you are interested, we can talk about it
or you can find it from the project report.
All right, thank you very much for attention.
Thanks.
Thank you.
I think we might have time for one question.
If you're OK.
Thanks a lot for presentation.
Can you quickly talk about the reward function
you are using in reinforcement learning?
Yes.
Variables are you trying to optimize during it?
Yeah, sure.
Thank you for this question.
So indeed, when you do reinforcement learning,
all when you do the QP, you have to define the objective
function.
In reinforcement learning, the reward has three terms here.
The main term is you take the point at the top of the robot
and you want to keep it where it should be.
So basically, you have a reference point
and you want to stay there, which means if you're
tilting, you have to tilt back.
If you're translating, you have to translate back.
This is the same in MPC and RL.
And then in RL, we also add actions smoothness
for regularization.
And the action here is a ground velocity.
So we try to minimize ground velocities.
And there is probably a third term that I forgot.
Velocity dumping, maybe?
I'm not sure.
It's in the code.
It's right here.
I can check.
I mean, it's good.
Yeah, yes, yes, yes, yes.
OK, it's somewhere in the code.
Thank you, thank you.
Thank you.
All right, thank you very much.
Thanks.
Thank you.