So, now, let's get started.
So, let's welcome Jan.
It's going to talk to us about the O3D project
about a realistic simulator, exciting subject.
Thank you.
All right, hello everyone.
My name is Jan Hanta.
I work for Robotica AI and I'm busy doing open
3D engine simulator.
I'm actually surprised the simulator was mentioned
in the previous talk.
I guess it's because of this talk.
But not many people know about it.
When you open rows to documentation,
that was also mentioned.
Thank you for that.
You see webbot, you see gazebo,
but nothing there, nothing extra there.
And we believe that's a big mistake.
And I said, as a member of Robotica AI
company that makes money out of doing open
source with open 3D engine.
Of course, I will upload at some point
some tutorials on open 3D engine,
but I never have time for that.
So, can you use open 3D engine for simulation?
Yes.
And you can use it because it's open source.
There are open source conference.
So, of course, we are talking about open source products
that you can modify and you can do whatever you want with them.
We do simulations with this engine.
You can think about this engine as open source
alternative to unity or unreal engine
with the small difference that it's in C++.
So, you can integrate your rows directly.
It's the same as gazebo.
It's very, very modular.
All the plugins or parts,
components, however you call it,
are called gems in open 3D engine.
And we are the autos of simulation gems,
multiple of them, to open 3D engine.
So, the good things about open 3D engine
is that everything is in there.
Whatever you want to do,
if it's animation, terrain, editor,
you can use the tools that are building
for games in open 3D engine.
And then we use it to make a scene for our robots.
The game engine itself is closed platform,
but the simulation parts is working on the under the looks.
And the biggest advantage of open 3D engine is the renderer.
It's super fast and it has all the features you might want to use.
When you want to generate realistic data,
it has physically-based renderer
and real-time global illumination
and as well as support for ray casting.
Basically, whenever you start rendering things with open 3D engine,
it doesn't look like gazebo.
It doesn't look like some cubes that are moving around on the scene.
You can really put some materials.
You can really put some reflections.
You can really work on it.
And this is something I presented on the left hand side
with the gazebo imported robot Photoshop
into open 3D engine scene with all the reflections
on the material of antibiotics, animal D, robot.
Of course, the robot itself is made out of slightly different materials,
but our graphic engineers didn't know that.
So the reflections are there,
and the quality is good enough for all AI you can imagine
to process the data to train the models and so on.
And this is basically how we roll in my company mainly.
So you can download the source code of open 3D engine.
You can communicate with open 3D engine,
using Discord.
And if you visit a six simulation channel,
most probably it will be me answering your questions.
The important part about open 3D engine is that it's very modular
and our part, the simulation part,
lives in O3DE Express repository.
So don't forget about this.
There are some simulation gems and projects templates.
My company also has a GitHub, of course,
with plenty of open source related to open 3D engine,
as we are the main contributor to this part of the engine.
So if you start pulling the code,
I also recommend to have a look into our repository.
Many for the demos that we presented at Roscon 2021-24,
2021-free 2022 and 2021.
Those are the build projects.
You can even find some pre-compiled projects.
You can just start it on your computer,
the quality of the graphics, and the quality of the engine.
Starting with your own project is also very, very easy.
You can download the SDK,
so the whole system is made in a way
that you can compile everything from the source,
of course.
You can use the SDK with everything building
or you can exchange some parts with having some parts written
from the source with some parts being modified by you,
and you don't have to bother about compiling everything every time.
For SDK, the only thing you do to start
is you pull the simulation gems,
as you can see on the first slide,
then I have a pointer even.
Then, of course, you create a project.
You use one of the project templates
and you click build.
Hopefully it should run out of the box,
and you can make your open 3D engine project
in less than 10 minutes.
There are three different templates available
for you to start with.
The first one is the one you also saw on the screenshots
in the previous presentation,
is with very simplistic warehouse,
actually that's the warehouse
that was imported from Gazibod directly
with some on-person assets.
It's just one robot in very small limited space,
and you can use enough to
and all the rows stack to manage the robot.
Another template with the manipulation,
manipulator trying to pick some objects from the table
with all the physics of the manipulation,
with physics 5 engine,
the same engine as in Gazibod,
working under the hood.
The last template with the fleet of robots
navigating through the warehouse scene,
the warehouse scene itself is very detailed,
very pretty,
but not optimized at all.
If you put too many robots,
you already need a powerful computer for that.
So how does it work with rows two?
We have rows two gem.
It's one of the modules of open 3D engine,
and it's enough for you to import this module.
You import this module into your open 3D engine,
and it automatically creates rows two node,
as you know, from any rows two package.
The gem itself takes into consideration
all the environment variables.
So your rows two pipeline can work
in exactly the same way as it used to be.
So you create your workspace,
you create your packages,
you build it with co-con,
you source to your rows workspace,
and then you start with the same terminal,
open 3D engine,
and open 3D engine can discover everything
that you've already done.
For our base component,
we use rows two frame component.
This is corresponding to something you know,
a process TF.
So basically that's a spatial temporal representation
of an object in the space,
and for us it also serves a purpose
of providing a name space to the system.
So you can display the TF trees,
you can display all the name spaces,
and all services topics,
and everything you have in there
will get a proper name space.
The rows two gem also implement the sensors.
Those are the very basic implementations,
for example, for camera.
It just displays what the camera sees,
without any extra distortions,
without any extra noise models
that we can apply later on to make our simulation
even more realistic.
There are also sensors like contact sensor,
of course GPS,
IMU, LiDAR,
Automatry.
LiDAR can work on either GPU or CPU,
and Robotech AI has also
LiDAR implementation for gazebo,
that is definitely the fastest on the market,
open source available to everyone.
For Robotech Control,
Robotech 2 package contains of the
acarmon control, twist control,
and a very basic rigid body control,
so we can just move your robot around as you wish.
We've simulating the robot fully,
using acarmon,
or simply with moving this as a break.
It has a full control of joints,
it can display the state,
of course virus to services messages topics.
It's exactly the same one as you would use with any other simulator,
and it happens only after you include the gem,
everything is available for you.
There is a screenshot very similar to what you saw
in the previous presentation,
basically with move it to,
and this move it to is connected to the template I displayed earlier,
the template that was made to try to manipulate
with some objects.
So there is a camera on top of the gripper,
next to the gripper,
to see the object from the above,
and move it can actually integrate with
our simulation without any problems.
Robotech 2 gem includes also implement
also the importer,
so if you have your simulation ready and gazebo,
you can import your robot.
I'm not going to say it will work out of the box,
because O3D is slightly different than gazebo.
Everything under the hood is different.
All the sensors are different.
There are some different options to configure.
So of course it will help you to start,
but it will not solve all the problems.
So the importer itself, again,
can use your rows to sourcing,
so if you have any rows packages connected to your robot,
if you have any specific sensors,
with specific implementations connected to your robot,
the importer will be able to read it.
The importer supports Sakura and multiple mesh formats.
There is a general open-free engine,
it's not well documented.
I'm doing my best to fill all the gaps between
the projects I'm doing for different companies,
but there is a tutorial about importing turtle but far,
and all the problems that may happen
actually happen when you import this robot,
so it's a very good source of information
and you want to try with your own robot.
Since we are trying to compete with NVIDIA,
of course this is not easy,
we are also implementing USD support,
and probably it will be done before the end of the year.
So I'm a C++ guy.
I don't have any experience with Python.
I know that there is a lot of implementation happening
with Python rows, especially in academia,
but we want to be fast efficient and we want to use
what open-free the engine gives to us.
And open-free the engine gives us the possibility
to implement rows called directly within the code
of open-free the engine.
So as a small example, I decided to show you
how to implement a gem,
applying to open-free the engine,
that would publish the current light entity via rows two topic.
It would also subscribe to a topic to allow the changes,
and would also create a service for changing the same intensity.
Don't get fooled by the picture.
It's just different colors with the light rendered
in open-free the engine.
We are changing the intensity.
So for two stats with the implementation,
the only thing you need to do when you create your gem,
that you can create either via a graphical user interface
of open-free the engine or just with some scripts available
within the engine, Python scripts.
You get your new gem.
You need to connect to rows two gem,
so everything is available for you.
All the goodies that are already implemented
that are open source are available for you.
Since open-free the engine is implemented in C++,
but it's also available for iOS, Android and so on.
The whole build system is, of course, a CMake,
the most popular right now, I guess.
And what you need to do if you want to connect
to rows two gem additionally,
you of course need to link to the library.
That's built in the beforehand.
Via CMake of rows two gem,
we also let you connect to your rows two implementations.
In this example, I'm not going to use any specific implementations
other than standard messages and standard services.
But as you can see, this magical command target depends
on rows two packages would just do everything for you.
And by doing everything for you,
I mean that if you create your header,
you can use includes, you know,
from all the tutorials on rows two directly.
Everything is available for you.
Everything is there.
This is a header of the component.
I blurred out a bit because I didn't want to bother you
with details that are automatically done by open-free the engine.
But if we want to create a publisher,
a subscriber and a service,
the only thing we need to do is to include the headers,
as we would do in any rows two package
that we implement standalone from open-free the engine.
And we would have to, of course,
create or define at this point only.
The service, the subscription, and the publisher.
There is no difference between this code
and between the tutorials on rows two documentation website,
other than different names.
In this code, in this example,
I also connected my test component to tick bus
to be able to answer to every tick in the simulation
because I wanted to make it very simple.
And I wanted to be sure that I publish a message
on every tick of the simulator.
And I created a helper method set lights intensity.
And I connected to the entity with the lights.
Because this is something I didn't mention.
This simulator works in a way that you define objects,
like entities.
And those entities have some components,
which might be, for example, a sensor,
a texture, a collider, like in most of the simulators.
For the code itself,
we implement everything in the activate method.
So, every time we activate this component,
we want to have our publisher subscriber
and the service ready.
To start with service, first of all,
I want to get access to rows two node.
And rows two node is already in the system.
After everything is linked with rows two gem,
it's a single tone, it's available for you.
And since we are not implementing stuff in Rust,
as yesterday's presentation mentioned,
we also need to check, of course,
if there is no new pointer problem.
We also try to get this basic component
that I mentioned earlier, the frame component.
And we want to get it to double check the namespace
of our service that we create,
because when we spawn 10 or 15 robots,
we want to be sure that they get the proper namespaces
and messages from different cameras can be observed separately.
For creating the service itself,
it doesn't differ from the code you know.
So, we get a node and we create a service.
The service here is very basic.
So, we use the trigger, standard service trigger.
We don't care, of course, about request,
hands may be unused.
We only care about response.
And there are responses, of course,
false, if we cannot set the light on,
and true, in case of success.
So, first, we check if the light entity is there,
because we need this object with light.
If the lights are not there, we cannot switch them on.
So, we can give an extra hint to the user
within the service response that the lights
are not there in the scene.
Other than that, we use the internal method
set lights entity, intensity sorry,
to change the light intensity.
For subscriber and publisher,
again, it stays unchanged when compared to any rows to package.
So, we just use the node to create subscription.
In this case, we will use float to give the defined,
predefined intensity.
And as you can see, we use this helper methods from rows to jam
to get namespace to name of the topic.
So, we get the namespace, we add the current topic setting.
And we set the QOS, which also comes from the subscriber configuration.
It's a nice tool that is already implemented there
within rows to jam, that when you edit your objects
in the editor of the simulation,
you can just use the drop list to select the proper configuration of QOS
and so on.
So, this is just a callback that uses our internal helper
method set light intensity.
In case of publisher, what we do is just the same stuff,
so taking the name and taking the QOS,
and we create the publisher.
Some are there down there in the code,
when we implement the ontic method,
on every tick we publish a message,
we read the intensity of the light from the O3DE rendering bus.
So, O3D works in the same way as most of the engine
that most of the game engines,
that it defines buses for communication,
and you can communicate via those buses.
You use the entity as the address,
and you ask what is the light component
within the entity,
and what is the intensity of this light component within the given entity.
We get the response that started in this current intensity,
that was pretty fine to minus 100,
just in case before we create a standard rows message
float for T2, where we put the intensity,
and we just publish this intensity.
So, the code stays exactly the same as it will stay
in a standalone rows package,
and from my perspective as a robotic developer,
this is the biggest advantage of open 3D engine,
that I have access to everything directly,
there are no bridges, no man in the middle,
so basically one component less to maintain.
There is a bonus,
because I mentioned that in open 3D engine,
we can use C++, we can use Lua,
to some extent it's also possible to use Python,
but there is also something called script canvas,
which is just an interface for drawing the pipelines of your choice.
For example, you can on start of the graph,
on start of the pipeline,
when this script is initialized,
we initialize the object that is responsible
for triggering some messages once in a while,
it's called hardbeat,
and every 100 milliseconds on our pools,
we will publish a message to Topic Hello,
that is called Hello World.
So with such a simple script,
we don't even have to type C++ code,
we can just get a script canvas and we can draw it.
Very fast for prototyping,
very fast for doing anything you want to do.
Even when doing commercial products in my company,
we use a lot of that because it's simple and fast.
The same stuff can be done using Lua,
with the only difference that this Lua one-liner publishes,
the message once I didn't write the full loop here.
So yeah, actually this is very, very brief presentation
of Open3D Engine.
I was afraid I would run out of time,
but I recommend to have a look into it
if you want to get nice rendering,
if you want to get nice effects,
because this is what Open3D Engine gives.
And since I have one more minute or even two more minutes,
I can show you a short video with a track
going through the mine facility in Chile.
This is a digital twin of huge mine facility.
Maybe it's not perfectly visible when it's under the ground.
We use it for simulation.
We use that for our client to provide answers
where the tracks should be in the mine.
This is the demo from Roscon 2020 free.
We did it to show that Open3D Engine can handle a lot.
There are 12 robotic arms here.
There are 56 AMRs all equipped with LiDAR sensors
and camera sensors.
So there are 72 LiDAR's more than 50 cameras in the scene
and it works on my notebook in a real time.
This is another demo.
Okay, those are recordings,
but it could work on my notebook in real time.
The agricultural demo, the demo with some robots,
I decided to show you this video.
Mainly to show you the quality of the rendering.
Of course, you cannot see it well in the screen,
but displaying it in a 4K screen,
having all this data at the same time
and running it on notebook.
Okay, with Nvidia graphic card, of course,
but still notebook, it's still very, very cool.
So yeah, I guess that's all.
I will go back.
Thank you.
Hi.
I just wanted to ask,
because one of the biggest reasons
are sometimes in industry
people choose to use game engines instead of a zebra,
for example, not because of the simulation of the robot,
but because if you want to do highly dynamic environments,
it's an absolute pain to do that in a zebra.
So I wanted to ask how easy it is to, for example,
add tons of actors, moving parts,
things flying around, things that add the simulation in.
It's perfectly doable.
We have it in our simulators for our clients.
Some parts are available, open source,
some parts, unfortunately,
we need to hide behind the paywall.
We have levels maybe made for agriculture,
where we have more than 100 people walking around
and automated way with all the robots moving between them.
We simulate different weather with rain, snow,
and other things falling from the sky.
We simulate fogs.
Everything is possible with open 3D engine.
I'm not saying it's easy.
I'm saying it's doable.
And I don't know other game engines,
but as a C++ person,
for me implementing staff and open 3D engine is easier
than for example, new unity.
I don't have comparison with Unreal at all.
But only with unity.
We also did many things for
outdoor foundation with unity.
Simulating the city, my company is responsible
for digital twins of Japanese cities.
And we are porting that to open 3D engine
and it looks way better and it works much faster than any unity.
Hello, Jan.
Thank you for your presentation.
Do you have any plans on integration with the drone code stack?
We don't have drones,
or the objects that are just moving on splines in the sky.
This is the biggest missing point of open 3D engine.
There is no wind, no water simulation.
So unless you simulate it yourself,
what we did in one small project was, as I said,
it was just an object that was moving on a spline
with some additional noise to it.
But we didn't have anything more than that.
So you're not planning to integrate it with PX4.
For the moment,
we don't have projects like that with clients.
And this is how we roll and how we try to support the open source community
that we always do projects that can give us some money to live.
And we try to publish as much code open source as possible.
So as long as we don't get any projects,
we won't have the capacity for that.
But of course, it's open-source.
We are invited to help us.
And then maybe we will get someone to help you.
Thank you, Jan, for the wonderful presentation.
Since I just started to mention the AI and similar stuff,
would this, did you ever try or would this be able to handle,
I don't know, some really fast image object classification or object acquisition?
Because I see it can be used there.
So this is just a simulator.
And what happens here is that we produce objects.
Sorry, we produce images.
And those are the images that you can analyze outside.
You can get those images the simplest way would be via rows two topics.
And then you can do it outside.
You can send the response back what you detected was happening there.
You can also integrate this into the code base of open 3D engine.
This is one of the core parts of money actually to generate images for machine learning
training and verifications and validation.
Really realistic for that.
Thank you, thank you for the response.
And just to say this might be a really good alternative for my club's plugin
for Unreal Engine.
This could change it.
And the pictures look even nicer than there.
Thank you very much.
I believe there is still a gap between Unreal and open 3D engine.
But the nice thing is that here you get the source code.
You can change anything you want.
And just to your needs.
And then if possible, if the customer allows you publish it.
So to the community.
The predefined objects looks most nicer than the five objects.
Okay.
Very good.
Thank you.
Okay.
Thank you.