You're becoming, you know, it's been a long day and I know tomorrow is going to be another
longer day, who's looking forward for the after parties today?
Yes?
Who's enjoying Brussels?
Good?
All right, I need to break the eyes because I don't know you.
This is the first time I've paused them, so I want to get to know you.
All right, so first of all, this is who I am, and I'm going to apologize for those
in the recording, especially because I move a lot.
And I like moving a lot, so it helps me deliver better, and I think so.
I speak too fast, so let me know if I'm not pronouncing correctly.
I'm Mexican, my primary language Spanish, so I have a hard accent.
Let me know if I'm not pronouncing anything, or you don't understand what I'm saying.
So I worked with the Linux Foundation, hence the Penguin.
I am the general manager for Drunkot, Drunkot is a project within the Linux Foundation.
We specialize in drone stuff.
I am initially a contributor, so I'm not a business developer or a sales or marketing person.
Although I had to be training that because the position required it, but I was initially
an individual contributor, and I'm still I'm, and that's one of the reasons why I'm here
because I actually need help, and that's kind of like the reason why I'm here to recruit
some of you.
I've been working for more than 10 years in robotics, some of my colleagues are here,
so if I say hi, I call it the Roth Aerial Robotics Workgroup.
Has anyone here been working with robotics at all before?
Okay, like a quarter of you, nice.
Anyone know about Ross?
Good.
All right, so there's an Aerial Robotics Ross community workgroup that I call it with Kimberly
Maguire sitting right there.
And then I also call it the Spaceware Linux Initiative that started last year in which we're
trying to build a Linux distribution that is certifiable for space, and we're working with a few
companies, NASA included, and that's a really cool effort.
Hopefully next year at 1226, I get to talk about that, but today we're going to be talking
about drones.
All right, so what is drone code?
So we're an open source UAV ecosystem, but honestly, when you hear about a foundation,
like an open source foundation, what is the first thing that comes to mind?
Like anyone?
No, my.
Yes, that's a very good one.
It's like the funding to the project, right?
But it's a bit more than that.
And sometimes some of the foundations miss the opportunity to tell that story, and I'm
not right to do a good job here, but this is like the second time I do it this time.
So we're trying to be a neutral home for the open source projects.
What does that mean?
It means that any company within the ecosystem can contribute, even if they're competitors.
And that causes a lot of challenges.
We're here to promote open collaboration, open standards, open source, and accelerate the
innovation in the industry.
Robotics industry has been around for, I don't know, for longer than I existed, probably.
But it's been very slowly moving towards commercialization.
To the point where right now, you see a human, I robots, and like their famous with AI.
But honestly, there's still a kind of gimmicky.
There's not real applications yet.
Like I haven't seen one that works perfectly yet, or that has been deployed out there.
Maybe I'm wrong.
Where's your hands if I'm wrong?
You know, an example?
No?
All right.
It's good that I'm still right about this.
And we want to support commercial adoption of open source, which is the biggest thing here.
Because if there's no commercial adoption, there's no industry, there's no industry.
How are we going to fund this, right?
Because initially, this is what we're trying to do here.
We want to make sure that the project continues to be alive.
So what we form here is an ecosystem.
So drone code sits at the center.
But we have the open source projects.
These are some examples I'm going to talk about the rest and develop it.
We also have industry standards, which is like an open hardware project, a map link, which
is the standard for communication, a protocol, which we can define the behavior of certain
components of drones.
We have industry members with some of the companies that actually support us.
And actually contribute back.
And industry associations, like the Linux Foundation, the open source robotics alliance,
which hosts Ross, and we work with them, or the OpenCV Alliance as well.
So we try to put everything together and make sure that there's a concise industry.
And then there's a network, a concise, sorry.
There's a network that we're trying to push things forward together.
Why am I telling you all of this?
Because like you just saw not a lot of you know about drones and I just wanted to give
you a quick context and what's going on right here.
We've been around since 2014, but the projects have been around since 2009.
It's not something new.
It's been around for a while.
It actually was born out of ETH in Zurich.
But it has grown tremendously out of that.
To the point where I just pulled the analytics and we have already 13,000 unique contributors
over time.
So even though I say we're humble as more community, we're growing over time a lot.
But we're struggling with some of the things that will allow us to grow more, like
CI.
Just some more stats, so you know exactly what's happening here.
Almost 60 million lines of code distributed across all the repositories.
Which are almost like 100 repositories.
We have 20 companies that are supporting us and we have the top five projects which are
there.
PX4, which is the firmware for drones, picks up, which is open hardware, the flight controllers
that actually control the drones, map link, the message protocol that I was explaining, map
SDK, which is an SDK that allows you to talk map link because map link is basically
an XML protocol.
So map SDK is a wrapper on top of that that helps you build APIs to control it.
And Q1 control, which is a very interesting project based on QT, and it's a mission planning
software or a ground station, how we call it, which allows you to control the freaking drones,
which is pretty cool.
All right, so let's hope the internet works.
2009, this is where we started.
And it's important that we see where we started.
Has anyone heard about PX4 or PX4?
Yeah, just a few.
So this is like, look at the machine, like, that is actually an Intel NUC.
Like, very old Intel NUC.
Those are parts put together from off the shelf in the electronic store.
There's nothing secret here.
This was like, a lot of pieces that were put together from electronics that were already
available at the time.
And we started from there.
We moved a little bit further in a few years because it was a basically a research platform.
And as you can see, you have more stable flight now.
It's still a NUC machine, but no GPS flights now.
It's autonomous flights.
We have plane detection, a non-board camera.
So we started getting tonsils, putting more features on top of this.
I'm not going to play the whole video.
I'm going to share this slide later, so you can see it at that home.
But this is one of my favorite videos that we have.
This is actually came out in, or developer, about into 1222.
Look at the configuration of this drone.
Look at how it's going to be moving in different directions.
This is an opening copter.
So PX4 allows you to fly the vehicles that you saw already, and then PX4's like this.
I have a non-traditional geometry, and allow you to do this type of movement.
This type of autonomy.
So as you can imagine, this is going to bring a lot in a lot of complications, especially
when you find the test things.
These are some of the other applications.
Walmart was using or firmware, and or hardware to do the delivery in the US.
There's companies that do mission inspection like Wintra.
This is a vertical and takeoff.
It takes off with the propellers, and then it transitions as a fixed wing mid-air.
This is a vehicle that has a very funny characteristic.
The payload, the camera, it's like a $100,000 camera.
It's more expensive than the drone.
So you've got to make sure that that camera is going to be safe at the end of the day, right?
So how do we actually ensure that?
It's hard.
It's actually hard.
We have another vertical takeoff and landing.
This is actually a very cool vehicle.
They can configure it differently.
This company is called quantum systems out of Germany, and we also support other water vehicles.
So how do we make it so that all of these freaking applications fit in the same firmware?
How do we make it so that we can test those things?
And how do we prove that we're safe?
So the honest truth here is that we can not guarantee that.
We are only a humble open source project that when we depend on their contributions.
I'm not a new authority to be able to test this.
Like the FAA in the US actually runs tests for aviation companies to be able to guarantee the safety.
You cannot expect an foundation to be able to do this.
But what you can expect is us doing our best to the best of our ability to actually guarantee the safety to the best of
our possibilities within our resources because it's expensive.
So the answer to how do we do this is the PX4 out of the pile of the firmware.
And this is going to be a little bit deeper into the technical side, but I'm going to
try to explain the functionality of the firmware so that I can finally go into the CI and testing
and how do we do it.
So we're running on top of the narthos, everyone knows what the narthos is, real-time operating
system.
Basically it's like an operating system, the guarantees the execution of the commands whenever
you say you wanted to do it and it's a gross characterization, but it's like a think of fancy
scheduler that guarantees that whatever you said is going to happen happens.
We're a C++ based, we're modular, meaning that within the operating system, different components
of the firmware actually run out of different modules, like think of different apps.
Like the app that runs logging is different from the app that runs the estimation or the control.
And that's important because you don't want the logger to actually fail and then the drone
is going to crash because the log is not working.
That's not something that we want so we want to make sure that it's separate.
But how do we actually connect all everything like all the modules through a middleware, middleware
is basically a published and subscribe component of our system that actually just basically says.
Okay, this is the driver for the GPS and this is the location that we're publishing.
And then there's a different modules that want to maybe subscribe to the location so they do that.
And we do that through a middleware we call Europe, which is the DS compatible for those of you
that know Ross, it's actually compatible with Ross and I'm going to talk a little bit about that.
Everything is better realized in Trebsafe. We have great hardware support, I'm going to talk a little
more in the next slide. There's support for custom builds, which is one of the biggest things you're
in something I wanted to talk to you about because we have hardware support, like I don't know,
think of this as one of the hardware. Okay, we support this hardware, but we have
variants of that hardware too, maybe you want this version of the firmware that doesn't have
this feature, we support that. And sometimes we even support upstream, which means we have to test for this.
We have more than a million of vehicles using PX4 right now. This is data that we surveyed from
the companies that are within our memberships, but actually we don't really know. It's impossible
to know for the open source project because we are not charging for our license, we're not actually
demanding from the companies that are implemented, that they share this info, but the members
that are the companies in Green Drone Code, some of them are manufacturers of the components.
I'm 10 minutes over. Okay, good. So I'm going to go quickly.
And we have 30,000 developers like SLB4. We got flight modes, which are provided instead of
helpers, the developers to actually be able to sort of autonomy. We have flight tests that help the
flight modes to actually be able to control the behaviors. We have throners where you can say,
if you're going to take off, take off a certain altitude always. Maybe you don't want to
every time you take off, set that. We have a device interface. If you're a developer, you want to
make sure you can subscribe to things. And then we have control allocation. We infranslate the trust
and the pork. Like I was showing you, we support different geometries. So it will be hard for
the developers to do the controllers to actually define and hard constrain the controllers
to the geometry because we want to support multiple geometries. So we have a translation layer that
helps you write the controllers so you don't have to think of the geometry. You think about it later.
And we also support cross 2. This is an example of a hardware. This is a pixel. This is the
one of the projects that I was telling you about. We support more than 80 boards. We have more
than 100 sensors that we support within all of the hardware collections. Mostly, the architecture
has the M32 based, IMAX and RIS5. But we also support payloads, gimbals, batteries, the bloggers.
There's a whole ecosystem that is just doing this. And like I said, the mission planning software,
it's also part of the ecosystem, something that we also have to test. We have the rawst to support
with things to the middleware. I'm going to gloss through this because we're running out of time.
We also have a simulation engine based on gazebo, also based on open robotics projects,
which allows the developers to actually do very advanced things and test them.
But how do we actually test it? And sorry that I took half of the thought for this. In my mind,
there's going to be five minutes. So the developer testing, I separated this two things.
Unitests. Integration tests with simulation tests. Hardware in the loop tests. Okay, that's what
we can do in the cloud. We run in CI. But what about this? Flashing and installing on the hardware.
Setting up this hardware. Setting up the parameters. How do you want to tune this?
Actually, flying with an RC, or maybe flying an admission, autonomous mission. And how do you
validate those missions and those controlling those flights actually work? So we do two different things here.
Testing the cloud or mostly in the cloud because we also do hardware in the loop.
Let's run with the unitesting. So, or fortunately, our unitests coverage is less than 60%.
Less than 60%. Why? Because it's been really hard for us to enforce this.
We've been slowly building and growing this number, but it's really, really, really slow.
It's very hard to incentivize the contributors right now to get this. But we do have them,
and we do run them. The test run in about seven minutes per pull request. And we have about,
I don't know, like 10 to 20 pull requests per day. So sometimes, in a given negative in Monday,
I would wake up, go to my computer, and we would have like 50 pull requests that came in on Monday.
Because I don't know, developers worked over the weekend and just took me to the pull request on Monday.
And I'll surprise. Everything in there was running on greekobactions and the runners from greekob,
which meant that a pull request was sitting in queue for hours, sometimes days.
Because the collection of every test we did just took hours. So, this is an example right now.
This is the current set up and this is just only taking seven minutes. We do integration tests.
This is a bit more complex. This is actually three step. We have a test runner,
which is a Python script, and we have different tests. This are just an examples, but we have way more.
And each one of those actually launches px4 from 0, from scratch. And then px4 launches gasivo,
from scratch for each one of those, which meant that at least you have five minutes wasted in there,
in setting up the environment. And then you run the tests and it took like 20 minutes each.
We fixed those. That set up. I actually fixed that last October.
And then we realized that when we fixed the execution time, we found out that there was actually tests
in there that were failing because there were real errors. But we couldn't see them because there were
timeouts in the execution because sometimes when the test runner launched, it tried to launch px4.
And for whatever reason, the resources in the system were lost or maybe it didn't launch
actually launch px4. So there was a timeout, but you didn't know about the timeout until like it was too late,
like 60 minutes late. So we had so many failures and we didn't actually know because we couldn't
have, we didn't have the observability into those issues. So I took it to the task. I tried to fix
those issues and then I found way more. These are just the only tests that here that sometimes fail
in terms intermittently. And we haven't figured out why yet. Thank you. I'm almost done.
This is just an example of what's happening. Actually, this is from the screenshot from the
last 9-15 minutes execution time overall for all of the tests. It was a huge effort.
But mostly it was me scratching my head against Gid Cup runners because I actually didn't know how
they worked. And the whole setup was to me honestly something that I never dealt with.
And then finally we do hardware in the loop and we do this through Jenkins because one of the
maintainers in his house has a test rack with the USB hub with a bunch of FTDI to USB cables connected
to all of the hardware. And every pull request guess what? All the hardware got flushed.
All the firmware got flushed and then you configure everything. And then you check the status
and then you reset devices and then you run tests for everything. This used to take hours to
per pull request with crazy. It's crazy. Like why did we think this was a good solution?
Because we didn't have anything. We wanted to have something and we had to start somewhere.
The only reality is that we shot this down last month because again,
the hardware, first of all, we didn't have every hardware that we support. We only had the ones
that the manufacturers shipped us or the ones that we could afford. And second, we had a lot of
time loss too when we tried to flash. Sometimes the FTDI drivers didn't work. I don't know why.
It was a Windows machine and then we switched to Ubuntu machine. We had driver issues. We had to
reconfigure Jenkins. It just wasn't nightmare. We couldn't support it as a maintainer.
We had to spend more time reviewing the pull request and actually fixing this infrastructure. So we decided
to turn it off. So if anyone here wants to pitch in, thank you. I would please talk to me.
All right. So lastly, the most complex thing of all, how do we actually do build,
how do we publish artifacts? So our primary development environment is Ubuntu.
Whatever the LTS is, right now it's 24 or four. We build using GCC. But since we do
embed it, we have to use the ARM cross build GCC version and we're locked down to version nine right
now because of historical reasons. I'm actually working in trying to put that to I think 13.
I don't remember which one this. But we have a script that installs all the dependencies.
Actually, we have that running in CI and we know that if you want to run the PX4,
we guarantee that if you run Ubuntu.ca station, the fresh machine is going to work because we
have it in CI. Finally, we didn't have that last the beginning of last year.
So this is how you run it. 9 PX4 and then the variant of the hardware.
So you always need to specify which hardware. So how do we make it so that we actually
burnt all of the 80 hardware boards that we support? Well, we used to have a hard-coated file
in which if you add a new support, you also have to make sure that you add that line
then make sure that the string name is right. What happened was that sometimes the manufacturers
forgot to add it there and then someone reported an issue and guess what, it wasn't in CI, so we
never catch that. All right, that's an issue. Second issue, it took so much time to run everything
because we were running a matrix and github actions which meant that it was every put request
was launching 80 runners. So if you have 10 pull requesting queue, that means 10 plus 80,
how much is it? 800. And each one of those took 15 to 20 minutes. That was impossible for us. So
we created the Python script and we actually grouped together by manufacturer and we started
caching those builds and that made it so that if you're using the same manufacturers, sometimes
the dependencies when you're making the build are the same. So you could save like up to 80
percent of your time on builds. So we wrote this script and we put it on github and we
trim it down from 80 to 23 jobs. So we only now have 23 runners. We also did one something else
which is we moved into self-coasted runners in the AWS using a platform called runs on.
You have never heard of runs on. They make it so easy for you to actually run your own infrastructure
and really happy any three-for-open source. So if you really want to try it out, it's really easy.
So this is one group, not xpx4, and it's like all of the different versions. This is v6x,
all of these are the same hardware, but these different variants. This is the part that was actually
killing us. But thanks to the caching, thanks to the new runners, like look,
58 seconds in running one build. It took 10 minutes, at least before. Now it's taking a minute.
So it was a result. What took hours now takes 16 minutes. But still, it's a lot.
Here's an example from pull request from last night. Lastly, 57 checks. I didn't talk about
all of the things we do in CI. We have about 57 different checks. And an example, the container
build failed, because I wanted to be smart, and I wanted to push every container building to the
Geek of Registry. So I was doing a deal. I talked to my API token, and I was basically running
through the threshold and like the first hours of the day, which meant that from there on,
throughout the day, every PR is going to fail. So this is still out there. So if you want to
contribute something, does it really easy fix? And I'm not going to suddenly not going to be able
to do the human in the loop test, but we actually have a flight test team, and they do actually
do the testing. And they submit logs through the website that we have, which is called PX4LOG
review. You do upload the logs. They give you nice charts. You know what happens. And then the
developers actually tell you, oh, your motor is not aligned. This is why you have so much of
a ratio. Maybe this is why you're not able to move the drone direction you want. As I
conclusion, my friends, testing in robotics is tremendously hard. My robotics colleagues
here is not going to let me lie. We've been suffering, and we need your help. So if you're ever
thinking of collaborating without robotics project, I know the few. Tomorrow, we have a
debt room, and which building team is at UV? UV21 something. It starts to meet there, and there's
really cool content. So if you have time, please go there. And thank you so much.