Hello, and please welcome Roland and some that apparently have fun with calibration, never heard of that before. So, very happy to hear about it.
I need a microphone to actually say something.
Okay, welcome everybody to our presentation about the tools and methods to get top quality robot data.
So, why would you monitor data quality? Let's start with that question.
What kind of talk can we give and seminar have a combined like amount of more than 10 companies where we were annoyed at data quality.
So, we figured that this would be an important topic to present.
Whenever you join a robotics company, robotics companies are always very proud of collecting a lot of data.
But your data should also be good from start.
However, do you want to give data to your users as soon as possible? So, they can find the mistakes that you're not seeing.
Yeah, for machine learning purposes, everybody wants to train machine learning algorithms nowadays.
And for that your data should be good and consistent.
So, if you have collected data for five years and then you discover that your camera was terrible, then you've only collected data for one week.
So, try to get with the people quickly so they can find the issues again.
Yeah, nowadays I'm spending most of my time just begging the computer to write better code.
And that's not going to solve your problems. I noticed that a lot of people are not looking at their data anymore.
So, we are trying to convince you that you should take a look at your data.
Yeah, so we could only put a subset of common issues in this presentation.
Who are we? Sam, who are you?
I'm Sam. I've been doing robotics for over 15 years. I love robots of all kinds.
Even though my favorite, I humanize because they're complex and they're fun. They can do things.
And yeah, I follow my GitHub and ask me any questions afterwards if you want.
My favorite kind of robots are self-driving cars. I think they're pretty cool.
So, I'm working on end learning for self-driving cars at wave.
Yeah, you can follow me on LinkedIn. You can follow Sam on GitHub and you can follow the focuses in my backyard on Instagram.
Yeah, let's get started. So, we figured let's start about researching and planning your sensors.
One thing I at least always notice is that people have not the right expectations for sensors.
So, one of the things I often face is that people say, hey, Roland, can you make a machine learning algorithm to detect this pedestrian?
Which is at 80 or 100 meters away.
And then it's kind of my task to explain people that this pedestrian is only four pixels in the camera space.
Or only has like two lighter points hitting the pedestrian and that no amount of machine learning is going to fix that.
One two, which I found for this, which I really like, is that 10 confusion made a visualizer for lighter,
where you can compare different types of lighters. So, you can see what their range is.
You can see at what distance how many points roughly would be there.
They also have one for depth visualization. So, yeah, start building your robots by thinking about what accuracy do you need at which distance.
And yeah, what is the coverage around you and your robots?
And it's not only about your sensor itself, like how big is the image of getting how many pixels and so on.
You also need to think about the details on it because they may affect you more than you think.
So, for example, the interface of whatever sensor you're using, is it used be?
Well, you may have issues with the, you know, flakiness of the connection of it, which is going to affect you the data quality.
Or you had ethernet, like a lighter, cool, it's awesome, it's really easy to use anything else on it.
You can buy switches, everything is awesome, yeah.
But then you're paying a lot of CPU costs on the network stack, which may also delay your data, which began with affect your data quality.
Yeah. So, and, you know, every kind of technology has a trade-off, but you should probably think about it a little bit, or if you are not sure try it first.
And then whatever sensor you choose, the driver quality is quite important.
We all like open source here, so if your driver is actually open source, so you can look at how things are done and improve them if you need to.
That's great. If you can't, but luck, hopefully your vendor is friendly and helpful.
Also, is your driver efficient, because we all have seen, like, oh, yeah, this driver in particular uses a lot of CPU, well, maybe it's just doing something silly.
And the data format is the sensor giving you the data. In some format you can just already, you need to transform it.
When you transform it, are you losing anything that may affect the data quality also? Check it out.
I know, if you click, I really like that diagram up, so I've quoted another one that does the same thing, but also cameras, and you can also see the person in the camera, so you can place your little sensors and get a bit of an estimation of, oh, how many cameras do you need with what, yeah, the solution for the view, but lighters and so on.
And I made a preset of list of, you know, sensors you can buy nowadays too, so you can pay around.
I accept poor question.
Yeah, on my side, one thing which I often see going wrong is the coordinate systems.
It happens very often that people just think, oh, surely the coordinate system, surely the x-axis to left or up or whatever.
This goes wrong all the time. You really have to understand which coordinate system you're using, and the worst part of coordinate system errors is that you can actually
if you make a mistake upon a mistake, it's good again.
So I've often seen that people, for example, in the perception, like if you have a perception, prediction,
or like perception, fusion, prediction stack, people on the perception side may be do something right.
They use a correct coordinate system, then the fusion people use the wrong coordinate system, then the prediction people use the wrong coordinate system again, and suddenly it's perfect again.
But as soon as you start programming something on top of the fusion system and you actually use a correct coordinate system, it's bad.
If you fix it, you break your other parts of your robot.
So yeah, please, please make sure to check every step.
The only solution I have here is to document, to educate people, to visualize things.
My tip here, and people who work with me can confirm this, is that I make a 3D printable coordinate system for the ISO 885 coordinate system. I leave it all over the office.
So I brought a few, but I forgot to get them out of my bag, but I put them on the table after Sam says something.
You can download this on thingy first, but other thing to notice here is that different types of robots have different coordinate systems.
So if you print these things and put them over your drone company, then your drone company is not going to be happy.
So yeah, keep this in mind, and you can download this.
This is by the way, from this is the exact open data sets, they somehow used three different coordinate systems in the same data sets.
Okay, now time stamps, every company we're going to work with had issues with them, why?
Well, I mean, it's not that easy, really.
But so first thing you need to think about is the reference time of all your sensors the same.
It may be different computers, different sensors the glowing one not, it may have a clock with a battery, it may not.
So yeah, first check that, because did you know the world started to not Thursday?
You know, during the next time starting first of January of the 70.
So this happens a lot, just any kind of time sync where you can, but make sure you have a common time reference in some way.
Or you can't figure it out, even if it's offline, just figure it out in a way that you can ensure you have one.
Another thing is synchronizing the data of your sensors.
Well, do you call the hardware triggering, which is great, if you can do it, it can be complicated to implement.
You may have sensors that just don't support it at all.
You may need to have a mix of hardware triggering with software triggering, which is hell, so think twice if you really need it.
Because it's a lot of overhead, maybe the way you use your data doesn't need this.
I mean, it's great to have if you can, but think about it.
And then yeah, we have timestamps, but what do they represent?
Depending on the technology of sensor or what you're doing or what is running.
It's a times time, you're trigger time, but you told the sensor, now I want an image, light at point or whatever.
Or is it when you resist it from that sensor or from that machine or whatever?
Or when you pull this in your system, like pros.
Or when you record it, it didn't do a bug, for example.
Well, it doesn't really matter which one it is, but just know which one it is.
And if you actually care about multiple of them, well, represent that information.
And again, check your timestamps in your code. If your board just go around, look at your timestamps.
You will find more issues than you think.
It is very easy to miss you issues, like you may put a little bit of code in between getting the time and actually getting the data and you just messed it up.
It's easy to make mistakes.
I see this mistake so many times.
Yeah, so in terms of getting to know your sensors, Sam?
So, calibration, right? That's one of the topics you wanted to talk.
I'm actually not going to talk much about it, other than what do you calibrate?
Well, usually then twin six and the extrinsic of the sensor.
In twin six, for cameras, for example, it's like a distortion of your lens.
So, you can eventually undistorted and make straight lines straight and things like that.
And the extrinsic is nothing else that varies my sensor in relationship to another sensor, which is very useful for, well, projecting things in the position of another thing.
Right? Well, this is great if you calibrate it and they're too short of it.
And it's a pain and calculations go all for the time and the sensor changes during the lifetime, the temperature changes and whatnot.
Lock the data raw.
It's cool that you can change the data how it looks, but log it raw.
As far as you can, obviously within reason, I mean, we don't want like raw from a DSLR camera images, probably for robotics.
I think I may be wrong depending on your application.
And a story calibration that goes with that data at the time and make it so you can change it later on, in case you discovered that you calibration worked wrong, right?
And when you undistorted images, for example, or use that calibration on them, or Canon on Canonalize,
you information to change it from frame of reference, make sure you do it on purpose, don't just do it by default.
The more raw you work, usually close it to the sensor information, the original information is the best.
So just take a second to think if this is better just from or not.
And a couple of cool tools.
If you want to calculate cameras, there's a website, you just go and it opens the camera, if you have a connected computer.
And you can't really read it by showing your phone with a marker, it's probably not so.
Cool.
Yeah, in terms of lighter point clouds, contrary to the previous talk, I'm generally quite happy with lighter point clouds.
They can be very accurate, they can only have like a two millimeter error, but there are scenarios where they are not super accurate.
And my biggest problem whenever I'm working with lighter is that it's very difficult to explain to other people in the company what the errors scenarios are and what mistakes people made.
And it's also hard to predict errors if you are not familiar with lighter.
With cameras, easy everybody understands what a camera is, how it works, what you can expect from it.
And common issues I've seen is just that lighter to vehicle calibration or bad lighter to robot calibration, but lighter to lighter calibration, if you have multiple.
Again, not understanding time steps.
Dirty, if your lighter gets dirty, if you forget to clean it, you get way less range.
You're probably not going to notice if you're not frequently looking at your lighter point clouds.
And yeah, already set unexpected expectations of what a lighter can do trying to detect things at a crazy far distance if you don't get lighter points there.
So one thing which I think is funny with lighter is that everybody seems to build their own visualizer.
And it's very difficult to convey details with a plot because you're trying to put 3D information onto a 2D screen.
And then I always notice that I'm awkwardly trying to zoom in and zoom around to show people what the problem is with their lighters.
So because everyone is rolling their own visualizer, I also made my own, scored immersive points.
The features I have is that I always use it to embed my lighters and my time stamps into a Jupiter notebook.
So if you're running a server with this, you can do that.
And the best part is that it supports virtual reality like natively artificial books.
And it actually helps a lot that if you have lighters which are not calibrated with respect to each other,
show your product manager or CEO that this one corner shows up mode with times in same point clouds.
It's easy if they walk around it. It's harder to show on a 2D plot.
Yeah, you can help me extend this to. I brought a VR headset, but I don't think we have any time to show people, but if we have a break, we can we can.
Yeah, calibrating your senses.
So in terms of one tip here is to always try to visualize sensors with respect to each other.
So you can quite quickly see that things, or it's difficult to see that something is wrong if you only show one sensor.
But if you try to, for example, plot the lighter points on the camera.
And you see that things don't align. You notice something wrong.
Show bounding books on both or try to label bounding boxes in both spaces and see if that's lines.
One easy to break is project camera down on the ground.
And if your camera projections nicely flow into each other, like the features on the roads are matching up in the different cameras.
Then you, these are confident that it's probably well calibrated.
If you have an HDMI and you have good localization, that show map on your robot align with actual world.
And having good raw transforms makes this trivial.
But it's also a good challenge to make sure you have the correct raw transforms.
In terms of visualization tools, I really like the folks love Samloff's RVS.
As it's still an open source conference, folks love it's not open source anymore nowadays.
There are some branches, like folks love or leak leak.
It's a clone of folks love before they like close their repository.
This is also rerun, which I haven't tried myself, but they are committed to stay open source, so that might be something to take a look at.
This is what let me see.
I tried this multiple times and it always works, but of course not when you're trying to actually get for presentation.
We had a really cool, you really have to imagine that there's a cool video here of our robots.
I don't think I'll try at the end if I have some time left to rescue the video.
Oh.
Yeah, you can see it on my LinkedIn, I guess.
Good luck.
Let's see. What did you want to say, Sam?
Okay, so we're talking about calibration and data quiet and so on, but calibrate yourself too.
You know, when you every company has something that they call golden data, like this is their golden tables.
This is all the good data you can use for training, for least on that, you can double check things with it.
Well, why do you consider it's golden?
It's this in opinion of one person, many people.
Is it because it's better in one aspect than the data you had before?
Is it more than one aspect?
Is it avoiding something that you had an issue before?
Maybe you know, you were recording wrong the timestamps.
Maybe you were doing, you know, some other thing the sensor was badly mounted.
Something was referred wrong.
Maybe it was the wrong sensor.
Or does your golden data incorporate something that you didn't have before?
And are you sure that fixes what you were doing?
And doesn't break anything else?
Just be careful and check.
Maybe as to yourself, if I had perfect data, how will it look like?
Like, and from that question you may see what you actually need to change in your system to make it look like that.
Better, ungolden.
Cool. I wanted to end with a couple of easy tricks.
So if you do a couple of simple checks, you can save yourself a lot of hassle later.
One easy trick which always helps me is monitor the time between samples.
So if you see that as a large gap between different images, maybe you're probably dropping them.
Also note here that often you start off with a robot with bomb camera.
And then this check will show that everything is correct.
And then you start adding sensors and you start adding computing things on your robot.
Suddenly your buffers are falling overflowing.
If you have this check in place already, you're going to immediately notice it.
And otherwise it will take you months before you figure this out.
Yeah, monitor where you start and enter runs.
So if you have a good localization system for your robots, check if it's already getting a plot from one person.
If you have a localization system for robots, that your robots start the next run at a place at end of the previous one,
or they're magically transport.
If you expected it to transport as fine, but if you didn't expect it to probably drop in a run.
Visualize what data you record.
So keep checking your assumptions.
Are objects visible when you expect them to be.
Are your sensors clean working right side up?
Yeah.
Did you put your left camera on the left part of your robot?
Classic.
Okay.
I think or all these talk, we're trying to say visualize your data.
Whatever you can and make any failure as objects as you can.
So plot the things that will look wrong because you've had it wrong before, probably.
So do mistakes, mistakes are good.
Look for the detail.
After you do that, well, automate all these checks as much as you can.
You know, if you cannot automate, if the image is blurry, if you're dropping frames,
if there's occlusions on your sensor, automated as much as you can.
Every time you find a new issue, if you find a way to automate that check, automate it.
And then take a look again.
Visualize, but take a bird's eye view of it.
Make a dashboard.
Take the trends of your data.
Take for the outliers on your data and just keep an eye.
And every time you see a trend going back down, you go down to the detail, automate, and so on.
By doing that, you know, in a very little time, you will have really good quality data.
Okay.
Last but not least, this was the tools which we mentioned in presentation, which you want to check out.
There's sensor preview app, immersive points for your 3D points in virtual reality.
The ISO 8855 coordinates system, which I now put on the table here.
Foxbox, click click.
So it's a clone of Foxcloth.
If you don't want to use Foxcloth, rerun, which still commits to be open source.
There's a lighter planner by tank renovation.
Yeah, give it up for these tools.
That's it.
Thank you.
I'll try to rescue the cool video while semistairing questions.
Yeah.
Hi.
Is that okay?
Not really just to follow up on rerun.
It's the default visualizer for the lay robot ecosystem.
For like, you know, the fucking face, also arms.
They use rerun.
And it's kind of like, yeah, it's a Foxcloth alike.
It has like profiles and opens channels.
You know, those squiggly graphs and stuff like that.
So definitely worth checking out.
Don't particularly, you know, have anything other than that.
I will say if anyone has a good comparison by experience from both of them, I'm interested.
So yeah.
Yeah.
Let's talk about it afterwards.
Hi.
Do you have any recommendations about the timestamps?
Because when you're dealing with multiple time systems like GPS, UTC, and the local
time and the yard of the insulation, frostbarks get complicated.
Because every message has two times sent and received.
So do you have any recommendations?
How to document that?
So it is somewhat easier to manage.
Honestly, you know, it's hard.
It is a hard problem.
So that's why you need to really put some time on Ethernet.
And if someone really dedicates the time on your system,
just write down good docs, write down your expectations,
and the detailed view of how you work with it.
So everyone understands what is your perspective when you are thinking about the timestamps.
It's not easy.
Otherwise, we will not be talking about it.
But yeah.
Cool.
By now, by the way, we've got the video working.
Let's see if it keeps working.
No.
The answer is no.
We'll keep trying, after the next question.
Yep.
I would like to ask if you have some recommendation to calibrate for example your sensor to your vehicle.
So for example, if you have a very good localization sensor and you know your lighter,
how you connect them like, because often the frame of the robot,
sometimes it's like the center of mass that it's floating in space,
how do you approach this calibration from wider to robot, for example?
In every company I've worked, there was a different system to do so.
From the most just literally measuring real life where your sensor is and do your rest with that,
which honestly gives pretty good results because if you do the exercise of comparing with sensor to each other
and on some ground tools, which within the speak about ground tools, but it's great if you manage to have ground tools somehow,
like a big on system, if you just a GPS, RTK, for example, or anything you can come up with,
like markers around in a well known space, like, let's say if you work with cars and you have a place where you sell the cars
to shop every world with a bunch of markers and, you know, at least know that the walls are annoying.
So you have a reference system to check.
Again, I don't think there's a very easy way to do this.
It's smaller robots having it easier, bigger robots like cars.
When you reference points, it's like the bugwheel, sorry, kind of thick of the word, but you know,
the bugwheel line where you don't really see it, you kind of really measure the accuracy, you need to trust the data sheet.
You usually need to do some optimization method that based on actual real data will give you the best estimation,
which it's hard to get right, but that's the best we can do.
I think nowadays, otherwise maybe some ML network will come soon and we'll help this.
Cool.
I'll try to show the visualization again.
So what you see here is the foxcloth view from our car.
So you can see the cameras, you can see, but if you use the pink points, are the radar points,
and they nicely align with the features of the map.
And if we, and also the green blue overlay you saw, that's the map we feed to the robot.
You can see that we've got three cameras on the front projected, and then nicely flow over into each other.
You can see the predictions of where the mother wants to go, where our trajectory planner wants to go.
The best thing which you can't see because equality is associated is that I even got the wheels to spin in the right way.
He's the most proud of that.
I am Luke Man, I spent an afternoon doing this, and also they are tuning to the correct direction.
Again, this was the most proud of, but it helps a lot.
It helps a lot to see, to save your robots and validate that your radar points are actually at objects.
Your cameras are well calibrated with respect to each other.
Your robot is doing what you think it does.
Yeah.
This kind of visualization should be easy for everyone to access, because people think about the data at different levels,
but they should always be able to go back and make sense of it.
Cool.
Questions?
We had one question from the chat, which is any tips that apply specific to several step stereo cameras.
My experience has been pretty bad with them, so I think they are cool when they work very well.
And the vision is awesome, but when it's not, it's hell.
So I've tried to avoid them.
I don't know what the current best way of evaluating stereo cameras.
But I once made it to which try to overlay the two cameras into one.