Hello, and please welcome Sid, that is going to talk to us about Lighthouse Pottinging
System, something we have been using a lot, a bit crazy, and that has been doing awesome
job.
About no pressure again, so that is going to be great.
Welcome, Sid.
Thank you very much.
You know, I have to say there is way more people than I expected, so thank you for your
interest.
I hope the presentation is really good.
So, as alongside my name is Alvarado, I am a PhD student at the EREA Paris on Swarm Robotics
Facialization, and I spent the last two years of my life trying to coax a VR gaming headset
to do things it was never meant to do, and I am here to tell you all about it.
We have a question on my laboratory, you know, VR gaming headsets are really good
at the localization.
They are very fast, very responsive, really high precision, so can we do that, but for robots?
It is also the answer is yes, and not only yes, yes, it works surprisingly well, like
really high precision, really high accuracy, very low cost.
So, I hope that in this talk, I can give you a general overview and introduction to this
technology, how it works, what are the core concepts that you need to know to both understand
it and you see it, show you some of the open source tools that exist, and hopefully give
you a general idea of whether or not it is a good fit for what are the projects you want
to embark.
So, what is a lighthouse?
So, a lighthouse is a little box over there, it is used for localization on the valve index
and the HTC Vive VR gaming headset, those are a little boxes that you put in the corners
of rooms, and they are sort of emitting laser pulses.
Now, the helmets and the handhelds have little sensors that they receive this light, and
that they can compute their own position.
Now, why will we want to do this?
Are there alternatives?
Yes, yes, there are many alternatives.
In fact, if you want to use the best of the best, you will use a camera-based motion capture
system like a icon or a quality system.
They are absolutely amazing, insanely high refresh rate, super limited precision, but they
have one caveat, they are a centralized system, you have a bunch of cameras, they are
recording your robots, they send out the information to your computer, your computer,
crunches numbers, and then your computers know where the robots are.
The robots don't know where they are until you tell them, which is fine if you have like
a dozen robots, you can just send it to WIFE or something, but the moment you start dealing
with swarms of a hundred or up to a thousand robots, this becomes a very serious problem
on the bandwidth of your system.
If you are into any conference, do you know how the Wi-Fi works when you have more than
fifty percent in a room, try a thousand robots, just does not work.
Also, they are a great use of expensive, we have one in my lab, I believe it costs like
40k euros, which is fine, it's a very good technology, but it's not appropriate for every
single scenario.
On the other hand, the light-up-use-in-system has a fairly decent update rate, five millimeters
is good enough for a robot like this size, or that's enough.
But more importantly, the moment the laser hits the sensor, the sensor has all the
information it needs to compute its own position, thus the position is no by the robots.
Then if the robot wants to tell you the sensor controller, it can do so, but you don't need
to have like a hundred update Hertz rate on the transmitter just to get like a close-loop
control working.
It is also fairly expensive because it's based on very available hardware.
So let's go into the details.
This system is based on two different parts, we have the base stations which are little boxes.
They have a bunch of little motors, lasers and mirrors that spin around and project a spinning
pattern of lasers around it.
It costs about 160 dollars, if you buy it on a steam, they sell it there, and you have the
sensors.
The sensors are these little things that actually receive the light, they are based on a custom
made, see those call it TS for it to 3-1, which has a really fun story, they were
custom made for this and this application only by a US company called Triacemic
Conductors, and for some reason, which I don't know, they just decided to sell them.
So you can just buy them a diggy key, they are there with addition and everything.
Sadly, the light house does not have a data sheet, it's not open source, so thus
why the technology is not super wide spread, even though the parts are actually available.
Now, there is actually two types of light house version, one on version, two, this whole
presentation is on version two.
The difference is on the beam pattern, the light house one has two motors and those
a sweeping horizontal and vertical laser pattern, and the light house B2, they actually
manage to fit the exact same horizontal vertical information on a single motor, so they use
just like a single spinning motor and two V-shaped planes.
From this moment onwards, if I'm talking about light house, I'm referring to the V2,
that's why we are going to be using.
So, you have a sensor, you have the base station set up, the laser sort of flying around,
your laser hits the sensor, what happens then?
So, the chip has two the lines, one's called envelope, which just tells you, hey, there
is data here, it's just a single in-trop line, and then you have the data line, which
receives a bit stream.
Now, to be specific, this is at 12 megahertz Manchesterian coated differential, this
trip of a pseudo random number of sequence, but for all practical effects and purposes,
it's just a really huge 96 kilobit bit stream that you know, like this is an at table,
you can just look it off exactly which bits go where.
Now, a small parenthesis, because this is important for anyone who wants to implement this,
in my laboratory, we found out that you need to sample this line at least 32 megahertz
otherwise the signal is just not recoverable, 12 megahertz is aggressively fast for
a signal, you have to like sample on GPIO, you have about 80 nanoseconds between pulses,
and this turn out to be the mainly mutation of implementing this technology on a particular
low power micro processor, if your MCU can do 32 mega samples on a GPIO, they're good
to go, if it can't, you are going to have to find another one.
Pardon, this is close. So, when the base station starts to sweep, it starts at the right,
and it starts sweeping, and it starts sending the bit stream, it starts bit 0, 1, 2, 3, 4, 5,
blah, blah, blah. The base station rotates at constant speed, and the bit stream is transmitted
also at constant speed. So, by the time you hit about 90 degrees, you're already at
bit 48,000. It keeps going, you keep receiving a part of the bit stream, and by the time
you finish the health rotation, you are at 90 size K, roughly bits. So, you have a sensor,
it gets hits by the laser, and you receive that bunch of data. You grab that data, and you go to
the gigantic look-up table and check, what part of the bit stream did I just receive?
For example, if you notice that you start counting and you say, okay, I got bit number 24,000,
you can get a very direct correspondence, that that means that you just receive
a laser was transmitted at 45 degrees from the base station, and that's the whole point of the
bit stream. This is just a very complicated way of telling the sensor, what was the rotational
position of the lighthouse when the laser hits the sensor? Why? Because, if you have, for example,
two base stations, position at no locations, and one of them is reporting 30 degrees, and the
other one is reporting 110 degrees, then the only possible position where the sensor could be
is this. Then, since we also get not only horizontal, but also vertical, and got information,
we can also extend these two three-depositioning, and thus you can do a full triangulation,
and you get the position of your sensor wherever it is. Okay, couple of text backs, we need to
talk about a little caveat. This is very nice, but we just talked that the lighthouse
actually projects are v-shaped, this doesn't do like a little cross, how do we correlate those two things?
So, if you put yourself on the position of the lighthouse looking forward,
what you see is a little v pattern going from right to left, and what you get on the sensor is the
rotational position when those planes hits the sensor, that are called alpha one, and alpha two,
and to some confusing looking, but not actually a difficult trigonometry,
you can actually translate that to both affimus, horizontal angle, and elevation, vertical angle,
and we are back to doing triangulation. Now, small summary of the whole thing,
you have a lighthouse, it reacts a v. You have a sensor, it receives it and it gets a bit stream,
from the bit stream, you calculate the affimut and elevation, and from then you're trying to
regulate the position of the sensor. That's pretty much it. Except, you know, it's kind of a
hugely mutation that you have to know the position of the base stations to be able to compute the
location of the sensor. You're not going to go on to a certain time measuring where you put the
lighthouse with like millimeter precision just to get the thing working, that makes no sense.
So, as fixed up, thankfully, this problem was solved for us about 30 years ago,
by computer vision researchers, because this is a problem, as a stereo vision calibration,
and in the next slide, we're going to see that through some little mathematical intuition,
we can actually equate a lighthouse-based station to a tingle camera, and since we can
treat us, the camera that means we can take advantage of the case upon the case of
research and computer vision to solve all of these for us.
A little possible water.
Wonderful.
Charm, imagine this, you have a station, you have a sensor, you have the line that connects the
both of them. Do you know the angle of the slide, because that's what we've been getting from the
bit stream? Imagine you have a camera, like an image plane, like you put in front of the lighthouse,
and then you intersect the laser going from the base station to the sensor with that plane.
Looking at it on 3D, it looks something like this, you have a base station,
overlooking some robots moving around, and we want to transform as smooth and elevation angles,
which are kind of unwieldy into pixels. How do we have this information as pixel,
because pixels are nice to work with? Well, you intersect them with that imaginary plane,
and whatever they hit, they are your pixels, and you have a lighthouse image.
The position of the pixels is related to the
afimoting elevation of the angles you were doing, and the K is the
intrinsic camera matrix, the only thing it means is that we're working with an idealized
ping-ho camera model. It's a really simple model. Once you have this done, this is the crux
of light-hospitalization. The triangulation is nice. If you consider the camera,
then you have access to way, way more techniques for it. Like for example, a star revision.
Now, the star revision problem can be follows. You have a real-world object in some location,
and you have a view of that object from two different cameras and two different locations.
Camera based stations, where we want to call it. And the question is, can we get from
both perspective? Can we calculate the rotation and translation from one camera to the other?
Because knowing this is the same as knowing where the based stations are located,
as long as you know that, you can go back to triangulation. The problem is solved. Once you know
that, also, you can do a lot of really cool stuff. Like for example, you can do photogrammetry,
really reconstructions, or doing like depth estimation based on your images. Really cool technology.
Let's explore two ways that we try and we know work for doing this with light-hospitalization.
Number one is 3D essential matrix reconstruction. It's not as bad as it sounds. There is
the delay rate that does it for you. You have two ways stations. We are looking at a robot that
has only a single sensor. This is for a single sensor reconstruction. If you have at least seven points
that you know correspond between both views, you can use go-to-open CV and calculate the
essential matrix. The essential matrix is the matrix that relates both views of the same object.
There is a function for that. You need to use yourself only to understand the math.
You need at least seven points and they must not be complainer. The function,
a speed sound of matrix, and then you can do a single about the composition, and it gives you
R&T. Where the stations are, you can do triangulation. Now, if you have more than one sensor,
for example you have four, you can do something that's what a prospective endpoint problem,
which is the following. You have one or more cameras looking at an object that you know,
you know the dimensions on shape of this object, or you have known markers on it.
If you know where you are, and what's supposed to look like, you can guess from the picture
where the picture was taken from. It's a very simple concept. If you have a car and you have a
camera and you take a picture of the car, if in the picture you can see the trunk,
you are not taking the picture from the front. You are clearly behind the car. If you know the
geometry of the object with high precision, then you can get really high precision on where
the camera was when it took the picture. Say, for example, you have this PCB that has four
light host sensors. In four locations, if you know where they are, you take a measurement of each,
based on how it looks, you know where the camera is, where the cameras are. This also has an
implementation in OpenCB. You need at least four points, doesn't matter if you are a computer or
not, you put that in there, and it will give you the position of the base stations, and again,
you are ready to go back to the triangulation. So, this is all cool enough, a lot of math is fine,
but how well does this actually work? Now, here are some numbers to back. Back now, when I said
works well, this is based on a paper, me and my lab polish, as well as a paper that the
bitkris people polish. Both single sensor and multi sensor tracking gives you about one centimeter
of precision, somethings left, somethings more. But there's more really cool results that
we like to point out that the system is really stable. Like if you put a sensor on a table and you
measure it, the variation and the measurement, like the data of the measurement, it's less than
a millimeter, which means that you can put like two sensor, right next to each other, two meters
away from the base station, and the base station will be able to tell them apart, which is
really cool for like micro-precision positioning. This climber, these numbers were taken on a two-by-two-by-two
meter cube space, so it's not like we are talking about, yeah, we are very precise in this time
by 10 centimeters, no, no, that can be done in like this area. Cool, that's cool and all.
I'm so, I won one. Where can I bite? So I have good news and bad news for you. Good news,
lot of open source work. For those who have electronic hardware production capabilities,
here is an implementation of a breakout board for the sensor chip. This was designed by myself.
Starting a special, it's just the, it's just a reference design on the data sheet, but if you
once a PCB that's been known and tested to work many, many, many times, here you have our reference.
Please take it. It's made in Kaikav and all the sources and the real material is all open source
and it's all there. I have made many of this, they all work. One little caveat, the TS431 chip,
it's a BGA with like 0.5 millimeter separations between the pads, not easy to solder in this
slide with, you might want to use PCBA manufacturing. Firm or wise. Memalap publish a reference
decoder for the light-hospulsive based on the Raspberry Pi picot. You can grab a Raspberry Pi picot
connect four of the sensors to it and put it in a front of a couple of stations and it will just
start a streaming through serial, all the angle information it gets. It's pretty stable. It works really
well and it's very fast. It takes like 30 microseconds per pulse for for a computation. So if you
want to have a reference decoder, you can go here. If you want a reference on the map behind it,
the link to the paper explaining all of this is in the page for this presentation. We also have an
implementation for the NRF Nordic semiconductors 52 and 53 family, but for hardware limitation,
those can only handle a single light host sensor. That's nice. I don't want to do research on this,
you say? I want a buy one. That's the bad news. The closest thing you can get right now is a big
crisp company sells this very nice light host decks, which I believe we have a talk later today
with someone used them to locate a robot and they have very nice user interface. It's already
done for you. You can plug it in. It's made to work with the drones, but it works with whatever
you want to put it on. I believe I have permission to say this. They are working on a stand-alone
version of this, which will be out someday, maybe. Hopefully. Now, before we finish, I would
like to say thank you, both to the Bitcoin company and the Leap Survive project because they did
a lot of the, like, hard reversing the nearing effort on, like, getting this technology to work.
I am just the last, on a long list of people who have worked in the business of what you open
source and with other work, none of these will be possible. So, I thank there for the
contribution. And I thank all of you for coming here. I hope you enjoy the presentation.
Thank you. Thank you, Seid. I believe we have time for a couple of questions. Yes.
Yes, it is an interposition system. So, this answer has to be inside of the
the lighthouse permanently. How do you solve this? I sorry, I didn't quite get the question.
The sun's so as to be inside of the lighthouse laser. It should be in the area and according to
valve, the lighthouse projects are 5 by 5 meter projection cone in front of it. So, if you put one here,
it should cover all of the area. I have worked a lot on this come system. You have to be
line of sight. It's an optical system. But this is also why you put multiple bases.
So, your robot, we usually have a cannon feature or some kind of sensor augmentation that we take care
of the dead spot. And by having multiple bases, you will be able to cover more area as well.
But it's very much like a camera or a mock-up. It has to be in the line of sight.
I played around with Leipzig with multiple trackers and I'm using the five tracker.
The problem I always had is if I lose line of sight of one, your whole system de-synchronization
in collapse, you had the same problem or did you solve that? In our particular case, since we're
using this for robot locations, we do what Agno use mention. We don't rely if you're seeing
this, but we use it as a server of GPS. We do that reckoning while we don't have the lighthouse
and at one point we will get one of the pulses and we will be able to calibrate back on that.
The bike trucker has the issue that I think it only relies on Leipzig. I don't know if it has
an internal IMU to do it. Yeah, I think it has, for sure.
Oh, what's the idea here in this? What would it all occur in the store?
Who? I'm currently designing a VR headset, which is powered by Leipzig and here I'm using the
Bitcoin prototype, but in the future I'm going to design my own PCB with more sensors and
Raspberry Pi picot on it. So in the prototype I have 3 PCBs and I want to make a larger PCB
with more than what's and everything combined. Nice. Well, do please take the reference
of the signs if you want to use them for anything?
The latest time we haven't this and them like outside at noon, I'm sorry.
Could you question here so that the laser in the lighthouse is operate in infrared?
So I would like to know if this system works in direct sunlight.
So, yes, they are in fact infrared pulses and they do get interference from the sun.
I haven't tested this system outdoors because it's not meant for outdoors rotation,
but in our lab we do have very large windows and we do have moments of the day where you
got a lot of direct sunlight on top of it. We noticed that you get a bit less range on it,
but the chip self calibrates and removes any DC offset. So it works fairly well even when sunlight
is interference. Haven't tried it outside though, like a noon, no idea.
Go ahead.
Is this one more? No, no, no. Thank you. Did you actually test it on larger
ranges because theoretically you can have 16 lighthouse, which would give you fairly
big range. Did you stretch this out to the max at some point?
The max I have a tester is with the floral base stations. We haven't gone all the way,
but we are working on a project like the whole reason we are doing this is because we want to do
like a thousand robot test bed on a gigantic area. So we will be testing that very soon.
You can see your papers getting posted at some point. Okay. Thank you.