Hey everybody, thanks a lot for coming, much appreciated.
So I am dead program, some people call me Ron, a couple of humans call me dad, that's not
really too important, I am dead program, I'm not dead though, not yet, I'm a technologist
for hire, that means that I will, in fact, in exchange for fiat currency, create code of
indeterminate quality naturally, and I have a small boutique consultancy called the hybrid
group where we create the software that makes your hardware work, and some of us work
on a few open source projects, one of which is tiny go, so a very cool project,
Graham, talked about it a bit, and you're going to hear using tiny go, that's a temporary
condition, thank you by the way, also go CV, which are the go wrappers for open CV,
so computer vision using web assembly, first question, why, I mean why would you do that?
Well, I'm glad you asked that question, so what we're really talking about is an industrial
computer vision systems, right, like the serious stuff, generally running on an embedded
Linux device, so we're not talking about the browser at all, forget the browser, I mean
don't forget the browser, but we're not talking about web assembly and browsers, usually
edge computing devices of some kind, so they're very close to the problem they're trying
to solve, and often also cloud computing devices running in giant server forums and
mysterious locations, so what's on my application is like agriculture, transportation,
energy, security, and of course manufacturing, which is what I've been doing the last
few years building we do a little batteries using some of this stuff, so a lot of computer
vision applications have the same basic structure, really the same basic design patterns,
but there's lots of different parts to integrate, and naturally hard to run on different
kinds of hardware, if you have a heterogeneous environment in your factory or something
like that, you know generally it's been quite difficult to get it to work, also very hard
to update, I mean who, who among us is not brick to really important device, no, no blame,
no harm, no judgment, all right, and there's also some need for customization, it's not
just a matter of tweaking a few parameters, but there is some intelligence behind the people
need to configure these systems, in order to get them to do whatever the job is, they're
supposed to be getting done, so that's the reason why we created wasm vision, which
is basically intended to help you get going with computer, I handling the basic design
patterns that you would normally do, which is capturing video, processing this video in some
sort of way, and then saving it or streaming it or making some type of rest or GOPC or
other type of call to get something else to happen as a result of whatever it is, ingest
some data, so wasm vision consists of command line interface and the wasm vision engine
and then wasm vision processors, so the engine is written and go, go is a great language
for concurrency and you could do a lot of things at one time, it then combines open CV, anyone
here use open CV or no open CV, open CV is really the gold standard as at world computer
vision, it's been around for I don't know 15 or 20 years, it's got hundreds of not thousands
of algorithms, untold number of students and postdoctoral candidates have worked on open
CV, also ffm pad, which is really an amazing open source tool for processing video and
G streamer using GOCV to control all of this, and of course was zero, we've got a one
of the maintenance of was zero here, give a big hand, so we all, all of this is combined
into a single statically linked binary with absolutely no external dependencies, yes, this
all combines together, so here's the architecture on a weight, sorry, different architecture,
you can see this, so here's the actual architecture of wasm vision, so the wasm vision
engine, we capture some video, then we process that using the wasm runtime in this case
was zero, each one of these frames is then passed to an individual processor and we can
do different things by either capturing or outputting the different streams using those
capabilities I mentioned earlier, so wasm vision processors, they can be written using
GO, specifically using tiny GO, obviously we like tiny GO, they could also be written
using rust, very cool language, and C, yes, no one ever got fired for choosing C for your
industrial system, right, I know everyone's here is like yeah, but do I have to, I don't
know, you're going to have to talk to someone else if you have to, but you can, so let's
is all possible because of something called wasm CV, so wasm CV is in a series of interfaces
for computer visions, specifically for wasm, it's one of the projects that we released
the end of last year, and it basically, these interfaces specifically for computer vision
based around the patterns used in open CV, so it's defined using wasm, what is the, what
assembly interface type definition, it's like an IDL and then face definition language,
and it's part of the wasm component model, however wasm vision and wasm CV are not actually
using the wasm component model, reasons for that, just we don't actually need it, we're
just using with to generate the wapers that we use in these different programming languages
and also the documentation, also important note we're using on the wasm, something called
wipes that was actually created by Orcenium by Graham who was give a great talk earlier,
on the wasm, that's how we're doing the integration, so on the client, we're using
these wapers that are generated using wit with wasm CV and then on the host side we're
using wipes, so quick review on how computer vision works, the basic fundamental unit is
the mat, or the matrix, so I actually explain things with ledos, why I understand them,
as best I can, so then the mat can also have red, green, and blue colors, actually generally
it's blue, green, and red that that's not important right now, look if they're stacked
together in this kind of order, and this is just the way that this information is represented
inside of OpenCV, then with OpenCV, the capture process takes a mat, passes the type
of image processing, like perhaps a Gaussian blur, and then passes it to some other type
of processing, for example, a deep neural network to do some type of other processing or
recognition, all right, so how wit works very quickly, it's got interfaces that we define
collection of functions, it's got basic types, like floats, ends, and such that we can
define, it's got something called records which are basically like strokes where the information
is actually passed between the host and the guest in web assembly, and then resources,
the resources are very interesting and important as you'll see, it's something that's actually
entirely owned by the host, but that the guest can call, so the information is not actually
passed over entirely into web assembly, some people would like to do everything in web assembly,
I'm not actually a fan of that, and you'll see why in a few minutes. So let's pick a very,
very quick look at the wit file for the world, used by WawesomeCV, so we see we have a world,
it's got basic types, including the mat, and if we look at the mat, we see that we've got a definition
using a resource, we can construct a mat, clone it, close it, see how many columns and rows are in
that image and so on, and then we've got different basic core functions, for example drawing circles,
rectangles, and in this case performing a blur function, all right, so this is the definition
of in the wit format that we're then going to use to generate our wrappers, so everything is
generated from these wit files, we don't do anything manually, so the WawesomeCV docs themselves
are also generated, so here we can take a look at the automatically generated docs created
for the mat that we just saw before in the IDL, so let's take a quick look at some awesome
vision processors in go, who here is used go, excellent, so you're familiar with packages,
so WawesomeCV is just a go package, so we just say package WawesomeCV, and everything is generated
from those wit files using wit bind gen go, which is a set of tooling maintained by the
bytecode alliance, which helps generate very nice idiomatic wrappers for go from those wit files,
and you build it just using tiny go, you just say tiny go build output blur.wawesome,
our target is Wawesome Unknown, we generally don't use WCP1 or P2, we're just it's raw,
hardcore minimalist web assembly, so let's take a look at the code for the blur processor,
very quickly, so here is our tiny go program, package main, like any other go program,
we import WawesomeCV, the mat, and types, and then we have a process function, this is really
the entry point for all of the Wawesome vision processors, and it always is the same, it passes
in the single parameter, which is the image that we'd like to process in the form of a mat,
and then it returns a mat, that way you could do whatever processing, and then return the image of
whatever it is you ended up with, and so in this case we're going to say CV dot blur of a kernel of 25
by 25 by 25, so a 25 by 25 pixel blur area, we're going to log some info and then just return
that blur image, so now let us see if the demo gods will curse me or what they will do, so let's see,
make blur, all right, we're running, so you're wondering where do I see it, but of course in
WawesomeVision we stream everything that's totally headless, right, so I have one, I'm very blurry,
nice to meet you, the two are the supplies, all right, thank you, so
Rust, WawesomeVision processors and Rust, it's a crate, of course it is, who here is using Rust?
Cool, very cool language for doing things in Webassembly, so we use WootBineGen, which is one of the tools
used for doing these rapid generation, again idiomatic Rust is being created, and how we build it,
cargo build, again target Wawesome Unknown Unknown, because we're going for the most minimal possible
module in size, and then let's take a look at the Rust code, so Rust code is a little bit longer,
but not very much, no standard because it runs raw, bare, no operating system, we've got our
external crate that we're bringing in for WawesomeCV, and then here's our external function,
again it's a process function, what does it take, as is parameter, a mat, what does it return,
a mat, and of course what does it do, it calls CVBourer, it seems to come from familiar, right,
passing in the kernel of 25x25, it's doing the exact same thing except in Rust, and then we return
that process image, so now let's see if the demo God is once again, so if we run,
all right, so that seems to work, so far, so good, and then still blurry, it's the exact same
blur, it works in Rust, amazing, more to two of the supplies, and of course the programming
language of our forefathers C, it literally is my grandfather's language now, so we include
Heather Files in C, who here is UC, who here uses C frequently today, not too many,
it's not a dying language, it's just sort of on the lawn, kiss, good night, so we use with my
gen against, to generate the things, and naturally it's very easy to compile your C code using
playing, now to be fair, both the tiny doke compiler and the Rust compiler are doing all of
these same things, it's just they're hiding in the way from you, now hopefully that's all right,
as long as you know what it's doing, if you ever do run into a problem where you don't know
however, it might be a little tricky, so I guess maybe that's one benefit, let's take a quick look
at the C processor code, so here we've got, again, the process function, we pass to the mat,
we do our blur function, passing that size of 25x25, and then we return it,
you see a pattern for me, here, by any chance, and so if I say bake blur,
oh, make it, I don't know what make it is, so if we say make blur C, once again, we're running the same thing
and I'm still blurry, amazing, three languages, same blur, all right, let's take a very brief look
in a few of the processors, how we do it on time, we're running out, very little to do it lots
of time, no wait, strike that, reverse it, so basically the processors we provide them,
basic building blocks built in, but they can be trained together so you could take the
output of one to the next one, you know, because again, it's a pipeline, so the pattern,
you know, it's a create your own processors, so a quick tour of some of the included ones,
askify, this one is a particular favor of the mine, so askify's written and go,
we process our mat, we take it and resize it, we then convert it to asky art,
because we're actually living in the matrix, I mean, I thought that was what was happening,
and then we return that, so let's see if it works,
I'm actually, I should open now, let's go for it, here I am, hello, I'm alive, wait,
I'm in the box, I thought it was supposed to fit outside the box, now you might be wondering,
so what about the stringiness, like it's still screen, I look pretty good in the form of bits,
I have to say, all right, let's keep going. The mosaic processor, so we could basically call
any of the capabilities that OpenCV provides, one of them is loading deep neural networks,
so I really like the capability of taking a fast neural style transfer of network and putting
myself through it just to see what happens. So again, process function, now one thing worth
noting here is that we're calling, then, not reading and we're loading this model, so
while some vision can do the same sort of thing as a lot of other cool hipster software nowadays,
which is, if you don't have the needed assets or models or files that will automatically
download them for you, so this will download the mosaic nine-on-x file from one of the servers and
put it on your machine if you don't already have it. So we take it, we convert our image to this
blob, which is the format we need to pass into a deep neural network. We set our networks
input, we pass through the network to do the processing, and then based on those results,
we're going to take them and then display them back into this new image that we return.
All right, so a lot of stuff, but quickly, so let's see what happens in the land of the mosaic.
So far, so good. Yes, and here I am in the form of an artistic mosaic. This is my best form.
Now, this is all processing entirely on the CPU in my machine. I'm not using any GPUs whatsoever,
and it's got actually pretty good performance, so don't sell your Nvidia yet, but
so let's stop that real quick. All right, one of the processes we have uses O-Lama,
and it'll be here here at the Royal Lama. So one of the capabilities of O-Lama is
whether known as vision models. Vision models are generally used for one of two things. One is create
images from text, the other is create text from images. That's what I'm going to do. So here's a
quick look at the architecture. We've got the awesome vision engine that's doing our capturing.
Then using the O-Lama processor, it's passing each frame to the lava processor in O-Lama.
Let's take a very brief look at the code. So same pattern, once again, we process. We love the
configuration. We create some JSON that we're going to pass to O-Lama. We pass that JSON along with
the image that we've just captured, and then let its results come back, and then display them.
So, let us see what happens. Actually, first, let's make sure I'm actually running it.
And it doesn't look like it. So we have a stalker starts O-Lama.
All right. So it's running. So if I say
stalker, exact O-Lama, and run the nano lava model, which is a very small Lama model.
All right. So let's see if it's actually working. This one does use the GPU.
So if I run Nvidia SMI, I'm not seeing a Lama sharing up in there, M-I.
It should be. I guess we'll find out very quickly if it's actually doing it or not, won't we?
All right. So if I say make O-Lama, that should now run our processor. So how do we know if
it's working? Well, it should actually display some text. So it should be looking at me passing it
through the deep neural network model using this lava model. And then, if all works well,
the fan kicked in. So this is the CPU. I guess I didn't get the GPU to work.
But it'll still happen. It'll just be very slow. So in a little bit, it will come up with a
description of what it thinks they're seeing, which is hopefully me. Now generally, it's not very
complimentary. It makes it actually kind of kind of rude comments about my physical appearance to be
honest. I hear it shurning away. It's thinking very, very hard. We're getting so close.
Well, now the fan is working. So it's doing something. Oh, wait, here's something. The fan just
kicked in doubly. Now it's really trying hard. We'll give it one more minute and we'll go to the next
thing because I mean, who has time for this? Plus, it was a little gratuitous. I'll be honest.
All right, three, two, one. Sorry. No, a llama today. All right. But you got the idea.
It actually does work if I got in the GPU on. All right, video drone. So this is a flight control
station using face tracking with a machine learning model. For this, I'm going to use my DJI
tele drone that I brought with me and plug in my dual shock joystick that I stole from one of my
kids. Just as like I've spent a lot of time with this particular control surface, I will say.
One of my favorite things, playing with joysticks. Here is the actual drone. Let's see if we can
take a look real quick if I say make sure video. Then I should have my camera. Yes, here we go.
So we can take a look here's the drone. To tell our drone, I plug it in. I must turn it on.
This is breathing. We know that because there's a light. Wait, where's the light?
Maybe I turn it off. Oh, there we go. Okay. Excellent. I've got the joystick. You see that there.
I'm tangled in these cables. But that'll be fun. For this, I'm going to use something called
Gobot, which is written and go and I'm going to use a unit face detection model. Quick review,
computer vision models, take the images, pass it through different layers of neural networks,
in order to basically come up with a prediction about whether or not they see a face. With video
drone, we're going to fly the drone using that joystick. We're going to use Watson vision, the
process, the images, and then based on that, we're going to display whether or not we actually see
human. The quick look at the code. We start the joystick. We start the drone. That's basically it.
The one thing that's interesting is we're going to proxy the video from the drone to Watson
vision because remember it's got capabilities for streaming and we can do that. And now let us see
what will happen. First we must connect to it. We'll see the drone. Not that one.
I was a little too quick on the click. There we go. So we should be connected to the drone.
I am plugged in. We are running, I don't want to run that. I want to say run drone stream.
Switch over to my other one. I have to run multiple versions of go at the same time just because
I'm like that. Fasten what use it? That's right. I forgot. If we make drone, I'm not the
drone stream. I guess it would be video drone. So theoretically we're connected to the drone.
That's actually flashed over to the video so we can see where that's seen.
Not seeing anything. Should be connected. Oh, there we go. That's better.
All right, so that's the drone. And let's see, trying to remember the controls. I forgot to
look at the controls. That's always fun. Let's see. As I recall, this makes it fly.
Yeah, that was it. All right, so let's see if it's actually, oh my god, it sees humans.
Now, one thing that's really good to know about drones in general is that battery life is very limited.
So you don't have to be faster than the drone just faster than your friend. I think it's how that
one does. It's about an eight minute battery life. I probably should have warned you in the front
row before I threw the drone in the air. Yeah, all right, so it takes a look at the pilot.
Takes a look at the other humans. Let's get some altitude.
Yeah, there's quite a few of you in here aren't there. See, oh, this is it.
Awesome vision with live drone flight. It actually works.
Well, I'm probably out of time and I've already done enough for two of this flying. So, come home
little bird. And that's it.
There's my landing messages land land land land land land are really please land. All right, so
the end is here. So was that fun? Did you have fun with that? Yeah, yeah.
For industrial stuff, that's pretty fun, right? So version 0.2 just was released right before
fast. I'm conference driven development. It is a thing. Check it out,
watch them vision.com. Follow us on the featherverse. So I'm blue sky. And thank you very much.