So, I'm Evan Patton, I'm the lead software engineer for the MIT App and Venture Project.
We've been an open source project for about 15 years now, thank you.
And App and Venture is a web-based programming environment that you can use for making your
own mobile phone applications.
It uses a block-based coding environment, somewhere to other languages that we've seen in
the Deverem today.
And the product, though, at the end of the day, is an actual application that you can
install and run on your robot device.
So the project was started in 2008 at Google, it was eventually open source and is now
managed and run at MIT.
We've had over 23 million users use our system to build applications and they've made over
113 million projects amongst them.
We have about 1.4 million users per month, so quite a decent number of people around the world
using it.
But up the core of App and Venture's mission is this idea of computational action and the
principal investigator of our project, Hal Abelson, says that young people really should
be able to use computation as the means of having direct impact on their lives and their
communities.
And this is how they learn because they can actually build something that's going to make
a difference that they can see.
So App and Venture is used worldwide, it's used in pretty much every country on the planet.
We've got millions of users and countries and hundreds of thousands and others.
And again, as I mentioned, we have about 23 million users since we started.
And there are a number of projects, just some examples.
So one project is a group of students and Texas made an app for a blind student in their
class to help him navigate around the school.
Girl and Kentucky made an app to help remind her of grandfather with dementia when to take
his medications.
A group of teenage girls in Moldova made an app to help track water quality.
So they would go in there and measure the quality of water wells and then capture that
data in an app and then you could look up the water quality near you.
Group of young girls in the Darby Slaman India made an app to help schedule time collecting
water so that they were able to make sure that they got water in other resources they
needed for household chores.
And then some elementary school students in Hong Kong also made dementia related app to try
to help their grandparents kind of keep track of the things they need to do day-to-day.
Of course, it's not just restricted to young kids, right?
Like, our goal is that everyone should be able to build an app.
So we ran a global AI hackathon last year and this team from Brazil made this app called Zodengwe,
which is basically they use teachable machine to train a model that they could then run
inside of their application to detect whether mosquito larva were infected with the Dengwe virus
or not.
So I think a target, the eradication of the larva to try to reduce the spread of the disease
and they were able to do that with 95% accuracy so that they could take a phone go out
into the field, take a picture of some mosquito larva and then decide whether they need
to administer pesticides.
So it's a great way of how computational action be used to make a real difference with
what I got the mentor.
We also have a fairly robust community so you can go to community.appinventor.mit.edu
where we have people offering help as well as extensions and things like that.
So briefly, I'm just going to show app inventor, I know I showed pictures in the slides,
this is an app that I was working on earlier.
And what you do is you have this little companion app that runs on your phone as you can
see broadcast on screen there.
And then this is going to connect up and it's going to establish a session between my phone
and the app in my browser.
And this is typical of what you might want to do if you are using an external device
via Bluetooth, right?
So if any of you have people here used Bluetooth before for various things like actual
programming Bluetooth a little bit.
So internally underneath Bluetooth, you have all these UUIDs of things as you can see here.
Here are the UUIDs for talking to a micro bit and I've got my little micro bit connected
here as well.
And so this is what you may have to do.
But app inventor, we want to try to make things a little bit easier to try to give those
higher levels of abstraction so that it's not as difficult for people to make use of these
devices.
So in the context of this app, or I could go back, let me just go back here to my design.
And you can see how things work in real time.
I can, for example, let's say I want to make all these buttons fill the screen so I can
just say, you know, fill parent.
And then as I make my changes, you'll see that the phone updates automatically.
So I can very quickly get a sense of what changes I'm making and how they're going to
potentially affect what I see and how I interact with my application.
But this is really messy, right?
Like I don't want to have to deal with UUIDs and all that stuff.
So in app inventor, we did come up with a way of making extensions to the system.
And so that's what I want to dedicate the bulk of my talk to you today.
There's just a proliferation of Bluetooth being able to devices these days.
I've seen a whole bunch of them today walking around to some of the other devarums, some
of the other workshops.
So I thought it'd be a great opportunity to talk about how you can do this in app inventor.
So in app inventor, because we're actually building real native apps at the end of the day,
you're writing actual code underneath to make that happen.
So extensions in app inventor are written in Java.
Because of that, they're Android only.
They're also political reasons why they're Android only.
And you can use them both in live development environment and compile apps.
So for example, by pop back here for a second, and I go back to the designer and I expand
the extensions category, we have this Bluetooth low energy extension, which is one that we
publish from our group.
And then the microbit example, which is the one that we're going to go through today.
And what I can do here, if I go back, is these are being loaded in live companion.
That's the app that I just showed you on the screen.
But you can also compile your apps.
So it's beneficial to be able to do that.
There's a caveat when you're building and designing an extension.
If you need to add new Android permissions to make it work, you can only use it in
the compile versions.
And that just is something to do with how they Android system manages permissions.
So if you're interested in building an extension for app inventor, there's really just two
prerequisites.
First, you need the JDK.
Again, because we're talking about building Java.
And then we use Apache ant for managing our build system.
Again, it's a fairly old project.
So things like Gradle didn't exist back then.
And then there are a number of different ways you can design and build extensions.
What I'm going to be showing today is the first one, building it actually in the source
tree of app inventor.
We provide an extension template, which is a little bit more lightweight.
It makes it a little bit faster to build extensions.
And then there have been a couple of open source projects on our community that other contributors
have made, such as Russian fast, but I'm not going to go into those today.
I think the rush posts is something like a thousand posts long at this point.
Definitely a very robust discussion about how to use it and make it better.
In terms of our extensions, if you actually look at one under a microscope, it's really
just a zip file.
And it contains the following items.
You'll see an icon, for example, to show up in app inventor.
And then we've got both the Java compile classes and the Android compile text file to be able
to load it into the companion app, and then additional metadata for exposing the functionality
in the blocks, and when you're building your applications.
And so this is where the talk gets a bit code heavy.
I have uploaded all the slides, so the pre-tox system.
If you want, you can go and you can literally copy and paste all the code into a file and
it will compile.
But first thing you need to do is you need to be able to clone the extension repository
because we're building an extension for Bluetooth.
I've gone ahead and I cloned it by cloning the Bluetooth low energy extension branch from
our extensions repository.
And then this is sort of your standard Java pre-ample.
You have to have a package.
I created mine called microbit, and then I imported a bunch of standard dependencies that
you need for app inventor, including some annotations and runtime utilities and things like that.
And this is the most basic form of an extension.
You have some metadata.
You need to tell app inventor about both things itself, such as a version number, which
is shown to the user if they need to know what version they're using.
And you need to tell it that it's an extension, obviously, and then you can give things
like a help URL, for example.
And then you need to tell the system that's actually external to the system.
By default, app inventor kind of assumes that everything you build in the source tree is
intended for use and app inventor, and the cell is a known app quite.
And then the rest of this is just a standard Java object.
If you compile of this, it will work.
You can go and you can run ant extensions in the build tree, and you will get an extension
file out.
It will not do anything useful, right?
It just simply shows up.
And then it's in your app, but you can't do anything with it, because we haven't given
a functionality yet.
So kind of going back and looking at this, right?
So we have these UUIDs that the microbit folks have defined for interacting with a microbit.
So what I did is I moved those over into Java.
And we need to have a reference to our Bluetooth connection.
So that's in there as well.
And then when we think about app inventor components, we're really thinking about four different
types of blocks that they use really interact with.
And we're going to have to implement one of each of these to make the extension really
work.
So the one on the top left there is an event handler, app inventor's in a very vent driven
system.
There's all sorts of stuff happening on your phone, and you need to be able to spawn,
or at least to be able to spawn to those events firing.
So we can say we have a message received there from our microbit.
Likewise, we may need to send a message.
So we'll want to define a function block, like the one on the top right.
And then we need to be able to manage our references to our Bluetooth connection, which
is how we actually communicate with the microbit device.
So when we're defining an event, it's very straightforward.
Events are really the simplest blocks in app inventor.
All you need to do in your function is say that it's a simple event.
This case I'm defining the message received event.
And we know that it's going to receive a message from the device.
And an event dispatcher is this very special object, which is responsible for understanding
all of the internals of how app inventor works and how to send an event to the blocks
that are executing the result of that event.
So inside of this block here, you would put the behavior you want to do when the messages
received.
And so app inventor, the event dispatcher, knows how to find that and run it.
This is where you get some integrated interns of dealing with all the Bluetooth stuff.
So here we have an on-receive handler from the Bluetooth system.
And so we're just saying when we receive a message, we're going to dispatch that through
that message received event that we saw earlier.
But of course, we need to have a way of establishing that relationship.
So when the Bluetooth connection is established, when we get this on-connected event,
we need to register for strings from the microbit.
And then likewise, because we want to be good citizens, when we disconnect, we want to make
sure we unregister.
So we're not leaving around, you know, handles that would be otherwise garbage.
So the next block or the property blocks, here we've defined that we have an editor type
for this particular block that's a Bluetooth low energy.
So it only accepts Bluetooth low energy components.
We don't want to hand it a button, for example, because how do you have a microbit
talk to a button?
It makes no sense, because the button exists on the Android phone, and the microbit exists
on a microbit, it's a separate thing.
And so we handle our connection listeners there.
And so this will tell the system that we want Bluetooth.
And then likewise, you have a getter block, and the properties can be categorized.
So I've said here that a Bluetooth device is a behavioral thing, right?
The other options.
Appearance, clearly, there's no appearance here, because it's an external device.
That's managed separately.
And then our last block, as I said, is going to be our function block.
We want to be able to send messages to the microbit.
So when we get our connection, if it's not null, we can call this right string with response,
give it our UUIDs, and then we'll have a right handler, which comes back.
I'm going to skip that one because it's not really necessary.
And then lastly, this message sent block is going to be dealing with the right handler.
So after we write our message, we'll get a message back, and we can dispatch that through
the events.
So this is our final package extension.
And if I go back here, I pop over to Emacs real quick.
You'll see on the right hand side here, I've added all that code from all those slides.
And so this is the final working extension.
And then I had just previously run Ant here.
It takes about seven seconds, because of course it's compiling every app and mentor component
in this version that extension template I mentioned earlier, it cuts all that out, so it's much faster
it takes less than a second.
But that is what the actual compilation process looks like.
So now if I go back, and let me resume my connection here, we'll show you actually working
in progress, maybe, once again, just one end out, restart, okay, so, so now, like you would
have had to do something with this previously, and you don't necessarily want a student
to have to do this, right?
They have to know what you, you IDs to use.
They have to know that the strings they're sending are not UTS-16, or that the device doesn't
want them in UTS-16, and all this other stuff, right?
Like that is a little bit too much for someone who's just starting off with something like
the micro bit.
So we're just going to go and delete that.
And then I've loaded my pre-compile version of the extension in here, we see those blocks
that we just created in the slides.
Instead, I can just drop in a send message block, and I should have saved my string from
earlier, but that's all right.
And we're just going to have to say hello, pause them, and then I can add my message
received block that I defined.
And as we said, it's going to receive this message as a parameter, and to keep things
simple all I did is I created the label in my app earlier to display that message.
Okay, I'm going to just make sure my connections live, too.
I'm just going to change one of the colors of these buttons to make sure it updates.
Okay, so I'm going to go ahead, and I'm going to, now the micro bit here, you'll see
it's got this little frowning face, because nothing's connected to it.
So I'm going to connect, and it should change us to a smiley face, because now it knows
that it's connected.
And then I'm going to send my hello fuzz to message, and it should start scrolling.
The micro bit doesn't have terribly fast output, but that's okay for the purposes
of video recording.
So that's fine.
And so now it's saying hello fuzz.
And of course, we could program this app to say some different string or you could program
it to ask the user for a string.
We're just kind of trying to keep it simple for the demonstration.
And if I program button A to say button A when I press it.
So if I press this, we should see my screen update.
And if I zoom in here, oops, why did that happen?
There we go.
You can see now my message says button A.
So we've managed to reproduce the entire loop of going to and from the micro bit, but
rather than having to worry about the complexity of managing these UUIDs, right?
Instead of having to do this, the student can just, well, that's stuck.
Okay, the student could erase all those blocks, right?
We don't need them anymore.
They're not useful to us.
And so all of that complex logic we might have to write.
All those extra blocks that the user would have to deal with go away.
And so now we can just think about sending messages.
We don't have to worry about managing all that connection stuff, right?
And so that's extremely powerful.
And so just to kind of wrap it up here, how we loaded this.
So and I showed you that extensions panel earlier in the talk.
What you do is you click on import extension.
It will open up this little dialog where you pick your extension file from your file system.
And then you'll say import and then it will load it up into Apple Mentor and show up in your palette.
Here are those example blocks for reference.
Again, just so that if you go back and repeat this, you have access to everything.
And then just to wrap up with some resources, if you want to learn more, including seeing
all of our actual teacher materials around using micro bit in your classroom with Apple Mentor,
you can go to Apple Mentor.mit.edu and there's a whole IOT section in a data science section that's available for you.
Building your apps, you can go to ai2.appymetro.mit.edu and that gives you the code editor.
We were just looking at.
We're open source under the Apache license so you can feel free to contribute.
You can fork us on GitHub.
The community I already mentioned, a great resource for building your own extensions.
There are a lot of people there, publishing extensions are willing to help out new contributors.
And lastly, if you have any other things that you might need for me, you can feel free to reach out to me.
That's my email there at MIT.
And so with that, I'll take in the questions you may.
Thank you.
No questions?
Okay, so I can show you some more stuff.
That's fine.
I'm always good at showing stuff.
So that's the fuzzdom stuff.
I was giving this demo.
This is not an IOT related demo, but it's one of our data science classroom materials.
So if you're teacher and you're interested in teaching things like data science, this is a great little tutorial that we made.
To help you teach students about things like lines of best fit and cleaning up data and doing data analysis.
And of course, you could connect it with something like the micro bits.
We do all sorts of workshops where, for example, we'll take a football and American football, but you could do it with one around one as well.
And you attach the micro bit to it and throw it or kick it.
And then you get the accelerometer data and the students can visualize the accelerometer data in the app.
But I can come in here and I can make predictions.
That's going to load up this prediction screen.
I'm going to say show data and this is going to read some data that comes from a Google spreadsheet, which is the spirit lake spreadsheet.
So somebody's been going out every year since the 1950s and counting how many days that particular lake has had ice on it.
And so you get this kind of very rough idea of how that lake has ice coverage over the course of the year.
But you'll note that there are these two points towards the bottom of the graph, which are zeros and those seem a bit odd.
And so we have this anomaly detection algorithm that does a Z score based filtering.
So if I click on that, I can see that these data points are really out of range.
They're outside of the standard deviation by more than two standard deviations.
And I can actually say, okay, well that's very interesting.
But what if I figure out the line at best fit for this?
If we figure that out and it does the calculation for us using the standard RSME, we have doing that.
We'll find that eventually if this trend were to continue in the long term, we might expect that this lake will have no ice around the year 2175.
But the students can say, well, wait a second, what about those zeros?
Those are not good, right?
Was there really no ice on the lake that year?
Well, it turns out, in fact, the person wasn't able to go out that particular year.
So they still recorded a zero, which was not great.
But I can click on these and I can delete them.
And now my trend line extends out to 2280, right?
And so now it's like, okay, well, if we actually remove the bad data, we can see that we get this extension of when we think that the lake might not have any ice anymore.
And another cool thing is, in app inventor, we have a lot of blocks when we talk about simplifying various things.
Let me go find it here.
So this is AI analysis button, which I'm going to click.
And what this is doing is this is giving a prompt, they can construct a prompt and send the data from this spreadsheet off to a large language model.
In this case, we're using OpenAI, but we have an assortment of different models including some like Lama that you could self host.
But eventually it comes back with this explanation here of what it thinks is happening with the state of and the potential implications for what that is.
And so students can start to tie together all these different aspects of statistics and data science and artificial intelligence and build it all into applications that they can impact edge and share with people.
And so those are just some of the other ways in which we've been taking these really complicated things and making them very simple, right?
Like just calling the interface for this large language model is one block, right?
It's not constructing a JSON object or a whole series of JSON objects for the conversation.
We manage all of that under the hood.
And so you get this very high level interface for doing these very complicated tasks.
And so that's one of the real powers of app and then turn what we're doing with these block languages.
Thank you.