I want to start with a quick, about me jumping to a high level overview, jumping to a demo
of the project, do a high level overview of how it works together and I'll wrap up with some
quick lessons learned.
As you might have noticed from the title slide, I'm an engineer at Apple and platform architecture.
I've worked on a pretty wide variety of projects, but some of you might know me from
our SIFPLATIVE examples of what I post I publish about a year ago, SIFPLATIVE examples
at WWVC session from the previous summer that I worked on with Cuba and SIFPLATIVE examples
of repo I help maintain.
SIFPLATIVE examples is a collection of small demo projects to get you started on your favorite
microcontroller.
This is a range of devices, but the primary goal of this repo is to get you into Swift
and you can take it from there.
This would be things like linking an LED or talking over the most complex cases, controlling
a new pixel LED strip, and so this doesn't really constitute a real product, and so that's
what I'm here to announce.
I'm announcing a new addition to SIFPLATIVE examples, which is a full product you could build
and embed the Swift.
Without further ado, let's take a look.
So this is Harmony.
It's a Bluetooth speaker and Fairfluent Music Visualizer.
I'm aloft in white, there are stereo speakers.
On the front there are some human input controls, like playpods, buttons, and a volume knob.
And dead center, there is a Fairfluent.
So Fairfluent is a iron liquid that reacts to magnets.
So by behind the Fairfluent is a Ultra Magnet.
So I'm going to connect to Harmony and play some music.
This is Bluetooth digital volume, flight sensor headphone, or on the knob.
And as the beets, you can set a Fairfluent react.
How do I just go on?
So I'll come off there.
So let's take a look at how it works.
I'm going to focus on the firmware of Harmony, and not the virtual and mechanical parts.
At the bottom of Harmony is the PQ SDK.
This provides a whole bunch of functionality.
The one critical piece of the hardware API is it provides.
However, API abstractions it provides.
This allows us to control things like GPIO pins, I2S blocks, I2C blocks, and PIOs.
On top of these abstractions, I built an audio driver that can speak I2S.
And now we do control the audio pixel strip on the front and callbacks for the very
human input controls.
The PQ SDK also provides a Bluetooth stack.
So on top of that, I implemented the hand-lace for the various Bluetooth packets and wrapped
one of the audio decoters in a nice Swift API.
Additionally, bridging these two halves together, I use the CMpsus DSP APIs to create
an audio analyzer.
All of this together forms the Harmony firmware.
Now I want to dig in to how the audio data flows through Harmony as it's playing music.
Now, it all starts when a controller, a phone, or a Mac, connects to Harmony over Bluetooth.
Once the controller starts playing audio, Harmony receives encoded audio data which it stores
in a ring buffer.
Once that ring buffer reaches some threshold value, Harmony begins decoding the compressed
data into uncompressed PCM audio.
The uncompressed data is chunked into frames of 256 stereo samples and sends over to the audio
analyzer to handle.
The analyzer performs an FFT in real time to determine how much low frequency content is
in that frame, and it based on some threshold value chooses whether or not to power
the electromagnetic.
The frame and this bit of data are then passed to the audio driver to control the hardware.
The audio driver takes the raw audio data, sends it out as I2s to a digital analog converter
who signals them amplified and played out the speakers.
That single bit we had earlier goes through a different analog amplification circuit and
controls the electromagnetic.
There's a way more details about how this all works together.
So if you're interested, please go out and have to answer any questions.
I want to wrap up with some quick lessons learned.
The first is to find great mentors while I was building this project, the first 50% of
the work, I'm turning up to be working on firmware.
The next 50% was the chemical design, and then there was another 50% that was a electrical
design, and then at the end I didn't even know this was going to happen.
There was another 50% that was chemistry.
The point here is that I could not have done this alone, and I relied on the insight from
great mentors to guide me to the process and help me uncover the bits I didn't know.
So I'd like to do a shoutout right now to Kuba, Mejir, and Kevin for helping me put this
together.
All right.
The next lesson learned was to start with something that works.
When I started this project, I wanted to start with an empty Sophia file and be like,
yeah, I made the thing from scratch.
That was a terrible idea.
I quickly realized the scope was very large, and I was solving problems I didn't really
care about.
So the actual solution I ended up following was finding some example C code of how to control
the Raspberry Pi's Bluetooth stack, building on top of it, and molding it into the project
you see today.
The takeaway here is building from nothing, like going from zero to one is really difficult,
and often has you focusing on problems that aren't important to you.
Find a good starting point that allows you to solve the problems you care about, adding
or the other things.
All right, so I was a really quick overview of Harmony.
All of the resources needed to build Harmony yourself are available either right now or very shortly
on Swift Embedded Examples.
This includes the embedded software, the build of materials, and shortly the electrical
schematics, and 3D models.
And please check out Swift Embedded Examples if you want to build one of your own.
I'm excited to see what you guys do with it.
Thank you.