Right. Hi. I'm Merlin. Yeah, I'm excited to be here today. Well, so many people. Yeah,
wasn't originally scheduled to speak today, but I jumped in as a backup. So, haven't
had quite as much time to prepare as I'd like. So, I hope you forgive me if it's a little
rough on the edges. But yeah, I want to talk to you about live coding with Kotlin and
Compose Multiplot Form. Yeah, so I'm a full second engineer. I do a good bit of Android
development. I've had a casual interest in live coding for some time. And one thing that I work
with a lot, the programming language I work with the most is Kotlin, and I work with this Compose
UI framework. And I have this idea to repurpose it for live coding. So, I have a couple
questions for you. Who here has used Kotlin? All right, quite a few. Who has used Compose
Multiplot Form or Jetpack Compose for Android? Fewer, but a few. Okay. And he has used React.
More people. Okay, great. So, I will draw some analogies between Compose and React throughout
the talk. I hope that will help some of you make the content concepts more accessible for
some of you. Yeah, so let me show you what I will be talking about. Oh, this is not.
No, I have the slides here, but I can't. Can I? No.
So, about this. There we go. Okay. So, yeah, what I want to talk about is firstly,
Kotlin and the Compose Multiplot Form, then a little bit about the function or declarative
paradigm, which underlies Compose, and which is also popular with live coding, then I will
introduce the little library I have been building, which I call the couple. And talk about
why am I repurposing a UI framework for music for live coding. I talk a little bit about
the fundamental building blocks of UI and draw some parallels with fundamental building blocks
of music. I talk about state management and how the couple works under the hood.
I talk about implementing these musical building blocks, DSL for expressing rhythm
and connecting up building blocks together. Yeah, and then I talk about the
Multiplot Form element of it, so Compose Multiplot Form, it works on different platforms,
desktop, Android, iOS, web, and finally integrations with Compose UI and visualizations.
So, firstly, Kotlin and Compose Multiplot Form. So, Compose Multiplot Form is a declarative
UI framework with a functional paradigm. It's similar to React. It's not going to connect
slide. There we go. There are similarities with React, which is a JavaScript framework and Compose
is for Kotlin, basically. Kotlin started out as Java with less boilerplate. It was made by
some frustrated Java developers. Yeah, it's become a flexible Multiplot I'm language with support
for the web, with support even for iOS. So, yeah, that's the Multiplot Form element. Kotlin is
my main language and I'm using Jetpack Compose, which is the Android version of Compose Multiplot
Form and the version that they made first before they made Compose Multiplot Form. I use that a lot.
So, this is something that I'm quite used to using and I thought it would be fun to explore
what else can be done with it. Yeah, so Compose is the standard way to create UI's on Android
and also now with Compose Multiplot Form in Kotlin generally. Yeah, so the functional declarative
paradigm. It's popular with live coding as well. Some prominent examples, Haskell's title,
JavaScript's scoodle, scoodle closure has the overturned, they're all kind of in either functional
language or in the case of JavaScript, it's just a functional framework. So, this functional declarative
paradigm, meaning you specify what you want and not how to go about it. We'll see that in a
minute as well with some examples. Yeah, so while Kotlin is not a functional language, it
supports that paradigm and the UI frameworks compose and react above built on it. And what I noticed
when looking at these different live coding frameworks, it didn't find any that were actually built
on a UI framework like this, so there's nothing for Compose and I checked for react and I didn't
find anything there either which surprised me given the size of the react ecosystem and I wanted to
try this out. Yeah, we've tried this out with what I call a couple. So, leveraging Compose
Mountain platform for audio generation and live coding would be available under the MIT license,
it's still in the early stages. So, it's under heavy development right now and yeah, you can't check
it out already but I would recommend maybe checking it out at some later point when it's a little
further developed. So, we're talking mostly about the concepts underlying it in this talk and like the
particular interesting parts of using a UI framework like this. So, yeah, why am I doing this?
Why would I repurpose a UI framework for live coding? Yeah, it turns out basically that a lot of
the features that are useful for your eye development can be repurposed and it loads the barrier
to entry for developers who are already familiar with the UI framework in this case for Compose
developers. There's a lot of work that you have to do when you make a live coding library and some
of that work can be offloaded to the framework. So, I'm using it for state management, for animating
values, for dynamically adding or removing blocks that build up the audio basically. And yeah,
intelligent and studio also has like previous features for Compose where you can preview the UI
right in the IDE with live updates which is also quite handy. Yeah, so, fundamental building blocks,
I'm going to talk about what the fundamental built-in blocks are in a UI framework and then
draw some parallels with music basically. So, yeah, in a framework like Compose or React,
you have some fundamental blocks, you have layout elements, something like a row to put elements in or
call them. You have the actual UI components, you have buttons, you have text fields and so on.
You have state, state for values that leverage whatever state management your UI framework provides,
you have values that change over time animations. You have effects. So, performing an action whenever
a component enters the tree or leaves the tree. When I say tree in Compose, I mean the composition
in React, in this case it's the DOM basically whenever a component gets mounted or unmounted.
And you also have a need to sometimes make a values accessible from anywhere in the tree
like a theme that you want to apply to every component. So, how does this compare to the requirements
from music? So, rather than layouts and in music, you have timing. You want to play something at
the same time as something else, you want to play it in order. You want to express with them.
You want to play sounds. These are like your UI building blocks like a button or a text field,
you have instead like a spin or playing some samples. You have again state for values,
values that update and change the music as it plays and for these we can leverage the state
management of the framework. You have values changing over time to make the music dynamic and again
you can leverage the animation capabilities of your framework. You need to schedule and cancel
audio. We look at how Dacapo does this in a minute and this is something where we can leverage the
effects that UI frameworks provide. And then you have things that you want to be available
throughout your program like volume control or setting a BPM that's consistent for all the components.
And this is much like the feeling that I mentioned with UI. You want to make some of value
available throughout your program. So we start with managing state. So in Compose we have
something that is a true code state and this is similar to the used state hook in React if you
familiar. Whenever the state changes the Compose tree the composition automatically updates.
So we can basically for example set a synth. We say we want to piano synth and then we
we have a block that actually plays some music using that synth and when we change this value
then the synth will automatically update.
Then to actually make it work. So inside of a like this Dacapo library basically we need to use
something to schedule audio and to cancel it again. So whenever a component gets mounted whenever
anything gets added to the tree we want to schedule something, some audio that wants to be played.
And whenever we remove a component we need to cancel that audio, cancel that scheduling.
And yeah to that end Compose provides something called disposable effect allowing us to run
something when the component enters the tree and something else when the component leaves the tree.
And this is similar again to React's use effect hook.
Yeah then let's look at making values available throughout.
Basically for things like a volume control or setting BPM.
This is where Compose provides something called composition local.
This is similar to React's context if you familiar.
Basically a way to set a value and have it be available anywhere downstream in the tree.
So basically we make a Composable function here for an absolute volume where we set the volume
via the composition local. And you can also set relative volume the same way by reading the
current value of the composition local and modifying it.
And the same can be done for setting BPM for setting a relative tempo that say half as fast as the
current one. So this is where the composition local is quite handy.
Then yeah for instance in samples. So actually I'm building these from the ground up from scratch.
And this is fun. It's also quite educational because I'm not as familiar with some of these concepts
and I'm getting some familiar reality with like fundamentals here which is really nice.
So we shed your sense and samples through effects.
And then they can basically be embedded in other blocks to modify the sound.
We want to shape the envelope of the sound like an ADSR envelope attacked decay
sustained release. And for that what should the ADSR that look like?
So there's the option to basically use blocks and just say okay you embed your synth inside an envelope
block and that gives you an envelope shaped for your synth. But there's something more convenient.
So what compose calls modifiers? So these modifiers basically are used for for padding,
for sizing, for setting a border around a component. And I've we purpose this
modifier concept and I'm calling mine conditioners. So here we can basically say we play a synth
and we condition it with an envelope. And we can do the same for effects like if we have a reverb
component we can nest a synth component in it and then we have a synth with some reverb or we can
use the conditioning to basically achieve the same outcome.
Yeah, then basic timings and playing sounds simultaneously as well as in order.
So time primitive like playing sounds in parallel, cycling over different sounds,
one on each beat or splitting up time equally. So we can express basic timings like this.
We can play a and at the same time we'll play b on the first beat. On the second beat
we'll first play c and then on the next half beat we'll play d. So basically these timing blocks
can give us that control.
Yeah, we can create a drum machine basically just by playing different samples and
again use composition local to to leverage switching out the drum machine for a different one.
We can make more convenient DSL for expressing rhythms.
Something like this. Basically we use strings. We represent
breaks by spaces and on the other beats we play one sound or another by basically specifying which
sample corresponds to which symbol in the string.
There's one more concept that I want to talk about in compose. It's called constraint layout.
So basically rather than laying out components in a grid we can anchor components to other components
and sometimes when we make a complex layout that's more convenient. So we specify that a button
should show at the top of the parent and then a text here should show below the button.
And this is achieved by this constraint layout. I was quite pleased to see that actually
that there's a nice analogy there where something like a constraint layout is quite useful.
So if we're hooking up different, different audio building blocks. Let's say we have some kind of
sine wave generator and we have a delay element that takes an input and applies a delay to it.
We have a reverb that takes an input and applies a reverb to it. Then we can also use the same
approach as with a constraint layout to hook these up to each other to basically say hey the
sine wave generator I will call generator and then the delay accepts input from that generator.
The reverb accepts input from the delay. I see that I'm running out of time here.
So I just quickly say yeah I'm pretty much in this multiple platforms is quite feasible because
16-bit pulse code modulation is well supported but I don't want to reinvent the wheels so I'm
also looking to hook this up to some libraries under the hood something like super collider at
least on desktop but this requires careful abstractions and ensuring that the audio back ends
can support the features consistently. I will at this point I think because we've run out of time
leave it here. Thank you so much for listening. The cover will be available there oops I have
stopped showing slides so about that. Yes I will be available to you all there. If you've done
something similar or have any expertise you want to share please reach out I will be super interested
to discuss you can reach me there as my email and yeah if there's time I would happily take some
questions now. Thank you. Thank you Marilyn and I'm also the next speaker so I won't be
able to run and give the microphone away so is there anybody willing to act as my avatar in the
meanwhile just to bring the microphone to whoever is asking any questions for meddling you can do it
oh take care very much. Any questions? Any questions?
That was a copy what we'll have our presentation. Yeah that was a very happy to be there jean
Thank You