Okay, hello everyone, my name is Hamiropola and today I'm going to talk to about FF Glitch.
First, just a little intro about me, I joined the FF MPEG project in around 2006 and I was
an active contributor and I'm a maintainer for about five years, then I left the project
in 2011 but I didn't stop doing video related stuff and I didn't stop working with open
source and in the past few years I've been working in this project called FF Glitch and
today I'm going to present it to you.
So FF Glitch is yet another FF MPEG fork created mostly to make Glitch art and as most
of you know the hardest part in software engineering is company of both names so I just
took FF MPEG and Glitch art and pushed them together and so it's called FF Glitch.
Yet dear for making a FF MPEG for making Glitch art, you'll need a FF MPEG goes about 20 years
back.
Sometimes when working with FF MPEG I'd get glitches in the video output and I would
spend hours playing with those glitches but those glitches were considered bugs and we had
to fix the bugs and get rid of the glitches which I have always found very unfortunate.
So FF Glitch was born as a project, a project where those glitches are considered features
and not bugs.
So what do we mean by glitch art in the context of FF Glitch?
Some examples like SD card corruption or scratch DVDs, DVD reception issues, streaming connectivity
issues and mostly data corruption in somewhere or another that create some interesting
visual artifacts.
For example, this is a classic JPEG glitch from early digital cameras, it's one of the
oldest digital glitches that I could find in my photo library.
This is a picture I took with Sony Cybershot all the way back in 2008 or rather these are
two pictures that I took all the way back in 2008 with Sony Cybershot and thanks to
five system corruption the two pictures got merged together.
The decoder still managed to give us something, even though there are errors when traditionally
to one from the different pictures but it generated something, it's an interesting artifact.
Now I'm going to show you another example of a glitch art that was actually sent to me
by Marvin Schult, often known as A-Pyret on FFMPG and VLC.
This is a JPEG 2000 GPU based decoder glitch.
It no longer reflects photographic reality like the previous example of the merge photos
and it's something more abstract and more beautiful in my point of view and as a video nerd
I'm always thinking to myself what is the decoder doing to generate that and I'm going
to move a little bit away from image codex and I'm going to show you a glitch from video
codec.
For some going to show you the sample video that I use as input, it's a public domain
video of cars in a highway, pretty straightforward.
So the glitch is I found an experimental patch in the FFMPG development list from 2007 to
add WMV3 and codey support.
I found it in 2017 and tried rebasing it to the current code based FFMPG and it kind of failed.
So in the next slide I'm going to show you the output of the video encoded, the WMV3
encoder and let's see if we can spot the glitch, it's very, very subtle, there you go.
That's the results from a broken experimental WMV3 encoder.
The encoder is clearly generating a broken district but the decoder still manage to display
something.
Now that you have an idea of what I mean by glitch art in the context of FFG glitch, let's
move on to the project itself.
So early history of FFG, in 2015 I started following this Facebook group called the
glitch artist collective which still exists and is also an Instagram and it's a group
where glitch artist would share all kinds of things that have been working on and they
would ask all kinds of questions.
Now many of those questions were FFMPG related and I was happy to help whenever possible
but there was one particular question that got my attention.
This question was asked by Kasper who happens to be in the audience here today.
It went something a little bit like this.
Is it possible to edit the motion vectors in a video file?
And that got me thinking it should be possible.
If you're ethically you could just modify the bitstream of the video file, manipulate the
motion vectors and still don't know if we've a valid file that could be played in any video
place, even though it's not like the original file anymore.
I just didn't think there are any tools around that could add at the time so I set
out to create a bitstream editor.
Well, I didn't read it from scratch, I wasn't really really keen on writing an entirely
new bitstream parser and writer.
FFMPG already had the code to do all of that.
I could just reuse FFMPG and bend it a little bit for my purposes.
It took me about two weeks to get the proof of concept working and it went a little
bit something like this.
In the first pass, I would decode the file normally.
I would then export to decode motion vectors to a JSON file.
Then I could modify the motion vectors in any script or any other tool that I wanted in
the JSON file and in a second pass, I would decode the file again but this time while
decoding I would replicate the bitstream and while I was replicating the bitstream, I would
modify just the motion vector values.
The file isn't reencoded, it's the decoder itself that's used to translate the bitstream.
To achieve that, I kind of had to hack FFMPG, I had to hack the get bit context structure.
What is the get bit context structure?
It's a structure that's used to read values from a file or a stream and the video stream
is basically just a sequence of bytes which is just a sequence of bits and this get
big structure will read those bits.
It has some functions such as get one bit, get n number of bits or get the variable and
codes, get variable number of bits and to illustrate the get bit context working, I'm going
to decode a chunk of an MPG for a p-frame with motion vectors.
This is a p-frame with motion vectors overlaid.
It's the example on the FFMPG track.
Each error corresponds to a macro block and I'm going to decode the bitstream of one of
those macro blocks just the beginning.
We only need three bytes because that's where the motion vectors are going to be in the
beginning of the macro block.
Now let's see those bytes as bits, there's a representation.
The first few calls to get bit context, they are not related to motion vectors, they are
other flags inside or variable encodes inside the macro block.
Then after that we decode the motion vector delta x with three calls to get bit context,
then we decode motion vector delta y with another three calls to get bit context.
Now how did I modify the get bit context to replicate the data for the proof of concept
that I was writing?
Basically, I shoehorn a put bit context inside the get bit context, so what is a put bit
context?
Similarly to get bit, it's a structure that you use to write values to a file string.
It's usually used in the encoders, you allocate a buffer and then you just keep writing
bits to it.
Then has a function called put bits, really straightforward, so every time my get bit context
would read a bit, I would replicate that into a separate buffer with a put bit context.
Now let's go back, let's go again step by step, reading the bit string while performing
the replication into a put bit context.
So we start the same way again, read three values, now if you look at the bottom, the output,
bits, as the bits are being read in the input, they're being replicated in the output.
Then we read the first three fields, then we read the motion vector delta x, then motion vector
delta y, and then after that there's that tray one over there, that's just the continuation
of the macro block I didn't want to put anything after that.
Basically I would just have a buffer that would completely replicate the file in the output.
Now what would happen if instead of writing that value, I wanted to clear the motion vector
delta x, I know from the impact for encoder and ffm, that I would need to write a put
bits one instead of that chain that was there.
And I'm going to do the same process again, but do the transplications, instead of copying
all of that, I'm going to pause the replication and write the other bit.
So it starts the same way, I'm replicating the first few bits, then I pause, I read the
values of the get it context, but I don't write them after the output, I pause the replication,
then I write the bit that I wanted, which is basically a zero, and then I restart the replication,
and I keep going.
So now I have a modified bitstream that has the x value for the motion vector of the
delta is zero.
And then replication has been restarted, I just keep on replicating everything else.
The good thing with this approach is that the glitch file has a valid bitstream, so it can
be decoded correctly by ffmpag, we'll see you too, or anything.
And I don't need to edit the entire bitstream, I can just replicate most of it, and in
a very specific part of the code, I edit a couple of values and then keep replicating
afterwards.
This way, I can add new features to ffmpag glitch incrementally, so if I want to change the
DCT later, I can just hack some code inside the DCT writer and on and on and on.
Now let's see some real usage.
So now this is skipping forward, like the proof of concept was in 2016.
This is after in 2020, I added JavaScript support using QuickJS to ffglitch.
So this is actually what ffglitch would do between the first pass and the second pass.
So there's the first pass, then you get frame with the motion vectors, you can edit
them the way you want, and then the second pass goes on, and then you get the glitch
output.
What this code is doing, first check that the frame has motion vectors, because if we're
in a knife frame that it has no motion vectors, then it's going to clear the horizontal
element of all motion vectors, just loops through the 2DRA, and then that line this set
the horizontal motion vectors is 0.
And let's see the result.
That's the same file from before, but with the horizontal movement removed.
I'm going to show an example from a different script.
Let's see if we can guess what this script is doing in this case.
It's actually very simple, I'm just adding two vertically to all motion vectors.
One of the color distortions that are seen, that's entirely on the decoder.
It's a valid impact to a file, I just tweak the bitstream a little bit.
So each codec may give different possibilities for a glitchart.
For example, PNG is an entirely different codec, where it's entirely different ways,
and it's a whole new way of doing glitchart for each codec.
This is the just a simple PNG image of flower, picture directed long time ago.
Then PNG, the way it works, it has a byte that specifies the filter type for each row,
and then it has a byte for the row.
If I just modify the first byte using effect glitch of each row, I'm going to change
effectively, change the filter type, and with just a simple script, I can get this output
out of it.
Here's one good thing about a ff glitch, that it may seem like it's just a great glitchart,
but since I have a good understanding of the codec, I have to read and I have to understand
a lot of code, and that's a very very important point for me.
What I really love is understanding how codec works.
As a side project, as a side project with ff glitch, which in itself is already a side
project, I started trying to describe codecs in the most simple and visual way that I could think of.
So this is one simple description of a JPEG bitstream.
The website itself is interactive, so you can see how each macro block gets encoded, passing
through all of the DCT magic and how you don't need to understand what DCT does, it's just a box.
And then you get the bitstream again, then you can see which parts use more bits to represent
which parts use fewer bits.
Unfortunately, this is a working progress, it's been installed for like four years, I would
really like to finish it, but I don't know, I just haven't gotten around to it.
And I'd love to do this for PNG and other image codecs and maybe even some video codecs.
It shows that different codecs work in wildly different ways.
So back to ff glitch, which codecs is its support currently?
There's support for JPEG, MPG 2, MPG 4 in PNG, which I added last year.
The features that each codec has, so some have DCT coefficients, some have motion vectors.
In JPEG, you can change your quantization table, which is actually pretty interesting
because as we're missing with the webcams that they send data encoded in JPEG, you can
actually intercept the data just use the ff glitch to change the quantization table in the
beginning and create a virtual webcam, and then you can just glitch stuff in real time.
And ff glitch can modify not only the raw bits string, but it can also support modifying
the container format, because since we are changing the size of the frames, this has
to be reflected in the container as well, so that it knows how to find the frames correctly
in the file.
So ff glitch is going to fix up the indices, the sizes, checksons, and all of that,
so that the files are still valid.
Currently, AVI and Mover supported, but then I think it's already enough, I mean, we don't
really need that much more than that just to make the chart, although it could be fun to
try other formats.
And ff glitch has a bunch of extra functionality built in, I had already mentioned scripting
supports, throughput jfs, and then later added Python 3.
There's HAD support of Sdl2, so that I can use a joystick or the keyboard or mouse.
There's MIDI support of RT MIDI for controllers, such as this one, I'm going to, if there's
time I'm going to demo it later, and network support using zeroing Q. Now, why did I
add network support to this? I was getting lots of requests for different types of protocol
in different types of devices, and it seemed easier to just implement zeroing Q support,
so that people can write their own bridges.
They can write their own code, a simple script that reads from their device, and then just
talk back using zeroing Q.
The idea is to give the two necessary for a glitch, I just to create more complex stuff.
So ff glitch is basically three programs, or three binaries.
There's ff edit, which is the new tool, which is going to edit or export, apply the changes
to the bit string, it's the main tool for ff glitch, it's the MIDI MIDI stream editor,
then there's ff live, which is actually just the ff play binary from ffmpeng.
It has ff edit integrated into it, so we can glitch in real time, and then there's ffgc
where gc stands for glitch artists collective, because in my tutorials I would tell people
to use ffmpeng, and then they would say, oh, it's not working, because this feature doesn't
exist, so that's a feature I added to ff glitch, so it's not in the real ffmpeng, so
I just had to rename the binary.
It has some extra functionality to make more broken videos, which wouldn't really be interesting
in ffmpeng itself.
ff live got to a point where it works in real time even on a Raspberry Pi, and I
have spent a considerable amount of time optimizing it this past year, so that it runs
well on arm, which means it's on other point, which is upstream into ffmpeng, because ff
glitch is a full blown ffmpeng, so the way upstream anything, most of what I added is
the layout side of the school process ffmpeng, so we won't want broken stuff in ffmpeng,
but there's a lot of work that's common to both projects, and I've been doing, for example,
I added some support for ffv4l2, there was new devices that weren't supported by ffmpeng,
some bug fixes, just while I was writing the ffmpeng glitching, I found a couple of bugs
in her png encoder, tests, tests are always welcome, and optimizations, like I said,
ff live needs to be really fast, so it's optimizing it's funny, you know, it just
went 10 hours in a row, writing assembly, it's a lot of fun, and it's rewarding.
So I mentioned the rest of the ffmpeng, but the device I've really been working on is another one,
is the orange ffmpeng, it's very, very powerful, and it has HDMI input and HDMI output,
so what's good about that is that it works kind of like a guitar pedal, but for glitch art,
now I can bring it with me with just a power bank, and then I plug HDMI in and HDMI out,
unfortunately it didn't work for the presentation, because I wanted to have it here in the loop,
but not today, and what's next for ffmpeng glitch, I'd like to add to the part for more modern
codecs, such as hrc4, and if you want, I know this codecs will have many more features that will
be fun to play with, but I'd also like to play around with older formats such as GIF and JPEG 2000,
and especially interested in wavelet-based codecs. I'd like to add support for audio codecs,
because the same principle holds with audio, you just modify the bitstream, should work the same way.
Now, there's some people using it for vj, or for life performances, using ffmpeng glitch,
and they've been asking me to support these frameworks, siphon and spout, I don't know if you've heard about them,
I had never heard about them until a couple of months ago, but apparently every vj knows it,
basically you can share it texture directly with other tools such as the Resolumenters Designer,
and I think enough people have mentioned that first ffmpeng and them using it in ffg glitch.
I've heard that it could be used for a stegonography to hide information inside a bitstream,
but I haven't ever tried that yet. I've also heard that it could be used to test codecs,
security in hardware decoders, kind of like how h26 farge is used, we could create a very,
very specific bitstream that's going to break your decoder, most importantly I'd like to hear
your contributions and your feedback, and your pull request as well. And since there's still time,
I have a little gallery of some more examples using ffg glitch. So what I usually do is that I write
small proofs of concept, small scripts like this one, the video from my cat in 2018, just to show
how things work, and then I'll just pick up those scripts as examples and they either improve
them or use them as is and try them with different. Yeah, it's very, very fun and try them with different
videos. This is one of the first examples I saw in the wild, it's also from Casper and Tomacola,
which is not here today. It was actually surprising because they had written this blog post with that
and I hadn't heard about it for like Ambusta Yub, because there was a lot of social networks,
and then someone's like, hey, how come you haven't seen that? They're using your software.
Like, oh, I didn't know anybody was using it, but that was fun. There's a couple more from Tomacola.
He did a lot of early work in ffmpeg of testing of different things. And then there was an
okay, you can actually see the green bleeding from the top. That's the Yub green. Actually,
the motion vectors try to fetch information from way too far above where they were not supposed
to try to fetch anything. So then the decoder, that's actually a glitch in the decoder,
the different decoder if they're going to give different results for that. This one's from
Sebastian Béria. It's a scene from the series Dark and not how it almost let it go.
It almost looks like a painting in the end, so that's just very beautiful. I just love watching
stuff like that. That's why I would spend hours working on a glitch ffmpeg.
Now I'm going to do some demos with ffmpeg. Hopefully, it's going to work.
Let's see if this one works. Okay. This should work with the MIDI controller.
So the MIDI controller here, and I can just modify in a little bit.
And if I keep moving, you can see that I'm still here, but I'm just modifying the motion
vectors in real time. And then there's also this feature, which basically resets the, and this is
reseting in the decoder itself. So there's a kind of stuff that's changed in ffmpeg itself,
but not kind of nobody's going to want something that's going to inject a macro blocks inside
a p frame in normal ffmpeg. So this is the first example. There's another one that hopefully
is going to work if internet connection here is good. It's, ah, no. Okay, so I have the live stream
of the presentation. And I can glitch that as well. I don't know what the delay is, but yeah,
so since you can just pipe stuff, you know, I can just, I could actually pipe this into
streaming it to YouTube or something like that. It's quite interesting. There's this website called
a webcam taxi that you can find webcam from all over the world. And I would have this one from
this, these, these people playing pool in the Philippines, then I would just glitch that and then
return submitted somewhere else. And the longer you leave it, the more the color saturate and
have these kind of effects. So yeah, let's get to the presentation. And oh yeah, that's it.
Thank you very much. Those are some, ah, of me. Thank you.
If anybody's interested in talking about glitch art, let's go out for a beer later tonight.
I would really love to have more hardcore developers doing glitch art stuff because right now it's
only like a couple of us and it's a lot of fun to work with. Yeah, I think we'll have time for questions.
Yeah. Okay. Anybody questions? No questions. Yes? How do you do the setting of the
visual loss that you have to kind of be in your house? Yeah. Yeah. So it's, it's in the in the
encoder. So there's, there's some code that I had to change in the encoder. So I send a
MIDI command and then it just randomly sets i macro blocks inside a p frame and then at the end it
clears everything with an i frame. Oh, yes. Okay. I thought you were just filtering about the
encoding. Yeah. So the question was if, wait, what was the question again?
Well, if you're just filtering the bits to the encoding, how did you reset macro blocks to there?
Okay. So the question, if I'm just filtering the bits to how did I reset the macro blocks
to there's initial state with that command right now? Actually, there was, there was a lot more code
going on in ffclitch. So there is lots of code in ffmpeg as well in the encoder. I hacked the encoder as
well so that I can selectively add i frame or just add i macro blocks inside an i frame. So I mean,
I could spend a few more hours talking about everything that I mean, I even added pixel sorting
so ffclitch because it was fun. No, just just to see if it could be, if it could be done live.
So yeah, there's a lot more functionality that's documented in the website now.
For many, many years, there was no documentation of ffclitch, but now there is. So you could go to
the website and check it out. Thank you. Other questions? Yes?
I have, so the question is what it would be like to do glitch art with the audio codex?
I haven't tried it yet, but I assume it would be like noise music, which I also love,
I know some some people don't, but I really enjoyed it. So this is something that I really
want to get into. We're going to be fun to hack with MP3 files.
Yes?
For the way for this, we want to like check the x size.
Yeah, I haven't, I haven't tried it. I think the first one I'm going to go with is a jpeg 2000 because
that's not we already have. The question was a jpeg xx excess or something like that,
which is I haven't really looked at yet.
Okay, no more questions? Thank you very much. Bye.
Oof. Good.