Hello, everyone. Welcome. This is a talk about the new hardware for Boston for the video
infrastructure. I am Martin Brahm and I am Ankel, everyone in the team calls me Dexter
and now Martin will start presenting what we did this year to move to a new revision of
the video box. So this is the block diagram of the audio hardware that has been used last
year and full stem and also this year on all rooms except here because this is an experiment
through now. All is test and production. It's the right way. This block diagram of the audio
hardware we have two SGTL 5000 audio codecs and they both have two inputs and two outputs
and they work fine for audio because it was audio last year at full stem but they have very
small dynamic range and that is a bit problematic if there's any setting off it either clips
all the microphones or there's a lot of noise issues. So we decided to make a slight redesign to
switch to some newer audio codecs on the same audio board and these are very nice 32 bit audio
codecs that gives us somewhere dynamic range that we don't really have to do any analog gain
anymore to set the correct audio levels for whatever you plug in and this is also split up to separate
ADC chips and dark chips because we found out that you can get chips that do everything in once
but they do everything badly and if you get chips that do one thing they do it well it's like
it's Linux. We also changed the audio front end. This is a schematic of one of the inputs on the new
audio hardware and this allows us to have basically only passives in the audio path which makes
design a lot easier and we now have fentom powers so we can use any microphone and not just the
wireless receiver from for them and this is a lot simplified compared to the old board
it has less parts but these are all pumps and they are a bit annoying to power.
The only reason the old board had less parts than the new one is that it had a lot less features.
The old one was hard wired to only support the specific gain that we use for the default
them microphones output and so these all pumps their gains are fixed but for the new one the
chips that we use are both cheaper than this front end and also they have integrated programmable
amplification at the input stage so you can have the firmware set the gain programmatically
and so the new front end actually basically does absolutely nothing in terms of gain the only
part that has lots of components here actually just feeds phantom power into the input
if that's needed for the case when we want to use a condenser mic or something like that
like a non-standard set up in order to make this more portable and even to use it for other conferences.
But the audio board is not the only thing we change because if we change around all the audio
if we have to do everything correctly nobody will notice a thing but we also decided to replace
the way video mixing is done which is a bit more complicated and that is this setup this is currently
how it does not force them there is two boxes you might have seen them they are the black and
wood collard boxes in the front and back of the room one is capturing the video from the camera
and one from the laptop of the person giving the presentation and these streams are sent over in
network to the the cable link down here to a rack of laptops which mixes both feeds together
and send it to the internet which is the cloud so that's basically three devices for every
single stream you have one video box here one video box at the back with the camera and one
laptop the mixes the two streams that come over network from the room and then
so this is an experiment I made I now half year ago or one and a half year one half year ago
I decided it would be fun to learn go by doing some complicated graphics programming
this is the first running version of a software video mixer that's should be able to do everything
for that's the three boxes this in one setup and here it is that image is the wallpaper of
my laptop it is captured over HDMI and that is the internal camera of laptop and this is all
being composited in open GL together and output it and this is basically all that is really needed
to make video for for them but this is very slow this is the way I designed it as a novice in
GoPro programming I thought it would be a great idea to just capture the frames in a go routine
sent them an old channel to the rendering threads and then rendered them but that
falls copying a lot of data and copying data in video streams is slow so that's when
dexter help because dexter actually knows how to do go programming yes well at least better at
me and then we spend a lot of time refactoring and optimizing it so do you don't have any copies anymore
yeah so in the current version of the mixing pipeline we basically
use the video for Linux driver and so basically for the video for Linux driver what you have to do is
you hand it a buffer it fills it at some point and then it gives you indication that this
buffer is ready to be consumed while it fills the next buffer and you have like a circular
queue of buffers and then what we did was we have this frame forward the objects that keep track
of each such buffer and what we do is the buffer that comes from video for Linux that is
that has just now been filled gets directly sent to the GPU from the other thread via
sent pixels and we get the content of the buffer with just a single copy operation in the GPU
memory and then with the shader we can do whatever we like in terms of processing namely because
our capture cards output YUIV data we do that in the shader this is probably
quite a stupid idea but it works in practice we just use a shader to do all of the conversions
in the entire video mixing process and then we have two output streams one of them goes out into
window like a regular openGL window that gets displayed here on the projector so this
these frames you're seeing on the stream on the projector right now are being displayed by
this window and then we have other render passes that run the same pipeline for other targets
and namely the other target we use for false them is the stream that gets sent over the network
that's basically composited the stream from the laptop and the stream from the camera and the
nice false them logo and all that and that gets basically fed into the standard input of FFM
peg and FFM peg just encodes and streams that over the network and our most challenging aspect
of this has been to optimize this video mixing system in terms of memory bandwidth used because
these boxes the CPU and them is the Intel and 100 it's it's actually it has a pretty good
Intel graphics GPU but the bottleneck there is the memory bandwidth between the GPU and
the shared memory between the GPU and the CPU because that's only six megabytes per
six gigabits per second and we've had to optimize everything just to lower the memory bandwidth usage
at the cost of more CPU and more GPU usage yeah the video mix is also set up pretty modular
with a several ways to input and output data the most important one the first one I wrote
is the feed you for Linux source that just directly asked kernel for frames that relatively easy
to implement we have an image source that is used to display the little background behind the
two screens that are always visible and we have an FFM peg source because it's very convenient
to be able to input anything at the FFM peg support because FM peg supports a lot of different
types of input and if it's not supported you compile a different one also let's not being used right
here now but that's open media transport support that's a relatively new network protocol that
can stream high quality basically relatively in-groups video with low latency of the network
and I added an HTML renderer because it's nice to display graphics and well kind of less
minute I also added a PDF renderer it's not in here yet but that's currently being used to display
this PDF and because this presentation is running also in for some ticks on my laptop
being sent to the box running for some ticks that makes it for the projectors so we are
nesting for some ticks mixers here and for the outputs we have the OpenGL window but that's
what you're currently seeing and FFM peg again for text quality told for streaming and we have
another outputs to also send out open media transport streams that's also currently not being used
so this is the the HTML source that is rendering the current schedule of this room and it has some
custom CSS injected to scale things a bit and remove the background as a demo it's not very
readable at the top it that's what demos are for this is an FFM peg source it has a simple loop
the start-for-ing in the loop is something an FFM peg not even some ticks and I have not been
looking into how to do seamless loops in FFM peg but it's working and if it works yes also the
camera source works if he rotated yes perfect so yeah this is basically what some ticks does
so we've had so far we've managed to develop this and it sort of works okay in production
like for example this box here that streams out of this room but there's still a few challenges
that need to be addressed in order to make it a production ready video mixer for example
we currently support multiple sinks like multiple video streams being outputed but in practice
there has there always has to be exactly one window based output because that's where we run
the render loop it basically runs the same way a game engine would run there's like this window that
you render to and then we piggyback on that render loop to actually do the other render steps for
the other outputs that are based on FFM peg but currently we can't open a second window
and we can't just stream to FFM peg by having zero window outputs because we need a window
output for something to run the shader and that's something that we would have to research how to
do because we're not really graphics people we're more like just hardware people who happen to
be looking into how to do some stuff with graphics the other interesting issue that we currently
have with the implementation is the fact that the YUIV conversion and basically any pixel format
conversion is done directly in the shader in a way that just samples the pixels from the other
texture directly so interpolation is basically broken because you cannot do that because the source
pixels would not be correct if you interpolate them and then assume they are correct why
you IV data so this is another thing that we have to look into so we're basically also
soliciting contributions the code is under false them video phasantics repo and get help
and you're all welcome to look at it and messages and ellitas for extremely stupid things that we
might have done because we basically don't know much about graphics programming in general we just thought
that how hard could it be to write a video mixer and we just did some stuff and it happens to work
and it's actually fast but there's still some rough edges. Martin can tell you more about
shader pipeline while I try to do a quick demo of the scene transitions.
Yeah the currently the feeder shader is for all the most of these sources are why you
Wi-Fi data and I'm currently loading that into an RGB A texture because apparently there are
no Wi-Fi formats and open gel and then in the shader I do a rough interpolation of the
Wi data to make it look not crap and this only works as long as I have nearest neighbor
interpolation enabled because when I don't have a scale one of the sources down but like the
webcam currently is and I don't longer have one to one mapping between the pixels resource
and the destination due to interpolation the edges will start to appear two pixels away
this looks like some really weird ghosting and I'm pretty sure the solution is to
do the shader in multiple steps but the first decodes the Wi-Fi data to RGB and then in the next shader
actually do the video mixing but I have no clue what the performance impact of that would be
but I guess most of the people here are graphics people so maybe one of you will know
hopefully and another big performance issue here is when adding more sources we are rendering twice
by calling the shader again and will initially using the gel get pixels for something to get
the frame buffer for the second output which is very slow and I've migrated that over to
one of the aging APIs and it seems faster but I still have to feeling I'm doing it wrong
so patches welcome
so the other thing we currently have implemented is seen transitions so now if I
just unplug the HDMI from the laptop they should transition to another scene that shows the place
holder image and now hopefully if I plug it into the other laptop it should smoothly transition to
the output of that laptop fancy stuff and this is the web user interface of the video mixer so here
you can see the two output stages the one for the projector and the one for the stream and for
example if I switch the projector to camera full screen this will happen and you will you will be
able to see me on the projector so this works on the false them video box and yeah basically
that's the this is the scene that gets sent out to the stream online and the other thing I want to
show you before we wrap up let me just do this is to just open the code
and show you the actual video mixing because it's a really short piece of jail a cell code
the only thing we have is these functions that sample the input textures for example this is
the function that samples the YUI V texture that gets sent from the video for Linux buffer and
the whole like these are all separate functions to sample the different inputs input formats that we
might have and so basically we have this step here that's a like switch function to run depending
on the format of the current input being sampled and the whole mixing procedure is just this we sample
each input layer this code gets generated so that this block here is present for every single input
that we have so we just sample the input pixels and then we blend them together that's it that's
the whole video mixing and basically the layer data that we have here just tells us how to transform
this layer so it's basically the transformation matrix that just tells us that this texture gets rendered
into this rectangle and this one gets rendered into this rectangle and as you can possibly see the
text there on the slides is quite jagged and not really well anti-aliased that is because of this
function here that samples the YUI V texture and it samples it by getting pixels with nearest neighbor
interpolation it just takes a pixel and that's why it looks like this so if this gets improved
if hopefully one of you sends us a patch or an idea of how to improve this we will be able to
actually run this in production with feature parity with the current set up
oh no questions times up