Okay, you almost made it.
One of the last talks, as is tradition, we are trying to splice these a few jokes in
also as is tradition.
Some things are static and that's awesome.
Without much further ado, let's get started.
So I have a clicker.
That modern technology.
Yeah, so the core infrastructure is static as compared to last year, which is very good,
because it means that's workforce.
I mean, some details like new SSDs and services such, but by and large, everything is
continuous to be static.
We are still considering to rip out the core out of our next year.
We'll see how it goes.
But the nice thing about this is we don't have to spend endless nights on this, and
at least not as of right now.
We have found a place of stability.
Maybe that's the first learning for anyone wanting to replicate a conference,
work towards having stuff you can replicate and take out of a box next year.
Again, don't have to restart from scratch every single time.
It took us honestly too long to realize this kind of dynamic, but once we had it,
we started saving massive amounts of time.
Yeah, and there was a benefit.
You get a lot of sleep back if you implement that, because this is something that was
lacking from the last years, having actually time to recover and reach out to your personal
batteries, because of dealing too much with infrastructure.
That is needed to run such a conference at that scale.
For this out of itself, it's basically just one big router.
We have a handful of switches that are just distributing some
major points, like you've seen the t-shirt sale before for the point of
friendships, and we have some switches for that.
And our core network that we will see later, we have three
bigger switches from Arista that we just bought last year.
They're only 12 years old, and now 13 years old, but they still do.
Very good.
We have some very, very old crappy service sitting in the data center at the
ULB, but they will be thrown out hopefully anytime soon.
But we also have now a cluster that makes this independent of other cloud
service providers or cloud providers to run our workloads that are,
yeah, basically most before the video processing that is done on
side, and yeah, really happy to have made that switch last year.
So we are very, very free to move this year and not rely on any third
parties.
We've changed our monitoring.
It's now done.
We are premises and low key and visualized by the finance.
So we can easily share information with other parts of the team that is
changing over the years, and yeah, we have also public dashboards.
So you can see actually the status, if you go to dashboard that
falls in the dark, you can see most of the stuff that we are seeing.
Like, is the network saturated or not?
Yeah, if you like, just go on it.
It's in another nice version of all the data.
For video, this is main part of what we are doing when we are building this
conference.
I'm not sure if you've seen it or recognized that we have in every room
there's a camera.
I'm just pointing at it.
We have such lovely video boxes that are standing here.
And they're mostly open source.
The camera obviously is not, but the boxes.
And we have everything that we build.
You need to build for such a video setup.
It's on GitHub.
We have those boxes here that are sending all of the video
streams to a rendering form that we have at the campus.
We send these streams off-site for live streaming and distributed
ourselves.
We're not using anything like YouTube or any other big companies
out there.
And we also have our semi-automatic review and cutting process
that we implement.
So the speakers can do the cutting in the browser after they have
their talk.
And that's how we try to publish all of the plus
thousand talks that are given at Fossdom at this year.
I'm sure if you've seen the videos the years before,
this is all done by the video infrastructure that we have here.
And hopefully we will serve the purpose that we
have to do.
So I've got a bit into the video boxes.
The years they have changed.
Not sure if it.
Yeah.
We have some proprietary stuff going on here.
We removed all of that in the last years because this
was closed source.
And we completely read it all of the video boxes.
We started one and a half years ago to design the new
box with all open components.
And we attached the laptop to that.
And then we had problems with the USB-C and firmware issues
in USB-C.
So we made a final video box with yet another component that was not fully
open source, but helped us getting our path to that.
And this was the final video box that we had for last year.
It's basically its dump switch that we have here to connect everything
together.
And this is a USB hub.
And this is basically something that will emit impact data that is
grabbed from the HDMI sources.
And this is the final video box, the final design.
When you do your Thesis, there's always a final of something.
V2.x or something like that.
So for this year we have the next edition, which you can also see here.
To tear down the ring, you can have a look at that.
This is now the final design for this year.
Probably it will get some enhancements in the future.
My colleagues that are built this boxes are just giving a talk about how
they built this boxes over at the age building.
So if you run there, you will be too late because it talks already
only 15 minutes.
But basically everything inside there has been designed
constantly for us.
We've added some diagnosis on the front.
And I'm really like to see that because when I'm speaking,
I can see all of the VU meters going up and down.
And we also made it easy to adapt to all of the usual audio technology
that you have at a conference.
We use XLR a lot.
So that's also embedded.
Something that we haven't had for years.
Because we always needed to go through the XLR to the camera.
That's back there.
Which is quite shitty for all the folks that are running the video.
Because they have to go a lot.
I've had a lot in crowded rooms.
This room is in the crowded.
But if you have been to the one of the lower age rooms yesterday,
for example, it's been as a volunteer that needs to take care
that the batteries are charged.
And so it's a nightmare.
And so no everything at one place.
And don't need to do with that.
And not sure if you can see it if you zoom in.
But these are all custom-made stuff.
This is a switch that we designed.
And you can see that logo here on the printed silk screen.
It's a phosdom logo with a switch inside.
And we have the same logo here, which is our USB component.
That will do all the handling of the outside USB connections.
Mainly it's used for charging all of those belt devices.
And so you can charge it directly from the box now.
And we have the little LCD screen.
We have an audio mixer that is fully controllable via software.
So every audio that gets in can now be managed from a central place.
We have a red X4.
I hope it pronounced it right.
X86 CPU running here with Debian.
That will manage everything.
And we have the HDMI video grabber that we will hopefully replace
in the version that we can also put in.
Not only the laptop here, but also the camera.
So have it all in one box.
And don't need two of those boxes per room anymore.
Yes.
I'm not sure.
I have a link to the talk that my colleagues are giving.
And they're explaining everything they've done.
Everything that's went inside there.
We have the key cut.
And then specific that you need and everything that you.
If you want to build it yourself, you can have it.
I don't have all the schematics in my head.
I'm just.
But they should all be available.
If they're not file an issue, because open source and open hardware.
Yeah.
So the final video box.
If you want to have more details like this guy in the front sitting in the front.
There's a talk from marching in angle that actually designed all of that.
The PCB design.
And then getting it done at the manufacturer and running into the sweatshops.
That we're soldering that over the weekends in December.
They give a talk that's happening right now.
There's the QR code to the talk.
It will be released hopefully tomorrow when it's been cut.
You can then see what actually went into that system.
Running such a conference.
You can think of, oh yeah.
It's not that hard.
You have a box here.
You have a laptop.
You have mains.
And then you have a camera.
It can't be that hard.
Actually, it is that hard because you need to think of things like we all do in this in our free time.
We're volunteers.
We don't get paid for that.
We don't have any paid stuff like at other conferences.
So it needs to be bulletproof from a design perspective.
That's why we have schematics for everything.
Again, this is also on the GitHub.
So you can clone it or notice.
And this is what we hand out to our volunteers when they start on Friday in the morning.
Building up all the rooms setting up the cameras.
So they know what is connected where and what to set.
So everything can be reproduced and set to a known state with a known working state as we have defined it.
What's going on there?
Is it more interesting?
If you want to know this is me trying to meet with a member of the European Commission, or one of the offices of the European Commission,
to get them to send a letter to you to thank them for fostering some sort of political support.
Yes, it's important.
So how's the whole conference managed?
We have one big video operating center.
It's not like minority reports.
It's crappy HTML.
But it works.
You can see it.
We have a preview of what's going on in every room.
We have the audio levels that you can.
Where do you see it?
We have the audio levels.
If they're working and we can all see this from the remote and then can maybe change it if it's not working as expected.
So we have everything in the K building.
We're sitting there.
Volunteers that are actually listening into the streams.
If the video signal is okay.
If the audio signal is okay.
It's somewhat synchronized with my lips.
What I'm saying.
So they can see everything from a central place and everyone here in the room.
Not in the light and talk room, but in every deaf room.
We have deaf room managers that get their room and they can control for example, which is shown in the video stream.
And this is all handled by the deaf room managers.
How does it look like?
This is from yesterday.
When you have such a room.
They can actually see life.
The view meters of the audio levels on this side.
They could also temper them and just change things.
They can set the output for the video stream and also for the recording.
Should it be side by side presenter or should it be the slides that were shown.
And all of those things and it's very, very basic HTML.
So it work work on any device.
You can also have it on a smartphone.
I saw a lot of deaf room managers just using their smartphone to control the video stream, which is quite convenient for them.
And this is actually how the conference is running and how the video is managed.
All of these videos that are produced here during the conference are sent to a rendering farm, which is looking quite messy.
If you look over here.
This is our wreck.
We have some service in there.
We have a lot of cables and we've had a lot of laptops.
Since we started adding more and more development rooms that are running at the same time,
we needed to upgrade the laptop.
That's why we have chosen to change the position.
It's also better from the cooling perspective.
And we have now placed for a switch that has bigger uplinks.
That also is one of the core networks, which is where all of the rendering is going on.
This year we have 32 laptops that are doing all of that.
And yeah, basically it's quite fine.
We still use the same Ikea racks to stack everything in cooling is better.
Yeah, that's it basically.
And from what I said before, like getting back sleep or not being too sleepy when arriving here and dealing with the conference.
Like Richard said, when you have want to reach a level of stability,
installation of the router is one of the core things that we need to do to maintain the access here.
And also get the routes up and working.
This has been always tricky because we build up everything here on Friday.
We are allowed to open up the doors of the ULB on Friday in the morning, not earlier.
And since most of the equipment is sitting in the rack anyways.
And we try to pull out that pull forward this task to just come here and have hopefully everything working.
And as you can see by the timeline, it has come better.
It's a selection of the router and also of the routes that need to be propagated for that conference.
And still, it has improved a lot.
Now this year, I think it was the earliest possible date to do the on January 6 because we just get a slash 17 announced from right and see for a month that they're handing out for events.
Oh, a sign, not handed out.
But we can announce, we could announce it on January 6 and this was the earliest thing possible because we will just tear it down tomorrow.
And yeah, this was actually great because we had a lot of time to work on everything and see if it's still working like last year.
And also from the network that was up and running and we've had it the first time that we had it on Wednesday already up and running.
This is due to me being dump and ordering the wrong SSDs and we need to swap out the SSDs.
The consumer grade SSDs to enterprise SSDs in the servers and we needed to do that before the first event.
So it was on Wednesday so we could actually bring up the SSDs here and see if everything's working.
Later pickups on the network were fixed due to some persons unplugging cables in wild situation other situations but it was quite okay and no one needed to stay up for a long and everyone got enough sleep.
The monitoring of everything which helped us to stay sane to see if everything's in the desired state.
We have basically not changed it since 2023 and just kept it running.
We just adapt to maybe a new host name or a new damn distance spun up and this really helped us controlling if everything's in the desired state.
And the video this year was a bit late, 2023.
This is when all of the video cameras have been tested.
We know that every data point that we want to see in our monitoring systems is there.
It should have been done by five o'clock this year that was the plan.
They had a bit of issues with one room because we always tend to break the switches.
No has it break us in the power distributions because we overload them and that cost us roughly two hours.
But we still we still okay with the time of 2023 because no one needed to stay here for long and the securities would like us to have here.
Use more bandwidth and this is where it comes to our sponsors.
So maybe to to somewhere is one of the reasons why we focused on the video box so much right now.
It's because this was the only major component which we actually changed out.
The rest again try and make something which you can reuse and just keep running ideally.
To lower your workload if you try to do something like this yourself.
Also be aware that lots of things where we just have a line have literally a dozen people or several dozen people working on things continuously like you just get the high level overview here.
So I mean yes use more bandwidth.
We deliberately don't provide any wired access anymore because you don't really need it because everyone has Wi-Fi.
But still if you want to do some weird experiment if you want to run this one server out of your pocket during the thing and make a nice tip out which it replies.
Actually here I forgot the one for 2012.
Yeah just just to it like use more stuff at the same time.
Yeah those are the two sponsors who we like to to think because without them we wouldn't have any internet here.
On the topic of sponsors I don't know if you have this right now or not.
Okay one way. On the topic of sponsors this is also something which I'm going to say in the closing keynote in a few minutes.
We are currently at a fragile balance where it comes to our finances.
So if any of you work at companies might potentially want to sponsor us for them in the future.
Your internal marketing department will roughly close the planning for next year towards middle of this year.
That's when they roughly stop their financial planning.
So if any of you find yourself into a position where you can talk to your boss, you're to your boss's boss to whatever that might also be good because we have a pulsating red zero roughly for this year.
And it's not a super comfortable place to be in.
Again, I'm also going to repeat this during the closing keynote because it's a little bit of sounding of the click rules.
But we will still do our next foster next year I hope.
Yeah I think we are.
If you want to clone first them we highly encourage you to.
For those who don't know for Asia some years ago started working with us that we helped them.
We are also going to try to help.
I forgot the complete name of the Indian but it's like the little largest Indian conference.
We are also trying to to support them to help and kickstart more of a of a repeatable process.
Kind of thing Bulgaria is also using for open fest like we want to to and aid others to run their own events.
Their meetups, their local conferences, their whatever.
And not have to reinvent everything from scratch.
We actively invite you to take our stuff and just run with it without you having to do all the heavy lifting which we did over the years.
And now if anyone has questions I'm going to see this microphone.
And we're going to pay help or take it with this one.
All right I have seen multiple infrastructure reviews from the past years and there was never no mention of how the streaming to the world is actually handled because I suppose you guys are not sending the streams from here from the ULB to the clients that are requesting them that you have some sort of CDN or something.
How does that work?
Yeah excellent question thanks.
Actually we rent out some VMs that cloud providers like Hatsner that are acting as the endpoints.
When you see stream.phosdom.org, pointing you to some of those instances.
Basically it's just an engine axe that's relaying the stream that's coming from from the rendering farm.
So if you're requesting a file, usually it will go through one of those VMs at Hatsner.
And that will hit the back end on our rendering farm and since the streaming mechanisms allow some sort of caching at the engine axe is on the thing.
So that's how we actually streaming through the world.
There's another question.
Can you determine from your data how many people have been here?
Sorry can you repeat and put the menu in front of your mouth?
Can you determine from the data you've got how many people have been here roughly?
Not really ever since privacy extensions have started to come and effect.
We stopped having any good numbers.
10k plus is an easy, very, very comfortable guess but we don't have any exact numbers.
It's a very deliberate choice to not require tickets, to not require registration, to not require payment or anything, to not require anything.
And one of the direct effects which we take into account is basically our inability to count total amount of people.
But we're basically filling as you probably noticed your B2 capacity.
What we can tell you just from looking on the hallways.
Last year we were pretty much at pre-pandemic levels but this year is the largest Boston we ever had.
In previous years you sold the laptops that are used in the render farm and blind up this year.
You are a person, probably 10,400.
I know I get the question.
It's a fair question to ask.
For those who don't know the laptops you see in the rendering farm photos, we used to sell it.
We used to bulk by them from refurbishment companies and then reselled them during forced them and then just hand them out for vouchers like you bought a voucher.
And during carried on you literally just got a laptop from the office or the rendering farm or whatever and you just took it and was yours.
We have issues with how the laptops are being charged and how this integrates with both the rendering farm distribution and with the video boxes.
So for now we have decided that a the laptops are powerful enough and b to not have complications with how we run this thing at scale.
It's better to just keep them. But you're far from the only person to ask and it was always super nice to be able to just cycle good laptops into hands of people cheaply.
But for the foreseeable future we will keep them and store them.
We would much prefer to not store them. We much prefer to just sell them off. There was the initial reason because we don't want to store them.
But we kind of have to know.
And we have to share the infrastructure here here at the U.B.
I've proved sometimes troublesome with power, water, etc., water and any major issues this year.
Yes, we managed to successfully break some power breakers especially in the network operation center.
So we're flying blind there because it's like a routine for us to break it every year and we forget over the year and then we are being reminded of that.
And so we have two power distributions in our knock and we're always using the wrong one because there's a special one with more I think 64 amps that we could use and we are still using just the power sockets on the wall.
And we had maybe we learned from that as well. We should write it down.
But we had some rooms also that have issues that they overloaded the circuit.
But nothing like last year where some random person needed to debug power issues due to a failed device and it was constantly standing at the power break and switching it back on to find which device it was.
So nothing has come to my mind. I did it with a broom cause it was high up.
So on the power we are actually purchasing proper power cabling and power sub distributions for next year so that should hopefully be handled as a fun fact which was not really infrastructure but very very exciting on Saturday morning.
We set up in the wrong room and there was a math class going on and they just ignored all the weird cabling and cameras and did their thing so we very very quickly
Toward on this room while they were giving them a math class and build up in the room next door where we were supposed to be in.
So do you have any good statistics about lengths of cables, number of access points, average data, amounts used across the campus?
We have good statistics but not on what you ask.
We actually now do have per room setup instructions with we know which cable length we need to run on here at the wall that you see this like a super Mario level in 2D.
But we actually have those instructions it's also on GitHub to see what every room needs for cabling.
We don't have that actually for our network setup because everything is cable and it's not being torn down and after event.
And for the USB part for the access point and such I mean in previous years we actually built out massive amounts of infrastructure for those who have been around for longer you will remember all of those struts in particular in John saw over there.
We have basically built out a complete conference and back then Cisco even used this as a test point and low test for their follow commercial hardware because so many people it's so much weird stuff at the same time.
So this was actually like a thing which which the research and development of Cisco appreciated.
These days it's also it's all just commercial stuff which which you will be pre installs and we get to use it's a wild mix of Cisco extreme networks and this year also your big team.
So if you look at the ceiling you can see there's one lonely Cisco access point and in front of it there's an extreme network access point there's an extreme there's an extreme and there's an extreme so we got way more density which is that.
And there's another extreme and we've we know that the ULB is redoing their infrastructure there they are equipping the rooms with more and more access points.
I'm not sure if you heard the 25 years talk today some of the buildings here are around 100 years old over 100 years old and they are there keeping up with the infrastructure like this year in Charleston the biggest room we didn't need to bring our own Wi-Fi which we've done for years because they installed a lot of high density access points that we can just use they just.
We just get a VLAN that we push over to them and they just push our SSID to their controller and then all of a sudden we have Wi-Fi working here so this is one thing that we don't need to take care.
Because you'll be is now handing that for us and that's really nice because we don't have to deal with all that.
There's another question somewhere in the third and in front there's also a question.
Yeah.
Do you guys do dedicated network cable runs to each room or are you using existing cable?
We guys and girls do like largely these days it's it's static and we pay you will be for the use of the premises like it's below market rate, way below market rate which is also why we think them as a sponsor.
But over the years we sometimes get a little bit of say of where the money is being used and network cabling is usually our top ask and over the years they started building out stuff more and more.
So the amount of cable which we have to run is going down and down.
So for long time for them visitors they would have known that there were cables running outside running in network and we would have network from this device patched via wild just fresh installed cat five cables that we just crimped.
We can get to the university land they just put our VLANs here in front of the front desk so we can just plug ourselves in and after the event they just use the normal VLAN on that port and this is also reducing friction that we have when you crimp a network cable it's not working and you start debugging things like that so.
There's one asking in the front row.
First thank you very much put together this amazing event and everyone who's volunteered and spoken at it.
I'm just wondering how many people are in the knuck or like in the do you need do you have like a dedicated team of people who actually manage network infrastructure at the back end side and if so how many.
You're just looking at them.
Well, thank you.
We have we have another two folks Peter and Walter that hopefully can see us and we have a new join volunteer our on all who will hopefully assist us also he used to work here at the you'll be before so he might be also beneficial for for helping out here.
I'm basically most of the active work routing so it's done for us and yeah.
But they have we worked done by the you'll be because they provide the wireless network nowadays and that's really really something we really.
Yeah, we can just can't be any more grateful that they just take all of that.
Anyone else?
Any more questions?
No?
Going once twice okay.
Thank you very much.