Okay, yeah, as a tradition, we are doing the infrastructure
of you now, also as a tradition, I was just in Formula
Nitirons of L, so I will leave on the hour at 1635.
So yeah, you clicked, okay, so for those who have been here in the past
in the infrastructure where you will see that this is pretty much the same as
the last few years, and this is by design, because boring is good.
One of the most important things which we did over the last few years and anyone
who wants to clone our conference or anything, you'll see more at the end.
Try and put stability into what you're doing, do not reinvent the wheel every
single year, it might sound exciting and everything, but it is actually good to be
boring, have stuff which works and then just make it work or keep it working.
So as of today, what we have is we have a Cisco ASR which is just
used to be like a super expensive router these days, you can get it for almost
tree, which is most of the stuff, in particular it does the Nit64, we're going to
get that to that in a second, we have a couple of old switches, we have a couple
of slightly less old switches which do 10 gig throughout all of our infrastructure
where we don't rely on your bees.
We have two really old servers which do a few things and we have three more
servers as of recently which run proxmox and then we can run all the different things
on proxmox, all the different services also broken out into different VMs, so it's not
like this huge pile of crap on one single server, we actually were able to segment stuff.
All the monitoring is done, we are permitting this low-key graphana if you don't use any of those,
highly recommend it, I'm obviously biased or we are biased but because we maintain
as in everything but it is really good and we have a T-Gron service where we can
persist data and where we can host longer run in services, basically under the year and
everything, it also like a few of the control service and such.
I forgot if we have an extra IPv6 slide, now we have this at the end, just click.
So for video, do you already want to take video?
So high for me as well, I'm Basti, I'm also part of the infrastructure team here at the
Faustam.
For video, you see that most of the rooms that you've been walked in, there's a tiny little box
here capturing all the videos, trimming it live into the world, hello to my family, they're
watching, I hope at least.
And we do collect everything from those boxes offsite and make it available to the folks that
are not able to join or live stream it and watch it afterwards.
We have roughly more than 1,000 talks that we are producing during each of the events with
those boxes and if you like, you can build them, it's all open design like the PCB sensor
on, it's everything is on GitHub available on our repositories, we have those boxes here,
they send their streams to the renderer farm, the rendering farm, dumps it somewhere on the net,
they'll do the cutting and we do all the pre and post processing until it hits our streaming
endpoints and then maintainers or death room managers or whoever gave a talk can then
open the browser, get an invite and they can just cut their talks and do their thing.
Video boxes look like this nowadays, it's just Linux with some advanced hardware that
there's able to communicate with the rest of the world, nothing very spectacular but it's
built by the awesome video team that we have and yeah I think it's one of a kind that we have
such cool features in there. Next year there will be some new features and I'm not going to
talk about too much what's happening with them but we will use them also to provide you a better
experience in here. Yeah I'm not sure it's quite quite dark that pictures, this is how it
looks on inside, there's a lot of tech stuff, you can eventually see that there's a fuzz
them logo on it because the PCBs are custom made, also those, they're fuzz them logo
within the backside but we try to do everything on our own and not rely too much on commercial
vendors here. If you want to know more details, there's an excellent talk from Martin and
Angle, they gave it last year, highly recommended to watch it, it's produced by one of those boxes
and conserved, the link is behind that QR code, it's not a trap, you can scan it or you can just
check last year's archive and it's not rig rolling, it's not never going to give you up,
no worries. To do such a thing, it's the same with the network and with the video we have very
detailed plans on what to do, when to do who's doing what for for each room, so we make sure to
not waste a lot of time of our volunteers, they're all doing that without getting paid, they get
some food and get a nice shirt, thank you for rolling, tearing by the way and that's it and so we
want to make it as effective as possible and so that's why we try to organize it in a way so we can
run the conference year after year without turning too much through volunteers. This is what
it looks like from inside of the knock or walk network operations center and video operations center
where we see everything that is happening, this was just, but earlier today we have the ability
to have a look at everything that's happening inside the rooms, you see the audio meters here,
we can see if the audio is correct or not loud enough, things like this can automatically be detected,
also the operator see that immediately if something's off, we can then for example just lock
ourselves remote into those boxes and just pull up the lever and get better audio quality,
we can see the near real time, it's streaming the data from the box so we can see everything's
fine, audio is working and if you want to photograph this I will put the slides just also on the
website so you don't need to take photographs of that. That being said, data is captured here,
we send it to a rendering farm, it looked like this, pretty bad in the last years, we improved it,
this is what it looks like this year, now we have better, better conditions, we most of the
things that we've changed with the central switches here, we have now better connectivity and this
is the main thing that changed from last year to this year, you see it, we now have two uplinks
so we're fully redundant with 20 gigabytes instead of 10, there's a lot of people crossing that
room and going there, so having redundancy if someone slips over the cable so it's not a bad
idea to have, so this is the only thing we changed for that year that we got a second fiber that we
run to that switch of the room and are redundant on that side as well. Time lines, Richie,
this is typically your topic, how it used to be and how it's now, so can you go back once,
because this is actually something which we never talked about, you can't really see it here
but this is something which I did a few years ago, I just label onto the hardware what goes
there, because usually it's a lot of panic and people asking what should go where and there's
actually one staff who ever seen the year ask me the same question and I always tell them just
look on the label, like no front or anything, but it is super convenient, like put documentation
into the places where you need the documentation next time, yes it's nice to write it on the
wiki, but like this kind of thing is tremendously important to make stuff boring and just work.
Give yourself this just like cheat code from the past and just document stuff properly in the
right places and make all the labels bright orange because you can see them across the room there
ugly but they work, so timelines, you can go through the details of all of this but the
gist of all what you're seeing here is things get better over time and we get sleep.
If you look here, 2015, Saturday, five in the morning, it was basically go back to the hotel,
take a shower, come back and continue working, try and not do this to yourselves, I speak from
experience, try and prepare and again reuse and make things stable. There's one of the main
messages every single year because we see this again and again and again and last year I visited a lot
of conferences all over the world and you have a few very dedicated people and they're driving
themselves against a wall, stop, think, engineer and try and become better and just finish your
stuff earlier so you can actually sleep and also have some fun. You can just click through the same
principle, you see it is just getting better and we maintain the stability that we actually
do stuff before and not just in time. When here we had stuff literally running on the day and here we
have it almost a month and it once. Next, so for 2026 we had a big plan to reduce stuff and like a
few years ago we made this plan, we are basically done, the road service are just sitting there,
we can just get rid of everything, yeah, need bandwidth and like all of those things and the
point is we actually planned this over several years and we executed over several years without
having too much work because the two of us are based in Munich coming here actually is a lot of
work and yes we have people on site but then it's not always the right thing and something is
just need to plug in and write this one command. It is hard to do stuff off-site to give you
self a good timeline and then just execute over time with the timeline and yeah, we run, next.
And since none of us speaks French, getting remote hands here is extremely critical so
better not do it from Munich and bring something without speaking French, that's another part of the
story. None of the security here is speak English, that is actually a problem for us.
So everything went well until yesterday around noonish we had some Wi-Fi issues, the
only issues we had with the network, at least that we were aware of, we had issues with our
AppStory upstream DNS resolvers and kind of blocked us with due to whatever reason and
this only happened on the 06-4 network that we're providing, we changed the NS announcement
from our internal DNS resolvers to some external, we changed then our internal 064 configuration
and reconfigured our internal AppStory service to Fryfunk Munich instead of the old ones that we're
not blaming here, maybe it was an issue we haven't heard from them, they provide a service,
there's privacy friendly and we just routed the traffic to Fryfunk Munich so pick shout out to them
for providing that and after some testing we were just back online completely on the traditional Wi-Fi
that you're seeing here, there's two SSIDs for those of you who have mentioned have seen it
on their phone, one is called fuzzdom, one is called fuzzdom dual stack, it had other names in the
past, we're not gonna name them, so everything we're trying, we just did some minor re-checks
and then everything went back to normal mode, why are we mentioning that, shout out to the
knock team of ffmook for helping us here, also I'm not sure if anyone of you has seen richies
introduction talk or opening talk for the conference, Fryfunk Munich is also listed of one
of the European alternatives that we have since we're talking about sovereignty, they're listed
there, if you'll free to follow those QR codes, I tried my best with having the fuzzdom local
in them, but they're also available later in the slides and they help us out with their service
which they provide for free and it helped us save our ass and run in the conference.
The upnings, almost like last year, we have sponsors, thank you called for sponsoring the bandwidth,
we see nowadays most people are not using that much bandwidth, I see the same here, you're
paying attention, that's nice, you're not compiling or kernel or something like that or downloading
lots, you're actually paying attention, which is nice, and they have mobile phones of course
and then we have a second uplink from destiny, also shout out to our sponsors and they're
providing us the bandwidth that we can use completely independent from what the university has
is the uplink. One fun fact about our network outage, so we had two times when we couldn't
sell shirts at the info desk, those devices are very picky, but actually there were people who were
running network tests while we had our outage and they commanded us on the performance of our
network while we were dealing with the outage. Part of the moral here is switch to a workaround
quickly and aggressively and then fix stuff and properly engineer and verify everything in the background
and then do the cut over back to a normal production, which in our case we had to do of course
while for iPhone is very privacy friendly, we still are not comfortable with like leaking too much
information off-site, we keep pretty much zero logs or anything and we wipe and everything so we
are very conscious about not generating any data which someone might be interested in, so it was
important for us to switch back, but basically click workaround and then back to proper production
as soon as you can. Next slide please, and also you can clone the conference, but those who
care already took screenshots or photos, so the ASR which we have is end of everything,
Cisco has extended that lifetime a few times already, I get to press this because we are staff,
but it's probably going to be completely deprecated but also we kind of like open source around
here in case you didn't notice, some of you probably did, so the plan is we replace the ASR with
network hardware, probably a register, same as the new distribution we have in other places,
but we want to switch the BGP session to routing the net 640 and SX4 is already, but net 64
to open source software instead of running it on close source hardware, we will have the network
hardware as backup for the upstream, just in case something goes boom, but most likely and keep us
honest, next infrastructure review, we will be able to just be on open source hardware,
well not hardware, but software for probably the entirety of the routing and everything. Next slide please,
the other thing, over 10 years ago I had a really stupid idea and I said hey why don't we switch the
main network to net 640 and it was like yeah I do wait a minute okay, for over 10 years I thought
this was the second large major network conference which did this and I have been informed
on last Friday there was actually the first worldwide where we did this and we found eight
ton of bugs in various distributions in the Linux current in a lot of software because this is
faster, so we actually like I have friends that drive in such we actually measureably improved the
I need to leave soon, the IPv6 distribution in software and like support and everything and even
like the adoption on the internet. The plan for next year, the primary network will be IPv6 only,
again keep us honest on this one and we probably start switching out the names of the dual stack
one just so we prevent people from auto connecting to really forest people to to look at IPv6
more currently the main ones and pre-empting the job so I'm going to make it right before the
closing is a get up which is still IPv4 only so if anyone from get up it's listening next year
one of your largest users is going to be IPv4 only, in a change we can just put the scene name
and just redirect to codeberg which supports IPv6, the other thing, the other thing chat.fosem.org is also
IPv4 only as of right now so keeping ourselves honest by next year we expect this to be IPv4
and IPv6 as well and also we are probably going to make a third SSID to just be able to encrypt
traffic we will probably make this IPv6 only as well. I think we are done and have some time
for questions but I need to run really soon. Thank you. Any questions?
What's the rationale behind having a cluster of think pads instead of a big multicore server
like a threadwer bar or something like that? I'm sorry I didn't understand. Oh sorry.
Why think pads instead of a big server? Why think pads? So the question is why do we have
think pads for the render farm? Several reasons just like splitting domains so the blast radius
of any outage is smaller but also we wanted a back rebate back up we wanted to have different
machines back then the ms and such were really brittle and hard way which could do this was
really expensive we wanted to have a keyboard and this place kind of nice in the way as well
there's units which they make which have all of this in that laptop and that's basically why we just
took a pile of laptops. All right did we trip the breakers in the knock or all the places again?
No this was part of last year's post-mortem to first go to the knock, secure the 400 volt output
and put everything there instead of the normal circuit that we're using the years before because
we learned sometimes we learn and then we apply that. Excellent question by the way.
And there's a guy who bought the 400 volt distribution boxes I'm machine sitting down there
because this is the idea to do that.
Hi great conference thank you. Me being network engineer is the special reason for the
MTU being 2004 800 on the IPv6 only network.
That's the thing is that we don't own the hardware and we don't operate it we just got the Wi-Fi hardware
and the Wi-Fi hardware and we just get the Wi-Fi sponsored by the university
and we get whatever day push over the wire and we just have an interface where they throw out some
wheelons in our direction and the rest is in our issue.
Which unfortunately need to leave because he's got another talk in minus three minutes but I'm still
available. You mentioned earlier on software to find storage curious what you're using.
We used built-in safe that comes with proxmox we switched to proxmox a few years ago and we're using
the built-in safe nice shot by the way. And of course we use safe we have folks on our team
that also do software-defined storage in their day job but we agree on using stuff because it
just comes with betterly included when you're using proxmox and it's yeah.
Thank you for explanation. How much infrastructure and help does a university actually contribute?
The infrastructure it's one half size drag that we own everything in there is owned by us.
That means there's a fuzzdom vis-a-vis this is like the club fuzzdom where all the hardware
also belongs. We get the app links we are allowed to use some parts of the infrastructure.
If you look at the lectern here's an Ethernet port over the weekend they turn the wheelon to our
wheelons that we agreed on and this is actually what we get from ULB thankfully also the access points.
You see one two three access points all of that we get from ULB.
Years earlier we brought our own stuff and did wild cabling these years are gone. We can share
the infrastructure and with them which is really really nice. Where's the microphone?
Ah, it's the app, okay. Hi have you considered using BPP FDIO SDR router instead of the ASR?
Yes we have. And will you use it? We're gonna see.
The things we had discussions two years ago around VPP with the people of VPP here on site.
But that time we needed to fix other things in our setup. We're actively working on building
a build of materials that we would need to do that all in server without relying on any vendor
or whatever would come to mind and trying to build out something that could work.
But we just started yesterday. So be with us. But the idea is ideally to go in that direction.
Yeah maybe add a other question. Do not six for as you said that will be that was for some time
of site but variables. Because the DNS six for didn't work. The DNS broke the DNS six for
broke. It was only the DNS six for part that broke. And that's why we did the people with
not six for experienced issues with IPv4 only systems. So you removed the not six for from the
ASR to some somewhere. No we fixed the DNS so the the not six for wasn't the problem. It was always
running. It wasn't usable. It was like that you moved the not six for to elsewhere.
Then it was a typo on the slides. Okay. It's okay. There's one guy over there right in your
hands for longer time. Last question. Okay we need to get we get kicked out here.
I'm interested in using the video capture boxes in the rooms for other volunteer events of
less technical nature. Do you know if there is any organization that would take some money and
make me one. You can rent from from us for for the nation for example. This is one of
things. There's also event in fra from the Chaos Computer Club CCC. I feel like they also do similar
things with infrastructure that you need to run an event. But if you're interested in those boxes
for other non-commercial conferences, we typically ship them to them for small donation and
get them back hopefully in one piece. If you're interested just come talk to us. Okay thanks.
Yes absolutely non-commercial. Thank you. That being said that was the last question. Thanks for
attending Faustum. Have a great rest of the day be excellent to each other and maybe join
the closing sessions in Jossar. It's just down the hallway and then on the right.