Thank you very much for your patience, first of all I've already asked who's heard of MSF,
but the people who haven't heard of MSF, this is us, international independent medical humanitarian
organisation. The independence is really important, that is what enabled us to speak out when
the witness, for example, violations of humanitarian law, war crimes, and that kind of stuff
and I'll talk a bit more about that in a minute. I'm going to talk very briefly about who we are,
what we do, where we use next to S, why and how, and then if I move really quickly there'll be
time for questions. I'm Ian, by the way, this is Shaw, and Ranzis is over here. So, we're quite
proud of what we do, we work all over the world, and almost all of our funding comes from individual
donors, and that's what means we can be independent. So we don't have to worry about government's
cutting off our funding because they don't like us denouncing their migration policy or enabling
or committing war crimes, for example, which you might have read about in the news recently,
gets more and more important. So, Shaw and myself, we work for MSF, our payroll is up there
in the general administration section. So, yeah, we do a lot of stuff. You might be wondering,
okay, find vaccinations, but what does NixOS have to do with that? The answer is that you'll find
NixOS in our systems deployed in the Clared, and also deployed in the healthcare facilities
all over the world, where you work. So, for example, in an ITFC, that's an intensive therapeutic
feeding center, which is used for feeding people who are either very small children or who are
too malnourished, and I'll show you some photos of them in a minute. This is a picture of two
ambulances in IT. The reason I'm showing you these is because there's a story behind them. In Christmas
2023, we were forced to stop activity in IT because our patients were being killed in transit
between hospitals. They'd be dragged out of the ambulance and shot. So, while we couldn't guarantee
the safety of our patients or our staff, we had to stop work. While we stopped work, and while
our hospitals were empty, we went in there with two of our guys, so Carl and Daniel, our X-ray and
network technician, we upgraded the network, and then we installed virtual machine running NixOS,
and on top of that, an X-ray image archiving system. And when hospital came back, it came back
better. The surgeons were more able to look at the X-rays of these kind of operations. MSF is a
big organization. Think about Fortune 500 company size, but we're about the fortune. Sure, and I work
for a part of MSF called the Operational Center Brussels, which in total is about 15% of the
global stuff. And IT inside the HQ is 505 people. Five people work on NixOS. So what I'm trying to
convey here is that we do not have a choice. We have to work at scale. Right. Next slide. This is
some of the cases where we work for a centralized data systems. I'm going to explain to you one of
the systems now. So this is my degree in intensive circuit heating sensor. You can see the solar panels
on the roofs. Those are new. The deployments are solar panels across all our hospitals,
has been enabled by data driven decisions, which are enabled by a platform called Things Board,
which is an Apache licensed MQTT dashboard. That's deployed on top of NixOS, and we
centralized power consumption information from all over the world on that platform. And for
example, the solar panels here have cut diesel consumption generators by more than half. There's no grid.
It's diesel, solar, or nothing. Next. So inside that hospital, you'll see kit like this.
That's a ruggedized filter. It's kind of mini rack set up with the UPS and this kind of stuff.
They're very, very rugged. I only know about one which broke, and the reason it broke is because
a snake pulled inside the power supply. The snake died. The power supply died. Very bad.
And then on top of that, we also deploy smaller pieces of kits. So this is a nook. It's a small industrial
panel of Intel PC. Very powerful for their size, and they fit in a backpack. So you can ship
them very easily to wherever you have to go. And we use them, for example, when you want to isolate
patient data, and you can also do things like using for active passive replication.
This is the example of this inside the hospital. Here, for example, you would see
patient nurses in health care workers watching around with tablets. Inside the therapeutic
feeding centers, it's really important to keep track of who you've fed and what and how.
The tablets themselves are linked to an application for DHS2, which is running on a pair of nooks.
And that's, that's inside that. This is two of our favorite pictures in IT, MSF, on the left
is Alex. He helped design the field network kit. On the right here is John. He's working on a vaccination
campaign in RDC against measles, where a quarter of million people every year are still infected
and more than 5,000 people a year still by. This is a platform. So we have the platform and then
we have the applications. You should recognize most of these logos, especially the first one.
But you might be wondering what it's Ansible doing here. And the answer is that we started using
NixOS before a subsnix became available. So we use Ansible to keep our secrets encrypted.
So those are the platform components. The applications that we deploy. So DHS2 is a public health
management information system. You can use it to track patient data. You can use it also to do things
like record information about academics and this kind of thing. Or I think it is a piece of software
from managing X-ray imagery and this kind of thing. It's also open source to relative to an
Belgian. And then BAMNI is an electronic medical record system. It's used to run hospitals. And
with that I'm going to introduce Charles and experts in BAMNI. He's deployed medical record systems
throughout Banger Dash before he worked for MSF. And then when he came to MSF, he started deploying
DHS2 all over the world without NixOS, which is what makes him ideal to explain just how much time
it's open over to you. Any time you do the mic once, it's okay.
Okay, thank you. Good morning, great. So, why do we use NixOS? So MSF is a big idea to do this.
So, we have to manage complex IT operations. And this complex IT operation will need a system that
is robust resilient. And we wanted a system that can evolve quickly by itself with a minimum
maintenance. And also we wanted a system that the same system we can run both for our field
operation management and headquarter operation management. So, there we select NixOS.
As it's a support for declarative configuration management, this will all know. And in
first lecture as a code. So, then I won't say how NixOS help us to overcome our many
operations of challenges. So, as we started this setup, we initially
developed what we wanted to get our platform, like as everyone wants. So, we wanted our
configuration to be centrally managed. It is secured and declaratively we wanted to manage.
Then the changes of our configuration we wanted to be automatically tested
then on deployed automatically. Then, it will be a security pass update. Then,
operation everything we wanted to manage with a very minimal effort with a very minimal maintenance
effort. Even access control to the server we wanted to manage automatically and declaratively.
And as our servers are distributed at just different places, we wanted a easy access to our
server with a minimal dependency on the network and we wanted to deploy containerized application
to the NixOS server. So, we will see how easy NixOS we have received majority of
on design goals that is set initially. So, let us go between history about our easy NixOS.
So, since 2018, we started to fast deploy our custom NixOS platform to manage a fleet of
Linux servers and since then we started to write our machine definition in a Nix code
along with the application configuration, application 6 is together and save it in a
GitHub repository. So, we can make all this possible at NixOS support security
configuration management. And then the center repository we use that is connected with all
the server we use in different areas. So, the servers run by the schedule and the pool the
Nix code and building side itself and get the update of the configurations. So, here is a
code snippet. How we define our servers in Nix code? So, everything about the server,
server time zones, other settings even even the disk partition, boot mode and the services we
wanted to run inside the server all are defined declaratively. So, here is a example here is a
here is a configuration code like for example, to all those servers we wanted to
disperse those content as a configuration and this part actually defined like in the name of
this file this content will disperse and plus all those servers. So, this is how we define our
configuration in NML. Civil I will define our secrets in NML file but that is a encrypted
with a civil fault. So, before moving a posit further just I wanted to show you this picture.
This is where basically our server as a few server ran. So, it is a picture of jump to the
refusing comes in cost with the Bangladesh. So, it is a very remote area where sometimes to run a
server very essential needs right power and internet supply is very difficult. So, we need to
run those servers under those extreme constant and we will see how using NixOS we overcome those
challenges. So, as we have everything in a code. So, this gives us a significant advantages right.
So, we can take the advantages of advantage of for example, every changes to be tracked then when
the deployment or we can take the advantage of gate automation for deployment for testing
and the gate ops operation we can run over it. So, now I am going to talk about how we regularly
upgrade and gate security pass update for our servers. So, as our servers are connected to our central
configuration repository it pulls the updates from that central repository and rebuild itself and
an upgrade itself and we also use in our server NixOS, in our platform we use NixFlex and every week
we do the flight lock bump with help us to pass the security update from the upstream and when
the NixFlex version upgraded twice in a year we also do the platform upgrade and twice in a year.
But, we run our upgrade addition in a three steps or three wave frequency faster,
middle wave and final wave. So, in the first wave we actually run the
operation in our relays and dead machines, middle wave in UAT and test servers and low SLF production
servers and the final wave we run in our mission critical application at the server hosting mission
critical applications. So, this process gives us a lot of advantages why? Because we minimally
wanted to disrupt our operation in the field and if application inherits any force we wanted
that to be surfaced in a faster and not hit impact the final wave servers for critical applications
running server. So, this is from why the way it started our faster the
notes we just next to my desk and the final wave ended of executing the field servers for
basically our mission critical applications run. As you see the doctor is taking
preparation of a surgery and we wanted him to be minimally disturbed. So, that is our ultimate goal
of every time whatever we want. We do not want to show the technical excellence, we want to
vary minimally disturbed our field operation. So, how do we test our next code? Similarly,
the count to two. So, as our everything in a code. So, every change is we do in our next
source configuration. Then we run the build test validation test integration test
recently we started to use the VM based test and we wanted to run our test as similar as
a production environment. The goal is ultimately the same. We wanted to get no
surprise after deployment. We also manage our code using a staging and main runs.
Particularly, the major critical changes in an next code we progress through these
staging runs and the regular operational changes like changes in a configuration which is less
impactful. We progress through the main runs. So, we follow this geared work forward to decouple
the development from the operation. Because for example, when I am doing some critical changes
or while my other colleagues he needs to deploy some small changes of the configuration,
I do not want to block him. Because my changes would be more
in fact, who is a man here, I need to be tested, who is gone through the gestaging
runs and the normal changes go through the main runs. So, how do we manage the remote access
to our servers? We manage remote access to our servers using relay. Because even this simple task
is not simple for us, because the servers are distributed across many places, not in a single
network, in a different network. Some are inside a field network, some are inside a
scenario, we use some are for example, in a hybrid cloud, some are in a cloud. So, we will
relay, we overcome the challenges and we are becoming less dependent on network, we can ensure
access to our servers. So, how do we, this is how we declaratively manage a user access control?
So, in a simple file we define the users, user roles and the roles access to the servers.
So, who is a transform to next code? Next code, that is transform that is the
son file into next option. Then when it is basically get builds, we get the users inside our
server. So, which means we are also declaratively, declaratively can manage our access control.
This gives us couple of advantages, first of all. So, we can prevent the configuration
grid and then we can for example, another applications like I am or they can also use the same
definition, which is a different declaratively. This is how we define our containerized application
next source server. So, we declare application in a Docker container. We intentionally chose
this so that, for example, I do not have any excellence with a next or next code. I do not know
nothing and about it, but it is still I can deploy my application inside out inside those next
source as the deployment process is automated and yeah.
So, this is how we define our deployment service as you can see in here, just I need to say
which application I wanted to deploy. So, I just write the repo name and then the browse name
and that is and then then the machine target machine, why I wanted to enable this option. So,
that is it. So, then the automated deployment options, just check out the emails and then
deploy it to the correct target machine. Yeah. And then after deployment, we will get ready
our application to be used. How do you manage our installation to a new machine? We basically use
next source anywhere and this code. We compose this to into an installation script as long as
server is resubbel by SSAs. We can install next source to there. We encrypt our data partition
is looks to this usually do because our sensitive patient file are inside of it. So, now I am
handing over to Ramsis or basically the main architect behind the Ramsis. Behind this system and
here we will talk more about our next improvement plan.
Yeah, go ahead.
Yeah, maybe just very quick because we are kind of running out of time. So, there is a couple
of improvements that we see in the system. So, as we mentioned, we started in 2018. So, the whole
ecosystem was a lot smaller back then. Things like shops, nicks and such didn't exist. Nicks
was anywhere that didn't exist. So, it is a bit weird. Maybe that we are using Ansible Fold for
secrets, but that is because we kind of had to roll our own system at the time. So, at some
day, we should improve. That is also linked with the whole encryption key thing because there was
yeah. It is a bit weird that we use the same keys for like secrets and SSAs. There is a couple
of things that we should probably migrate to like the system D in it already. We tried it once.
It didn't work. We should probably try it again. We would really like to use
verified boot and measured boot to secure our servers in the field, which we did a
minute so far. I think there has been kind of a bit of advancements in the ecosystem so far,
but it is still not the most straightforward thing to do. We have been writing on VMTest
which are very helpful to avoid issues. And then finally, one thing that we still do is we
evil and build everything on the actual machines, which has a couple of advantages for us,
but eventually we probably want to move away from that.
Okay, I'm going to skip very quickly past our next slides because I think we are completely out of time.
So, I just want to say, first of all, we have had some challenges, but some of the books have
been helpful. I want to say, first of all, Nick, so I see an amazing technology and it's a good
community. You all deserve a big round of applause for the products that you've managed to create here.
It's really a really good technology. If you have to choose one thing and I'm going to duck here,
it's stable place. Yes, that's, yeah, and I'd also like to give us a big thanks to Ramsey since
Nundite for their help. Ramsey's was the original architect of the system. We're
operationally independent there, but he's been great for helping us with our evolution. And that's it.