OK, and that's going to be our last talk of the day.
Talking about choosing what message came, and the EPSL.
Hello.
I'm Dominic.
I'm very humbled to be here, after all these speakers who came before me.
I've been coming to first them for 10 years or so,
and this is the very first time I'm presenting.
So, in advance I ask for apologies for.
Thank you very much.
I'm here to present the work of my team this year.
Some of them are in attendance.
And without further ado, let me say all about our alma mater,
EPSL, Ecole, Politic, Fédéral de Lousanne,
the Swiss Federal Politic-Nix School on Leggineva.
This is a rather big place.
We have thousands of students, 14,000 plus students,
about 300 academic staff, meaning as many labs,
like autonomous research entities, 6K employees,
and people from all over the world.
So, there's a positive side.
To that, of course, we have data centers.
So, maybe some of them run things on the cloud,
the public cloud, who does?
Ah, so, a number of you, who has an on-premise setup,
like with you?
All right.
So, apologies for people who are on the cloud side,
because some of the things I'm going to talk about,
specific to our position, which is to have an on-premise data centers
that have VMs and NFS.
We are a team of eight employees and five interns,
and, of course, we don't do just work press.
So, I'm going to speak today about solving a scale,
which is, I think, our main contribution,
operating at scale.
I think this, when you will find this of interest,
we do want to run a cluster of core presses as well,
and developing a scale, but there's a little trick here,
a little cache that I will mention later.
So far, that we are going to use tools
that two people are probably very familiar with,
who are going to use containers, pods, PSPFPM.
I expect NGNX to be a little more debatable,
and in fact, we probably will have to change that
in short order, and, of course, enough premise cache,
even though we are, we have our own cloud,
we do want to come the use of a cache,
I get that, get back to that point later.
Earlier, we figured first, that we wanted work press.
So, we were users, before 2009,
we were users of something called dryer.
It's another CMS, and we need to reform that.
And also, we wanted to federate all the, you know,
my lab, the TPSFLCH, and there are a single domain name,
and we figured, this place is a bit of an access control headache.
Why not have a galaxy of what presses,
all living under a particular segment of the Europe space?
And we did the math and came up with hundreds of containers.
Sorry, of what presses, and the first iteration of that was tough.
So, we actually went for a WordPress of the 2000s start,
you know, like, you upload your PHP to some commercial,
hosting provider, except in our case, the hosting provider was us.
And, whether you want to run WordPress or something else,
is kind of up to you.
A bit, because we didn't know any better back then,
and we had actually outsourced the exposition.
We were not said a DevOps team, we were a development team.
So, that's me about two decades earlier than that.
And, that was not really at scale.
We used our patch, and as you might know,
this is a broad choice for something of that scale,
because basically you are always paying full price for everything
on query, whether it be, whether it be the homepage,
or humble JavaScript file somewhere.
And, we run that stuff over, and the first,
because NFS is what we have.
So, we used NFS as the hosting provider.
And, that is a bit of a disaster.
Because, every time, WordPress starts up for every single click
on every single page, you have to pay a thousand times,
the RTT, the wrong trip time, to your file.
Because, there is none of the credit on the server,
and no any better.
You have, even if there is a cashmished there is,
you have to revalidate it.
So, in 2025, which is this year,
we had to do it again, and do it right.
And, we went for a much more modern, much more,
I will say, stand out, architecture.
So, we're going to zoom in on that bit here,
and later on that bit here, and finally, this one with a demo.
So, you want a nice cash, if you are going to be serious with
service at scale, you want something that turns your
footerabytes of data serve daily, that's what a day
of serving looks like at a perfil at CH.
Into just one terabyte, served by your architecture.
So, that's a nice error, and it pays for itself.
And, let me jump right to the chase,
as regards the serving infrastructure, how many positive you think,
we need it for 800 what presses and one terabyte of traffic per day?
Well, the answer is right there, how many colors can you see here?
Two, and the other things actually, you know,
screaming, they are humming, as they should.
We could actually make do with just one.
If we had, you know, but we have two for obvious, you know,
resiliency reasons. So, how does that work?
Well, first, we decided to change from Apache to a non-blocking
web server, something that lets the heavy duty
job of an in-road press run somewhere else in a PHPFPM site,
a site car, I would say, or separate container,
and NGX is able to serve hundreds of thousands of,
or tens of thousands of simultaneous connections,
out of a single process, which is more efficient.
And, obviously, we had to turn our general purpose
and our posting approach into something in which the PHP code goes into the container image.
Right, so it's harder to backdoor, and now you have a real
FS cache with metadata interaction meaning that you
revalidate the cache of PHP in milliseconds.
So, first thing that went away was our internal secondary cache,
which was called vanish, so out with it went a lot of our
operational complexity, which is what we think.
Now, NGX has to be interested with making it all work.
It knows about all sites, all your rows, and all variables,
like there is no two world presses that have the same username
or password in the Maya SQL or later, MariaDB databases, right?
It has to serve as I mentioned on a first line basis,
like the small files or the things that are in NFS,
because some of them stay there, like the uploads from people who want to publish,
you know, images and movies.
And the rest goes to PHPFPM, and that means that every single
world press query is started fresh.
And I want to spend a little time exploring how this is even possible.
PHP has this design of a site which turned into a skill of feature.
It forgets everything, every time.
It starts from a blind slate, all constants, all variables,
all states, all coordinates to be reloaded,
or at least pretend to, in a cache.
I'm not exactly sure how this is made efficient, but it is.
It involves something called Shen memory and a Zen cache,
from that Zen company who makes, who is in the business of open source.
We are effectively turning the entire world press into a function as a service.
So I'm very tempted to claim that this would be very tricky to achieve
in any other programming language than PHP.
So we are small team.
We need to be a parity cascade with all these hundreds of processes.
And therefore we wrote, and we offer, this is our flagship contribution,
the word press operator, something that does the heavy lifting of operating
all the scary Kubernetes objects, and NFS, MKDRs, and the PHP code,
all a breeze, all with the cop from one of the Kubernetes
operators from working Python, and in a fully IPF diagnostic way,
meaning that you can start using it right now.
I would like to take the time for them or for this.
So if you are connected to our cluster, sorry,
this is the path where I get nervous and mess something up.
Hopefully I don't. You can say,
come sit here, get word presses.
I suppose so, better.
So there's a number of them, but this is the testing stance,
but you could also do that in product with another coupon fee,
and get all the 800 word presses that I mentioned.
But there's someone in attendance who doesn't like me working on the production
instance, so we're regime from the testing stance, if you're.
So right now, you can see that I don't have anything like my sleeve,
and there's a fall or fall at the hell of us than Europe.
But we wrote this manager of this backend up to create
what presses and delete them, and I took the liberty of squeezing a few
seconds out of the demo by prefilling it, so that I only have to
create here, and that should create a Kubernetes object,
that is going to make the operator operate right there.
So it's already creating, it has observed the event, create word press site,
and it's doing things like creating a secret, a user,
a grant, looks to other legating work to another operator, which is the
Mary at DB operator. Then some PHP runs, some of it has warnings,
we could squash some of those warnings, some others we could don't.
Ah, to bad.
And once the operator is done, the number of plugins to be installed, it says succeed,
well, that's this great. If we go back to that fall for page right here,
we can see that something new is coming.
So did we automate ourselves out of the job? Why aren't we sort of?
We actually got to focus on the fan part, on the important part.
So as we, as you saw, we wrote a back, back office, sorry, not back end, back office,
up, that helps us manage the fleet, right? So you'd probably have to write that
should you want to adopt the word press parade.
And we fixed our wrong 22 power 22 problem, which was crashing production every day,
by moving the current workload out into their own dedicated set of pods.
That too, you might want to look into, if you're out to host work press, at scale.
And we developed a some scale and would like to increase the scale.
And that's a bit the reason we're here. We very much like
to get in touch with you outside in the whole way, right after the presentation.
And one way we did that for ourselves for starter was to write a development pit.
If you type these three commands, sorry, two commands, one euro,
in your browser, sorry, in your terminal, and then in your browser, right now,
you get to fully working what press, on your accession, Mac, or Linux.
So is this really for immediate consumption? Well, it depends.
We have this flagship contribution, which is the operator, and we do believe it's ready right now.
Some of the things like the integration with NGNX is coupled with the way we do things,
which might even be even more controversial than NGNX itself.
We use NGNC, but we don't use our goCity, I hear that maybe we should.
So we would need to work to provide this as an independent piece of infrastructure,
and we would very much like to show this story.
Should we find someone here or on the internet who finds interested in it?
Obviously, if you're one, our WordPress, you'll have red and white WordPress,
which you don't want, soon have to rewrite some parts.
The theme would be prime candidates, and some plugins,
it depends, like you don't pour in nuts and trusting in large menus of a PFL.
And some of them, like the push gateway, the thing that lets us, on the basis of the WPCron,
extract metrics on a hourly basis, like instead of every three minutes, like I don't know,
the number of pages per language, what we probably can use that right away, as well.
Thank you very much.
Thank you very much for this, so how would an update of the WordPress application propagate to all the instances?
That's a great question, so when should I repeat this or not?
I think I'm happy. All right, so when we upgrade WordPress, most of the time it works,
meaning that we just build another image, we actually have a homemade continuous integration system
that relies on text on pipelines, and most of the times, the website that pops up in the morning,
sorry, on the test instance works, it doesn't have any so-called migrations and it changes
required to the database to perform the update. When that is not the case, we obviously want to block
the production, I'm sorry, block that image from going to production, which is fairly easy,
because as we use Ansible, it just doesn't happen automatically. I do want as a team leader,
I want someone with both hands on the keyboard before production changes.
And the unsigned that case is that we will have to run a campaign of WPCleek commands that
will run the update, and we will do that as part of a warout that is organized on a special basis.
This part is not, we did not automate ourselves of a job.
The other question is, thank you. Thank you very much.