So, hello everybody, nice to have you in the modern email Devroom.
We saw in the previous talk that there was a need for performance testing and benchmarking.
Here we are.
So, I'm Benoitelier.
So, I work at Nagohar on OP-3 building an alternative to Office 365.
And I'm doing that by building Twake Mail, which is a collaborative email server.
So, it's a product belonging to Nagohar, of course, open source.
But built on top of open source building blocks, namely Apache James.
So, that's the main server developed in the Apache Software Foundation.
It's unique in that sense that to my knowledge, it's the only mainstream open source server that is in following an open governance model.
So, which is quite unique.
And why did I need it to do some performance testing?
Back when we started working on the project, back in 2014, we actually came up with a radically new architecture.
And quite happy, following talks today, to see that it's not longer that original to say that the mail server needs to be state-less.
And delegating its state management to specialise databases.
So, we're doing basically the same thing that's too hard, well-dark and difficult to some extent.
But we needed a tool to validate that the main server is not the database itself.
But that there's an in-direction level on top of existing database is an idea that is actually viable.
And also Apache James is a very old software.
First, good lines, dates back from 1999.
Top level project was established in 2007.
And the internal architecture is still moving.
And for example, one of the very big last refactorings that we did back in 2020 was to make the IMA player fully asynchronous.
And using React or the Reactive code principles, which is a massive code change.
And, of course, we wanted to have a tool to tell us if we're messing up something badly.
And shipping and performance regressions.
And on the actual wish list.
So, of course, we want to benchmark taking root choice and art up to true changes.
I'm sorry, they have been quite long.
We also want a tool to validate custom or deployment.
So, let's say I've got a government and they want to run 50,000 mailboxes.
I need to be able to run a benchmark on their servers to actually check.
Yes, on today you won't have load problems.
It won't collapse.
And to do this, we wanted to have a easy tool run efficient framework.
So, we wanted to be able to generate the load a very big load from a single point.
And, so that it's easier to aggregate statistics.
We also wanted it to be easy to use for us.
So, that means CLI code.
So, that we could review the scenario, commit them, et cetera, et cetera.
And, of course, we wanted to be adapting the scenario.
For example, simulate client that don't have quick rethink.
And what happens is if you run a huge proportional, best client does think collapse.
So, we actually decided to reuse, we actually did research and found no state-of-the-art tools for doing
extensive benchmarks with IMAP.
There's tons of tools around for doing benchmarks.
And one of them is Gatling.
So, Gatling is an open source product that has been developed by some French people.
We've been its Java-based, a phascala-based.
So, it runs on the GVM.
It feels at home for James' people.
It's fully asynchronous.
It's built on top of Akka-actual system.
So, basically it's just messaging, flowing around internally.
And this Akka-based architecture is what allows to generate an extreme amount of flow
from a single load tester, basically.
And it also comes with beautiful graphs.
But our problem is that Gatling only ship HTTP out of the box.
And we need it to develop IMAP connector for it.
So, I tried to do a little graph about the internal structure.
And there's a bit of vocabulary.
So, basically, we are running a simulation.
The simulation is a scenario.
So, what is a single session doing during the internal scenario?
And basically, the idea is to come up with a domain-specific language
that describes the scenario in IMAP.
The scenario is subdivided between actions, for example, select and checks.
I'm sorry that it overlaps.
And, basically, simulation will use a protocol to connect to your main server.
So, you will need to have described that.
Thinner is to inject user that would start running the scenario
and following an injection profile.
So, more on that later.
And then, basically, we've got an object that is IMAP sessions
that's an acto that would use the IMAP session associated to the user of that session.
To start another actor that would be a binding to the IMAP NIO library.
So, that is actually coding the main server and doing the IMAP operation.
Then, when we come back another actor would just record the timings
and be able to do the checks on the response and continue on with the scenario.
So, basically, most of the job was binding designing all those actions
and plugging them with actors onto existing IMAP NIO commands.
Life is complicated and full of wrong assumptions and things like IMAP NIO.
Sadly, it was offering no builds on very old Java abstractions that don't allow you to get notified
when the result of your asynchronous operation is actually done.
So, we had a contribution to the IMAP NIO library, which was accepted by Yahoo.
So, this is how the DSL actually look like.
So, that's no pointer.
So, basically, the idea is that we just name our action.
Then, use the DSL and give, for example, the login.
So, you have, you can see that you can store and also access information on the session.
So, there's that's what the dollar thing is actually doing.
And we can chain actions.
It's also quite interesting.
So, you've got things like waiting.
And you've got all the structure control on building the Gettling DSL.
So, for example, do randomizing things and what we are trying to do it.
So, I omitted much of the code for simplicity here is to replicate the load that we have on actual server
that we can see on our metric system and try to replicate exactly the proportion in that scenario
to try to be as relevant as possible.
And below is the scenario, we actually fit things and we execute the initial connection
and then during the scenario direction, we just do some new action and pose like this.
So, that's, but again, that's not getting a map.
That's how we use Gettling aim up to actually build our simulation at Nagoha.
This structure is ours and you can do whatever you want with it.
So, then once we get it, we can plug a feeder.
We can choose the injection profiles with the inject with a flat profile 10,000 people during the injection period.
It's actually a nice model because if you decide to eject people, you can inject people very quickly and do endurance tests during a long, long, long time.
They're just playing on the scenario duration or on the injection profile.
You can keep injecting people during the entire scenario duration to actually have always new user coming through the simulation.
And you can also do a breaking test by taking a number of user that is very high and see how high you can go.
So, once we are done, we can use scalar build tool to actually run Gettling.
So, you've got a very nice execution report on the CLI.
And once you're doing that may be a mistake from the guy writing the slide.
So, imagine that it's not a J but a I.
Sorry, this one hopefully is I map.
So, that's the execution report.
The top graphic is pure propaganda, that's not the interesting part.
The interesting part is in here.
And hopefully you are able to unfold the actual section.
So, what is interesting is that you've got meantime, meantime sucks.
You have interesting things like 99%, which gives a better representation of the worst case that your user will be exposed to.
You can see also the number of operations that is done per minute, a per second, sorry.
And the percentage of failure that you have.
And basically, after that, we also are able to see here that's the injection profile.
How much user were active on the system over time.
And here is the number of requests per seconds that the system succeeded to handle.
And above you see the response time.
So, when you're Java based, it's actually quite interesting.
You can see, for example, your garbage collections.
What is also interesting is that you can get that exact same two graphs for each of the I map command above and see the revolution of the time.
So, I'm almost so basically you can use it.
It's HGPL V3.
It may not be a modern version of getting V3 to go contribute a great if you want.
We also have some additional related resources, but that is not released as a separated DSL.
That is the J map simulation.
I'm sorry.
And we also have SMTP tool that sucks for doing the same thing because when you have a hammer by every screw looks like a nail.
We also had, but it's not open source, it's for private deployment validation.
A tool for provisioning data.
So, creating tons of folder of emails on to with nice distribution of email sizes for the virtual user that we've been seeing before.
And of course, it complements all of Apache Jamestack performance toolbox.
That's it.
Do you have questions?
Thank you.
Correct.
And Jameter is an Apache project.
I would say, Jameter from what we've been looking at first did not have built in I map connector.
Second, I think, don't have, I've, we've not been seeing that a domain specific language that was that well developed on top of Jameter.
And actually the DSL was very important to us so that we could replicate any client behavior easily in the performance test.
Does it answer a question?
And there's another point which is architecture which is if I'm correct, I'm not an expert here, but Jameter is actually using a fret pool to simulate things.
It's way less efficient.
So, we were able to simulate easily 20 to 50,000 people with a single machine, which Jameter to my knowledge is not able to do.
So, I think it was a cute.
I didn't get fully if Jameter part of the test with this functional or not.
If it would be, did you ever run it in a comparative fashion?
Can you give any insights about it?
It's the same server shootout.
Okay.
So, the Jameter test shoot is fully functional.
It's way easier to write because Jamep is HTTP and JSON based.
So, basically, the code is way less complicated than for IMAP and to our for us.
It's directly part of our Gatling tooling and not extracted as a separated library.
If there's interest in that, I think we would be happy to extract it.
And for the comparative section.
So, we, of course, measure better performance with Jamep.
However, this tool, don't directly reflect direct user and direct usage.
We try to fit to something that is representative.
But part of the design choice is that it's willing.
We don't invest extensive amount of resources to try to replicate the real world.
It's this versability and complexity and so on and so on.
That we are reduction of compute is at least factor two for us.
And our specific implementation of JMAP behind it.
Thank you very much.