Good afternoon everyone, this is Jose Castillo, I'm here with my colleague Raul, we are both
performance engineers in Red Hat, we work with in the open shift and Kubernetes ecosystem
performance teams and we are going to discuss today a performance comparison and evaluation
of different Kubernetes, multi-cluster networking solutions, basically we compare the
scaper, some mariners anistio in terms of throughput, latency, resource consumption, and power
and other metrics. Thanks Raul, this was part of a research effort that we did last week
with some colleagues, psi, which is now on Nvidia and André, which is a professor in in
in Italy, don't mistake. Next slide please. So this is a number of what we are going to discuss
today, first of all we are going to discuss why we need multi-cluster solutions, why multi-cluster
solutions are relevant today versus like huge single-cluster instances. Then we will discuss each
of the solutions that we analyzed, some mariners cover anistio, each of them are very different
in a lot of aspects, the network architecture, the network technology that they use to transfer
the data, not only that, the use cases that they cover, the end user for each of the solutions
is different, so I think it is important to understand the difference between, between those and just
to justify why we chose those solutions. Then we will describe the test bed that we use for the
comparison, it was a vermetal enterprise, a couple of vermetal enterprise environments,
some of the tooling, a lot of tooling for data planes, some for throughput, some for latency,
and also the tooling that we use for the power and energy metrics. Finally we will discuss the
results and we will commence some future work improvements that we have on mind.
So first question, why multi-cluster, why we use several Kubernetes clusters, and this is a trend right
now. So besides a couple of use cases that really need huge clusters, like I'm thinking like
training, A, A training and stuff like that, in general there's a trend where we see that the
nodes of the governance clusters are usually distributed within smaller clusters. So there's a bigger
number of clusters with small numbers. Guides that for several reasons, we may need several
clusters for high availability reasons, we may need several clusters because we want to implement
a disaster recovery solution, sometimes is because we want for quality of service reasons, we want
to deliver data closer to the end user. So there's plenty of reasons why we would like to have
several Kubernetes clusters to deploy applications. On top of that, there's hybrid cloud,
sometimes we need to have or we prefer to have some applications running on premises, on the data center,
and we have other environments and other applications that are better located in cloud environments.
So multi-cluster is is really important nowadays. There's the management question of multi-cluster,
so there's a lot of solutions there, open cluster management, advanced cluster management and others,
but today we'll be focusing on the data playing aspect, and on the networking aspect of the
multi-cluster framework. And this is the first solution that we're going to be analyzing,
it's called Samariner. Samariner creates an IPSEC. It's also super wide guard, but IPSEC
is the default standard for creating this connectivity between the clusters. One of the workers,
no, the fish clusters is marked as the guide window, so the east, west, traffic in the left-cluster
in Kubernetes 1, goes to the guide window, and the guide window, using kernel networking and the IPSEC
module, establishes a tunnel with the other cluster. So that's how so Mariner works.
Of the different solutions that we are going to analyze, Samariner is the only one that has
some Java centralized architecture. There's one cluster which is the hub, and the other clusters
are considered the spokes. And it requires some previews, the use case for Samariner, then user is
the cluster administrator. It basically is a solution that connects clusters. The second solution
that we're going to analyze is called a SCAPER. SCAPER is very different instead of connecting
clusters, let's say the other way, it connects different namespaces. The end user of SCAPER is
an developer, so you don't need to be an administrator of the cluster to be able to deploy a SCAPER.
If you have user, tenant, user, level access to a couple of clusters, you are able to connect
to those namespaces using NTLS. Basically SCAPER creates a pod, which is a router pod,
and all the traffic goes through the gateway pod in user networking kind of traffic.
It's also interesting to note that the SCAPER cluster can be scaled out
or is entirely, so it's a really cloud native solution and framework.
Finally, we analyzed the performance of STU. It's just very now because it's a service
mesh solution, very popular within one cluster, but it can also be used to connect several clusters.
There are several modes of functioning of STU. Historically, STU had this
envoy sidecar, so for every pod you have this sidecar running envoy and that's where the
gateway magic happened and all the advanced level, the seven features. There's another way
more of a operation, the downside of this envoy approach is that this very resource
consumes a lot of resources. Basically for each pod, you have an extra container, right?
There's a newer solution, which is called Lambient mode, in which you only have like one gateway
per each of the workers. Unfortunately, this new solution called Lambient mode doesn't work with
multi-cluster, so we were forced somehow to use the classical envoy solution for the testing.
The same way that Samariner connects clusters and SCAPER connects namespaces
is the basically connect services within clusters, and it's also a solution that can be deployed
at a tenant level. There are some trust that you need to establish within the clusters,
in the case of STU to be able to connect the services across the cloud continuum in the
form of certificates and trust issues, trust change. Here we see a comparison between the three solutions.
Basically, Samariner is the only solution that's more centralized that follows a
Habspoke architecture. It's the only level three solution we use the CIPSEC or
wide-walling kernel working mode. We're going to see after words when Raul discusses the results
that there are some limitations to the IPSEC kernel implementation, and this translates in
in some bandwidth limitations as well. What each of the solutions connects, as I said,
Samariner connects different clusters, and it needs to be done by the cluster, I mean,
um, Istio connects services and SCAPER connects different namespaces.
Then user personas also different for SCAPER than user personas, the developer or the,
the tenant user of the clusters, for Samariner usual is the cluster admin,
and in Istio we have both, both kind of personas.
Security features are a little bit different, both SCAPER and Istio, leverage MTSS,
and Submariner uses IPSEC.
It's also relevant to note that the SCAPER and Istio are solutions that are independent of the container
network interface, so it doesn't matter if you're using OVN, SILium or whatever, but Samariner is
dependent on the C&I, and it supports a subset of the, of the C&I implementations.
All of them obviously support multi-cloud that does the main scenarios, multi-cloud and hybrid scenarios.
All of them has the same kind of Apache license, and the only project that's not C&CF
hosted is SCARENTLY SCAPER, and this is the test with that we use for the testing.
We had a couple of vermetal clusters, enterprise servers with a couple of number nodes.
We have obviously the control plane, three masters, and we isolated all the gateway components of
each of the solutions in one particular node, in one particular work in each of the clusters.
We did that to make sure that we were able to measure the overhead of each of the solutions.
So we wanted to know the price that we need to pay in terms of CPU memory and energy
for connecting the clusters both for Sumariner, for SCAPI and for IPSEC.
We also wanted to make sure that dormitory and not approach for the testing was consistent.
So we applied this performance profile in the cluster, the idea was to
to reserve some CPUs for the operational system, to have some CPUs also for the workloads,
for the client and server pods.
We wanted to make sure that we weren't interrupted from the operational system,
so we wanted to make sure that the pods that run the actual workload were not
interrupted at any given moment.
It's also important for us that the workloads run within the same node, as you guys know,
there's a penalty that you pay if the interface is connected to one.
The PCI address of the interfaces in one node, and you have cross-numan node traffic,
so that's something that we also wanted to avoid for this particular testing.
This is the performance profile, and then from the pod perspective,
we just pointed the workloads to this performance profile,
and we disable, we didn't get any interruptions from the operational system.
That's to make sure that the workloads were completed,
and the results were reproducible.
And I'm going to let Raul discuss some of the tooling and the results of the testing.
Right, thank you, Jose.
You hear me well?
Okay, so, I'm going to speak about a little bit about the tooling we use
for layer for testing, let's say, TCP throughput in latency.
We use Uperf, there are many other alternatives like NetPerf or Iperf,
but for the particular testing we use Uperf because we were more or less used to the
it results, so I consider it stable tool, and which returns quite deterministic results.
This is something we are looking for.
For layer for layer 7 testing, HTTP testing, we use work tool,
kind of, or those who don't know what it is, is HTTP loader tester,
but with the difference that with the original work tool, work tool has an adoption,
to what the option has the nuance of generating a constant fixed amount of a fixed rate of HTTP
request per second, which is something necessary to measure latency safely.
And respect to energy monitoring, we use a Kypler,
which is going to be a sufficient power level exporter, which we basically saw,
operator, that deploy is a diamond set with a magic, we'll talk about it later.
And we also use the BCM of the BurmETal servers,
that is basically, we were taking a look at the BCM,
based on controller, BFC, BCM, the I-DRAC, and then this case, we were using Dell's servers,
and we were taking a look at the graphs in the BCM to ensure that the results exported by
KERPER were aligned with them. Also, the assets of the all the testing can be found in that repository,
with respect to HUPERF, as I said, we use it for TCP support and TCP latency,
for TCP support to use this configuration, which we see currently as the XML that describes
a configuration to send as many write operations as possible during an exact one minute.
So, quite simple, it's a placeholder environment by the border,
that we have to need to be replaced by the service AP in this case. It's important to know that
in the cluster communication happens at a service level,
especially because a service level, exactly, you can go directly to a pod
from that different cluster you had to go to a service in this case.
We also have another placeholder for size, because we tested multiple packet sizes for the throughput
test, and respect to TCP latency, we use the well-known TCP request latency profile,
which we see currently get a statin C8 measuring the elevation of a small read write operations
during one minute. Same as I said before, we have some environmental variables that need to be set
up front in order to configure the test properly. Poor monitoring capillary, so poor monitoring
is basically the operator that replace a demon set, and demon set that replace say this of pods in
every note that collects real time CPU board consumption from notes. Those poor consumption
numbers can be collected from the RAPL running, it's probably made from Intel,
and I think the AMD has another alternative with a different name. I don't remember what was the
name, but basically it's a high precision monitoring of CPU packets and RAM energy use it,
and redfish, which is a real time platform power usage by the AVCM controller. In this case,
redfish was our choice, because we wanted to measure not only the CPU and the memory power consumption,
but also the full stack memory consumption, including network devices, cooling funds,
police supply overhead, and any other device that could be affected by this performance test.
I'm going to speak a little bit about the results, so respective throughput, we can find out that
you can find out that the highest throughput was found in Istio. We saw that Scott Per was limited
at 8 GB per second, and some RNA comes with a low throughput. In this case, we read it around
three GB per second, it was the ribet because of an IPSEC kernel implementation limitation,
that limits the IPSEC kernel established for IPSEC kernel that some RNA uses,
it's only processing in a single core fashion. There are active efforts to improve that logic
in the kernel side, there's even an RFC with a proposal to improve that logic, but as far as I know,
it has even been any significant progress on this matter yet. The red lines are represented in the
baseline, which is in this case. Our regular Kubernetes cluster IP service, so as you can see,
there's a relevant penalty for using multi-cluster networking, but ATDB latency. The latency
is well measured with the application a lot to work to, in this case, we configured it with
to send 10,000 requests per second across 100 simultaneous connections, and we've seen two threads,
plus sending a plus requesting a 1KB body size, 1KB static file from the ATDB server.
In this case, I work with you in a regular NGNX pods, replicas, Serbian static files.
Well, we can see an significant latency overheading esteem with the cluster,
much higher than the previous and the other solutions, so at this time, we can conclude that
with this we can conclude that the esteem could be a good solution to achieve a good throughput,
but at a high latency. With respect to resource consumption overhead, we are measuring here,
we are plotting here, the representing here, the radio between the throughput and,
oh, sorry, yes, this is representing absolute numbers of the CPU consumption,
and you can see the submariners, the most resource efficient, followed by esteem,
and we can see that we saw that as a cover, consumers, the most number of CPUs at all packet size
and threads combinations, but to say, yes, and this was the one I was talking about before,
that basically represents the radio of resource consumption compared with the throughput achieved
with all solutions, and we can see that the scalper and submariners do much better than the
scalper in terms of throughput per core, especially esteem in this case is clearly winning the other
two solutions, power consumption, these are the numbers we saw from, that were returned in
from issues by an exposed by, by Kepler and the Kepler pods, and we saw that the higher CPU
usage is basically correlated with a higher power consumption, which is predictable, but this
is something we wanted to know as well, and this is a nice graph that basically summarizes all the
investigation with all the research we did, the closer is that the bar points to the borders means
the better, I mean, for example, we can know that for example, some mariner is the best at the
latency and the number of truckshenshells and CPUs it, and on the other side we can note that
esteem is the best in terms of throughput, so depends on the requirements or your infrastructure,
or solution, you will pick, you should pick one of these options, the most balanced
at one seems to be scalper, but for example, if you are, you really need a high throughput,
I will go with esteem, and in case of getting a required latency, that means that we will need to
go with some mariner, so it's a matter of finding the balance, evaluating what are your needs
for your application and infrastructure, and try one of these solutions, and that was pretty
bad, it's, yep, and the future work, yes we have plans to run, to try to run real work benchmarks,
involving measures in misadding databases and applications, based on microservices, also explore
the capabilities of the control plane, like service discovery times, service failure time,
those KPIs are not, are not, were not measured, that did in this research, but
are really good KPIs as well, depending on the workload you have plans to run,
and yes, scale the numbers on scale the number of clusters in their tests, so depending on
the number of clusters, the solution will probably be affected, the results of the solution will
probably affect it, so that's something we will need to test going forward, and well, this is
include ambient mesh mode in the rest-test metric space, probably that has that some
mistake because as far as you know, ambient mode doesn't support multi-cluster get, it does, okay,
that's good, not the, awesome, that's great, great news, thank you, that is pretty much it,
you have any questions, we have some minutes,
yep
yes, the question was about the word will be the use case in considering that the latency numbers
are much higher than the baseline, well in my case it depends on the workload, I mean,
not every workload is going to be a database basis, it's not going to be to communicate with
a database, possible you need, you know, not every workload requires a very super low latency,
not every workload is such a good workload, so it really depends, I mean, there are,
it depends on the application you want to run, I guess,
just to complement yes, for point, and that's why we want to test with real-world applications,
and we mentioned that as a future work, I mean, we just basically were measuring network
characteristics, by itself, we doesn't make a lot of sense,
especially because these two is a solution for multi-class, for micro services architectures, so that's
100% the next step to have like realistic applications with databases and stuff like that and see
how they behave, also scaling the control plane, the number of workers, there's also these metrics
that we're discussing that I think is very important like from where you create the service in
one of the clusters, how long it takes to be available in the other cluster, so I think there's a lot
of space to explore new metrics, and that was not our intention at all, yes.
That's an excellent question, so we didn't apply any space, I guess, the question was if we applied any
tuning to the solutions, so we did not, because we didn't thought it was fair, so if we tune
one of the solutions and not the others, I mean, we weren't like a very complex matrix to test all
the possible solutions, so there are all like upstream solutions, so we did use upstream, the
open shift, the downstream for the platform, but there were all like the upstream
deployments without out of the box configuration, so we didn't test any specific tuning in
each of them, interesting, what does it say about it?
Yeah, it actually depends on the solution, right, because for instance, so the question is,
how do we plan, how do we do capacity planning about the resource consumption of each of the
solutions, I guess, and the answer to the depends on the solution, for instance, should mariner
allows you to have this worker label as a gateway, so you can have for instance,
in AWS, a different flavor for that worker, and you can be very intentional about the
workloads that you schedule or you avoid scheduling on that cluster, you can even reach an extreme
where you have this worker research just for gateway functions, in the case of a scalpel, it's
just a pod, so you need to be careful where the router is, is a schedule, and in the case of
Istio Raul, if you can help me there, I don't know if you have like an solution to
Istio the same, Istio has a gateway pod that they gave a pod, what's running on top of them,
what we call the gateway node, in the case of the cluster that was receiving the traffic,
because the traffic goes out directly from the worker node, and then reaches out the cluster
gateway node, note. Yeah, and that's why we tried to measure in one of these slides, the
the raw overhead of each of the solutions, so we want to be able to do some capacity planning,
like as a cluster administrator, how much resources does it cost me to connect these clusters
using each of the solutions?
Okay, and this single server is not like a cluster, right?
Yeah, I guess this was not covered as part of the study, this was like, this is a, like,
egress traffic scenario, we do cover those in other testing that we do, but that was not like the
point of this, this particular study. Yes, yes, for sure. We do egress and egress traffic a lot.
In this case, we were trying to focus on multi-clusters scenarios.
Yeah, apart from the results I shared with you, we also tested
pod to produce in host networks scenarios, is that answers your question?
Yeah, the results are not there, but
but yeah, there should be in the papers, right?
Yes, please.
That's an excellent question. The question again, is if any of the solution support high availability, right?
I can say on the top of my mind, Scapper for sure, because in Scapper, you can
scale horizontally the router pod.
I'm not sure about Scapper, honestly, I need to double check because you have this gateway node,
so I'm not sure if automatically recovers, like you can have several gateway nodes, but I'm not sure
if there's like an automatic failure detection or something like that on the top of my mind.
And with this, you're all, yeah, with this, yeah, you had a study, they move, with this takes care of
pushing the confirmation to the employee proxies, so that's, you can scale out the number of
replicas on having a cluster of ESTOD pods, so one of them should be active, and you can scale
out the number of gateways across multiple gateways nodes, so it shouldn't be any problem with
the respect to high availability.
Thank you very much, everyone.