Thank you all to be here today.
So my name is Paul O'Guenner-Canaison.
Today I'm speaking about the Linux Foundation
Energy C-Pass project.
And most specially about the SVT-Trace tools, which
are latent scenarios, this tools that we develop during
20, 24-year in the context of virtualized networking platforms.
So first I'm working at the company, which is called
the Stafford family news.
So we are company, which are experts in the Libre and the
Free and Open Source Technologies.
And we are based in Canada and Europe.
And we are mainly working on a project which is called
C-Pass with is a project of the Linux Foundation
Energy.
So quickly, what is the C-Pass project?
We will not take too much time in it.
So this is a very, very, very oversimplified schema
about all the electric energy is produced,
transported and distributed to your home.
And as you can see, there is a place
that we call a substation.
And the main purpose of substation is to make some monitoring
of the electrical energy and some other stuff.
And it's currently, and since the decades,
lots of hardware devices and there is transition
of into this substation, into digital substation.
And more and more digital devices.
And for this, we want to implement these virtual devices
into virtualization platform.
And this is where C-Pass operates.
C-Pass is mainly an open source appaivizer
and designed for the digital substation.
And the main purpose of C-Pass is to hosting virtualization
and automation and protection application
with real-time constraints.
So this is the schema of the semantic technologies
we are using.
It's the main technologies all about a test of clustering,
features, and a lot of virtualization stuff.
If you're interested about a more deeper presentation
of the C-Pass project, we are already
on some presentation in the past few years
in the energy dev room.
So if you're interested, you can take a look
at the past few presentation.
So let's go now in the subject, which
is about the need of low latency communication
in a substation.
So to do a little bit of context in substation,
as low latency communication are a priority.
Because, as you can see, substation,
they are safety and reliability is critical.
And for example, during a storm, there
is a trees are formed on a electrical power line.
The current must be cut off very, very, very quick.
And as the substation is monitoring
this electrical power lines, we have a real-time constraints
and the virtualized protection application must react
as soon as possible.
So to communicate in a substation, there
is a standard, which is called IECS 61850,
which defines a lot of protocols.
And one of them is called SV for some pair values protocol.
And it's just basically an L2 protocol,
which can be identified with a photo field,
it's an internet photo field.
And it follows a publisher and a subscriber schema.
The publisher machine is sending periodically
the sample value packets at a fixed pace, which
depends of the frequency of your electrical network.
And the subscriber beside this packet
and then decodes in and then an application
do whatever she wants with the data in this easy.
So if we take a look of all those these in the past architecture,
we have a publisher machine.
It sends the SV to a C-Pass IPAvisor.
And then we have the subscribers, which are virtual machines.
And all of them are receiving the same sample values
and receive decodes and et cetera.
So what we need is to ensure we are very, very low latency,
latency communication between the emission of the publisher
machine to the subscriber machine.
And when I said low, it's the lowest possible.
So what I wanted to do is to first make
a lab of the transit of this SV from the permission
to the publisher machine to the reception
in the subscriber.
So as you see here, first the publisher machine is sending
the sample value from the networking interface.
It travels through the network.
And then it's received at the T1,
in T2, the IPAvisor network interface.
And then in T2, the IPAvisor, this sample values
is transmitted through an open-to-switch bridge
to all the virtual machines.
And so we have, at the T2, the reception of the sample value
inside the network interface of each virtual machine.
And finally, inside the subscriber, inside the virtual machine,
the sample value is transmitted from the network interface
to a user application, a protection application.
So as you see here, we have a lot of latency
that we identified with T, IPAvisor, T2, 4T subscriber.
And what we want to know is where the latency is
are coming from.
And we want to track where exactly there is
problems.
So for the next examples, I have taken the hypothesis
of a perfectly-published machine, which is sending
at a perfect paste the sample values.
And we will take here the track of the latency inside the IPAvisor
only.
So where the latency of this SV are coming from?
And all can we do that.
So the first step to do this is to identify inside the kernel,
where what is kernel stack, which is a security
by the kernel, when a sample value is transmitted.
What our kernel is doing when it receives a packet.
So to this, we use a tool, which is called P, the value
AU, which stands for packets, where are you.
And the main purpose of this tool is to show us
every kernel function which has been called
when a specific packet we have filtered
around on the nation.
As you see here, there are three main process,
the NICI-I-U-Q, the open-viswitch
diamond.
And finally, the Q-E-U process, and most
specialties, a specific thread of the Q-E-U
process, which is called the BIOS driver, which
it main purpose is to send the data to the virtual machines.
And what is interesting here is that we have identified
a variable of the sample value on the IPAvisor
and finally, a departure of the sample value.
And we have a departure, an arrival, an arrival,
a departure, and we want to know,
or can we measure the latency between these two points.
And so to do this, we use another tool.
That is called BPF-TROS, maybe you have here
about it, because there is a lot of, for the end
confluence, talk about it.
BPF-TROS is a arrival or can enter a cell,
based on the EPPF.
And the main purpose is that you have scripting language.
And you can make some custom action based on the
what can a tracing points are called.
And here, so the two tracing, we are two props,
we have identified previously.
SKB-TROS and the consumer SKB, which are K-TROS.
So when with BPF-TROS, when this function
are executed, we can execute some custom actions.
So for example, we can in BPF-TROS script,
we call a timestamp, each time of the SKB-TROS function
is called, and we call a second timestamp
when the consumer SKB function is called,
and then between these two points,
we can measure latency.
So the first step is to wrap this BPF-TROS script
into Python wrapper, and this is where the SKB-TROS
tool operates.
We developed it with two main features.
The one is a live feature.
What you see on the left, and it's allowed to show us
in a direct, an instant gram with the real time latency
of the transistor of our sample values
with some statistic about the average maximum latency,
actually, et cetera.
And the second one is a recall.
And in the log file, we record all of the latency,
we record it, and we use it for long test
or post-process analysis.
So let's do some practical case,
and let's see if we can track some latency,
sources, and maybe fix them.
So first, we do a nominal case.
So we are investigating latency between the T1
and the T2 point, and to see if we can optimize the situation here.
So what you see here is that we did a four-hour test
without any optimization in the IPvizer
or inside the virtual machine, and what you see,
is not very good results.
We have a lot of latency that are largely above 200 microseconds,
and in a real-time context, it's really, really bad.
So what we did is some optimization.
We isolate the KVM calls of our virtual machines.
We isolate the T1 main process,
and we have very better results.
We have less sample values that are above 200 microseconds.
But we still see what I call a glispice that I have identified.
Some of them are at the beginning,
and are a logical behavior.
The first one is open the switch that when it
start to translate the sample value for whatever reason,
it's used to translate inside the user space,
and then it can load the kernel driver,
and all of these are the rest of the SVR sending through the kernel space.
But the last two spikes we don't know
where they are coming from.
So this is here where we used LTTNG,
because it's a real-time programming.
It's becoming too complex to investigate just with
the graph.
And so what's you see here?
The first LTTNG trace shows the execution of the pipeline of some per values.
So first we see that it arrives in the IP address IRQ,
and it passed at the end with the Vios, PMVios driver,
and then it is transmitted to the virtual machines.
But we see here we have a bigger latency of 100 microseconds,
and we see here that there is a green process,
the QMU system process, and we don't know what is doing here.
And we see very often this problem,
and this is what is creating this big spikes.
So the solution is quite easy.
It's just a priority fix.
You just have to make the QMU system priority much lower
than the Vios to the Vios to thread the Vios to the QMU thread.
And the good result as a partition below.
We have 40 microseconds between the reception of the IP address
and then it turns it to the virtual machine.
So thank you for your attention,
and if you're interested about SVTRAS project,
or maybe about the CPass project,
you can take your look at the GitHub page.
Thank you.
In their example, you used an OVS bridge,
but is there any reason this couldn't work with a simple Linux
purchase as well, or does SVTRAS support Linux purchase as well?
So you're asking about the OVS bridge.
And if you could use this with regular Linux
purchase as well, regular Linux purchase, just.
And we didn't have an opportunity
time to test all the technologies.
We only tested the open-list bridge.
We did some tests with PCI password,
but it's some cheat.
And it's not quite the same because PCI password
you are reserving one Ethernet port for the virtual machine.
And the reason where very good, but it's PCI password
and not software turns it.
But we are planning to try with other logistics technologies
and also with other things like maybe
or express that a path or maybe a deep decay.
About OVS, that's called an up call.
This is how OVS is working.
So what you're seeing as a spike is an up call.
And you don't use OVS on a real-time system pass.
You don't do that.
It's not made for it.
But OK.
Did you use huge pages and CPU dedicated CPU for the VM as well?
Yes.
Huge pages, we don't need because we don't have
a lot of usage of memory.
We are not in case we are, for example, using DPDK.
But CPU allocation will use its KVM calls are isolated.
The KVM call ends a premiuming process
is isolated too on a reserved real-time call.
So if you see some TLB shoot down, you should use huge pages.
And that's going to decrease your latency as well.
But maybe this is something that you cannot measure,
because your latency overall is still pretty high
from an absolute number.
Because we have this stack running since here.
With real-time end-to-end, and we have lower latency
that we measure with cyclic tests and other tools.
And so I think you have almost the right ingredients,
but you need a little of tuning and just don't use OVS.
Or you make sure you don't have a call.
This is something that you can do, but you need to know how to do it.
Yes.
But we use open this switch, because it's mainly the only technologies
that results us a lot of our needs.
So there is still a lot of all the technologies
that maybe we could use.
But this is why we have planned to test
other things that only open this switch.
Anyone else?
Thank you very much.
Thank you.