Thanks, Eric's back at first time.
At 2020, Simon, he's also in the room and Christian, introduced the Eclipse Ice
Oric's project for the first time.
Today, it's my pleasure to introduce Ice Oric's tool and to elaborate on how it can
help you with relieving some pain.
You always have been middleware and I saw over and over again the last 20 years of
middleware development.
I'll start with a high level overview of what Eclipse Ice Oric's is.
And then I will deep dive into four specific pain points and how Ice Oric's
relieves them.
And I will end up with some community insights.
So Ice Oric says an open source project hosted by the Eclipse Foundation,
started in 2019.
And in 2020, we started Ice Oric's tool as the second generation.
And if we combine the whole contribution work of all the commuters over roughly ten
years, it's about more than 70 years of shared experience in developing high
performance in the process communication.
This is how it works on a high level.
Ice Oric's creates a memory, maps it into the processes.
We have send us in receivers.
This allows a sender to directly right into the shared memory and receive
or directly read from the shared memory.
Ice Oric then provides the whole communication infrastructure around its own entities
for sending receiving data, discovery, connecting them,
changing the data, which is actually just passing reference to the messages.
The whole management of memory around the numbers and
information and so on.
This allows you to pass a message without a single copy.
We also call this through zero copy communication.
Ice Oric's tool already surpasses the first generation, which we also call
Ice Oric's classic now.
It's written in Rust.
Runs already on a bunch of operating systems, Linux, Windows,
MegaOS, Q&AX.
It runs even better metal.
And besides the Rust APIs, we also have already language bindings for C,
C++, Python, and also an RC shop.
It has different messaging patterns.
The many to many published subscribe communication,
internet style, request response, and also a key value storage
in shared memory, which we call Blackboard.
And in addition, an event mechanism that allows you to
wake up a receiver that's waiting for data.
It's developed for a mission-critical systems in mind.
And that's no heap allocation during one time, no blocking calls,
and also does not have a dependency on the Rust under library.
Compared to the Ice Oric's classic version,
we have no more central demons of fully decentralized,
and the next step towards a mixed-criticality architecture.
Okay, so it's an IPC middleware, what else?
Yeah, it's solid foundation to avoid some of the hard pain points
that you often have because of the middleware.
The first and most obvious one is the pain you have with copies
and civilization inside your middleware.
So if you copy a message, this takes CPU cycles,
and sure the bigger the message, the more CPU cycles.
And advanced robotic systems can have gigabytes,
or even tens of gigabytes of data you have to pass between processes,
and spreads of execution.
Consequences that you have high latency and high CPU load.
Here on this figure, we see how the latency increases with the message size
for the IPC mechanisms that typically are used with Linux,
so message queue and unix domain so good.
Here a relive system where Ice Oric's is used,
it's autonomous driving based on an end-to-end AI model.
And we have six high resolution cameras,
with roughly five megabytes per frame, 20 frames per second.
But this gives you a camera data stream of already more than four gigabytes
that you have to pass through your inference process.
Now think about you have to copy them on the send-up side.
You have to copy them on the receiver side.
Then you may want to record the data so you have the next process and the next copies.
So it's quite easy to end up with 10 to 20 gigabytes per second,
just for this most subset of your system.
What Ice Oric's does is through zero copy IPC,
you have just passing references through the messages.
This gives you a constant sub-microsecond latency
independent of the message size.
Here we see the measurement on a Raspberry Pi,
which is not the most powerful hardware,
but even there our latency is below one microsecond.
If the message is as small, it's not that much of a difference,
but you have megabyte messages.
We are talking about sub-microsecond and not to be low milliseconds.
So it's a factor for more than one thousand faster in the end.
So this enables dozens of gigabyte per second IPC without CPU load.
And also a fast reaction time,
so it reduces the latency from sensing to acting.
And in the end, with this CPU time, you save.
You can use smaller, cheaper CPU for the same workload,
which can be crucial if you want to go embedded.
So if you have a pre-development on a rock PC,
and then have to go embedded,
this is really one important point to not end up with getting crazy.
On the other way around, you could also say,
the CPU time you save can be used for more workload.
The next point are additional threats inside your middleware.
So many middleware solutions come with background threats
for discovery, housekeeping, and so on.
But the question is, when do these threats break up?
And what are they doing?
And if you want to trigger real-time settings,
how can I configure these threats?
I'm going to nest the deep inside your middleware.
The worst case is that additional threats are involved in passing the message.
This means you have one or even more context features with every message to send.
One context feature is not that expensive,
but if you have thousands of messages per second,
you will end up in the thousands of additional context features per message.
Once the lens of all this, it's not deterministic,
and you have a high scheduling overhead.
Here's an example from the measurement.
We've got a little bit of ice-oys classic.
A user reported that they have some strange latency spikes
in a loaded real-time system.
And it was even a funny pattern,
and we started analyzing, and then realized
these latency spikes were coming from interference with
the monitoring thread we had that send the key
by live message over a unique domain circuit.
The easiest communication would be,
I have a node A sending a message to node B.
The most efficient execution would be,
I run these two nodes after narrowing the same threat of execution.
So first node A runs sends a message,
then node B runs consumes the message.
If you have additional threats involved in this message passing,
it's not clear if you will see this message.
It's a classical race condition here.
Ice-oys 2 has no internal threats,
so nothing happens behind the scenes,
and all the housekeeping we have to do is done
when we create or destroy the entities.
What is enabled is to gather with this zero-copy communication,
low latency with a load shitter,
and it's deterministic.
So this passing of messages along this threat of execution
will always have the message when the receiver is running,
and in the end we reduce the whole scheduling overhead,
which means lower CPU usage.
You on the figure you see what happened when we disabled this monitoring thread,
and we had the stable latency,
compared with the first version,
where we have the interference,
which is the thread in the middle.
The next pain point is in proper queuing inside your middleware.
There are middleware solutions,
especially around shell memory,
where you have no queuing at all,
which means a sender can outpace the receiver
and override messages that are not read.
If you have queues,
you have to face the challenge of a queue overflow.
Often what you do is then you block the sender,
wait for the receiver to consume the messages,
or you return an error and drop the new message
because you cannot push into a full queue.
Another thing is historical messages.
If you have a sender that's only sporadically sending some data,
like some state update,
you want to have receivers connecting to it
and getting the latest state.
But if the late shining receiver is started after the sender,
and you have no historical data,
it won't get a message to no data.
In this case,
you have to ensure that your startup order
ensures that all the receivers are first started
and then the sender,
which is kind of a restriction you have.
The consequences here is that you can have back pressure
from the receiver to the sender side,
or the other way around you have a receiver that's forced to keep pace,
with the sender to not lose any messages.
Here, an example where I have a low frequency,
low priority consumer that wants to read
always the latest for messages from my high frequency,
high priority producer.
That's actually a use case I had when I was working
on an ancient control system.
I had this low frequency, low priority consumer
that wanted to do some calculations on the full ancient cycle,
which had fear for your combustion,
and I had for every combustion the message,
and I wanted to have the full cycle
with all of the four latest messages.
As there was no queueing,
I had to do some work around and have some intermediate note
that starts catching these messages internally,
and then sends one message with all the four messages collected
from this high frequency sender.
That's kind of an ugly work around,
better as to have queues.
But if you have queues,
you could have a queue overflow.
This example here we have now a receiver,
which has a queue of four,
but the receiver gets delayed.
So your high frequency,
high priority publisher,
or wants to provide new data.
Depending on your use case,
you have different options,
often many of them are also not really good options.
So you could block the sender,
which we do not want to do in this case,
because I have a high priority sender,
and I don't want to get it disturbed by low frequency,
low priority consumer.
You could return an error,
and whatever shut down the whole system,
which we also do not want,
we don't want to have trouble with a low priority receiver here.
But we also don't want to let this receiver processing old data,
so it would be good to really have
four consecutive latest messages here.
What ISRICS do provides is receive a queues with
configurable queuesize,
and we have something that we call safely overflowing queues,
which is more or less just a ring buffer,
but it has the benefit that you can drop old messages,
that are unread and favour of new ones.
So in case of an overflow,
you can always provide the latest and messages
to the receiver.
In the end, we can configure this queue overflow behavior,
so we have the possibilities to either return an error,
and drop the new message.
We can block the sender, wait for the receiver,
or we can do this overflow that I just described,
and drop the oldest messages.
What we also provide is a history option on the sender side,
so we can, they are store the messages,
and if you have a late joining receiver,
we can deliver this message on the subscription you could say.
So this enables a decoupling of senders and receivers,
and also to have a very efficient memory queue
of only the messages that are relevant,
and you can drop messages that are no more of interest.
In this example here, we have a fusion node,
it has different senders with different frequencies.
You could decide that for your lighter sender,
you're only interested in the latest radius, so queue size one.
For your camera sender,
you want to have up to the four latest camera images,
but if they are getting older,
they are no more interesting, so you could overflow,
if you have a sender with a certain state,
you can subscribe and get the history,
or the latest message on the sender side still available,
or you could also decide that I don't want to miss a message,
I go for the maximum queue size,
and queue overflow is an error.
The last pain point are message callbacks,
or they say per message callbacks,
and this is typically the pattern that also rush is using,
and it makes you trouble because this is a hot coupling
between communication execution,
so your business logic on the receiver side
is forced to react on every message.
And if there are use cases,
but this is perfectly fine,
but if you want to process several messages
or receive us simultaneously,
you have to do some bookkeeping
and do some cashing beyond these callbacks
to come to the right set of messages you want to have.
It even gets worse if the message only lifts
during the callback duration,
which means you have to copy the content of the message
out of this callback for later processing.
Here we end up with many context features,
which is more scheduling overhead again,
and also more administrative effort on your side.
If we stay with this fusion example,
we have the different sendals with different frequencies.
Now your fusion algorithm could work like,
I always want to process the whole messages
from different sensors when there's a new lighter scan,
so that's my leading sensor.
Whenever I get a new lighter scan,
I want to get the messages from the other sendals,
and throw them in one algorithm that does the calculation.
This simple example, this would mean you have 16 individual callbacks
to just execute one fusion run.
I try to have it here depicted on the timeline in the figure,
and so it's a bit messy,
but if you have callbacks per message,
things get a bit messy.
What ISOH2 provides is totally decoupling of the notification,
and the related context switching from the act of messaging.
We have event as a separate pattern,
comes with notifiers, listeners, and wait sets,
notifier can send an event,
listener can wait on an event,
and with a wait set, you can wait on many events
in a single thread.
This thing can easily combine again
with the publishers, the primals, and clients,
and servers, and so on.
So it's easy to build a triggering publisher that always
wakes up a receiver when sending data,
or you can also have your per message callback again
if you really like it.
If you now bring this together,
the queueing and this event mechanism,
you can have very efficient execution strategies.
So in our fusion example,
I could say,
whenever a lighter message is sent,
I also have a notification,
and I wake up the fusion node.
All the other messages from all the other senders
are queued behind the scenes,
with the queue size, you configured,
and here you have one context switch,
if you want to run the fusion,
and the other queueing just happens behind the scenes.
It's also possible to use our event mechanism
for whatever notification you have,
so you could use it for timers,
for instance, and say,
I have a wait set that's waiting on new data,
or on the timer to expire.
So again, here,
let's conduct switching,
and more CPU time for you application.
Coming to some community insights,
I suppose it's already used in many different domains.
It's used in automotive,
from simulators up to automated driving,
all kinds of robots.
It's used for high frequency trading,
on industrial cameras,
we have different medical devices up to surgical robots,
drones,
so wherever you have a need for low latency
and high volume data transfer.
So you can think of ice ricks as being a low level,
basic software for share memory communication,
that can now be combined with other open source projects
to build something bigger.
So you can combine it with a network protocol
to have a full-blown communication state
with internal and external communication,
so for instance,
or GRPC, something like that.
You could also integrate it as an IPC layer
in a broader communication stack,
maybe based on a standard like DDRs,
which then defines a data model,
and an ideal and so on.
Ice ricks.
Thank you.
So I was wondering,
since you don't copy the data,
since I can only read it,
I cannot write it into that pointer.
Yeah.
So I need to make a copy myself.
If I want to modify it.
Ah, okay.
Yeah, that's this news case,
which we call the read-modify write pipeline.
Yeah, the current API that's not possible,
it could be a special option to also allow this.
No, this news case for who I have.
I think it's interesting if that is how it works.
But yeah, if you have kind of a special contract,
so sure, you would have to ensure that that's
not only receive off the data,
if you've started writing it and so on.
I was wondering about the blackboard mechanism,
so does that keep a history of all the messages
that have been submitted,
or is that just the latest messages?
Yeah, blackboards can be really good for reproducible
logging of when messages are sent.
Yeah.
The blackboard is actually one thing where we might have a copy.
So we would check if there's some contention on the blackboard,
if you can update,
and if someone is reading,
it could mean that you have another read again,
because someone was right.
I am an effort key by your story here.
Thanks.
Thank you.
If I'm doing zero copy IPC,
does it mean that I need to pre-educate my memory pool
for all of my messages?
So this is always what we do behind the scenes.
So if you configure your services,
you can configure how many,
for instance, publishers and subscribers out there.
You also configure your queue sizes,
and with these values,
we can kind of calculate the worst case amount of memory you need.
In the current version of icebergs too,
it's a bit of the maximum memory,
so it could be that it's the theoretical maximum,
but in practice you don't need it.
We will soon have the option for you to configure it,
individually.
If you want to have a safety system,
you can then configure it beforehand.
In the current,
I would say if we would run out of memory,
we'd just allocate more memory behind the scenes,
so things that are going on.
So that's also kind of smooth,
or then in icebergs,
one value really had to configure all the settings.
Do you have any industrial clients,
and the reason I'm asking is,
I wanted to see if you had ever integrated it
with ethercat or profanant for industrial use,
which is a highly deterministic environment?
No.
How was it?
So far we had no contact with business guys
who are kind of combining it with either cat
or whatever.
It's a lot less.
But for sure,
how we typically do this,
we have gateways,
and the gateway is grabbing data from shared memory,
and then sending it on the wire or whatever.
And you could then have whatever a strict cycle time
in which your gateway is running,
and then consumes,
without queueing the latest credit message
and sends it over etherneto.
Do you support dynamically sized messages,
or do you need to set up a maximum size,
or how does the work?
That's still the limitation.
So data lives in shared memory.
So if you have data that's based on heap allocation,
your heap is processed local, that won't work.
So if you have your own special versions of vectors
and so on, with your own allocator,
you could say, okay, I have an allocator that's allocating
each shared memory,
then you can also realize that.