You guys can help me, everyone.
Should I be louder?
All good.
Hello everyone, I'm Haris and this is Jintu and we are Colonel Mint Ares for RNBD Anti-RTRS,
gun modules that work in the storage and RDMA domain together and we work for INS Cloud
Germany, we work from the Berlin office and today we will be presenting one of the Colonel
modules that I just mentioned that RTRS is a reliable high speed transport library
over RDMA.
So everyone would have heard about RDMA, that maybe the buzzword for nowadays but it's
a very important technology that is being used, it's like one of the biggest standards
being used in the high performance domain, AI domain, all the machine learning things that
are being developed right now, mostly based on the RDMA technology and for good reason because
it is low latency, high throughput, it uses very less CPU and you can basically pump a large
amount of data through the network, zero copy, you can do zero copy, it doesn't have to
copy data, it can just map the buffers that you want to send through this 3DMA.
That's all well and good, it's fast, it's nice but it is a little difficult to work with
because there's a lot of things that you need to do for starters, for example you have to start
working with protection domains, when you want to establish the sessions, you might have
to create Q pairs, you have to create completion Qs, shared Qs if you want that kind of
completion technology and that's just the set of part right when you want to send data
when you send higher, you will have to do your own DMA, you have to manage memory regions
and that is the good part and then you have events that are happening in the network for
example, a disconnect event or an error event, you have to manage all those events through
this connection manager, so it's all a little difficult to handle and you obviously have
to react to those network events and do your own managing reconnection error parts and
the code is also sensitive, you see it's designed in a way to work with a G list but that
shouldn't stop anyone because you can basically just map a buffer whatever you want to
send your header or your IO in an SG list and send it across and so using RTRs you see
that all the complex things that you want to avoid while using RTR may are just hidden
below and every RTR is basically does everything for you, it does the connection management,
it does the completion gear management, it manages memory regions for you and all the
DMA mappings and RDMA events that I was speaking about right so it's a RTRs is a
basically very simple client server architecture and you have the module on the client side where
you have to do an open just like a socket, you have module on the server side also which also
you have to do an open to start listening for your connection and basically that's it if you
do an open on the server side and then an open on client side giving the IP address or whatever
the GID for example an RDMA, you'll have a connection in between and then you can start sending
files, RTRs allows you to have multiple paths for multi-pathing and you can basically design
your architecture in a way that different paths goes through different network links so you have
redundancy in network also and they the paths internally has connections that are perceived
you so you can have an IRIS you pinning for efficient performance and data transfer.
So as always as always saying the main highest of the logical relationship is the session
which you can establish giving a unique name and internally it has its own unique in a UUID
and in the session you can have multiple paths which I just mentioned through different
network ports and network paths if you want and each path will then have a Q pair which is linked
to perceived you so every CPU for every CPU on the client side that you create a connection
so it basically utilizes all your resources and also improves your completion better
so generally this is how the headers laid out you have the RTRs message but you don't have to
worry about that what you have to send is an optional header if you want for example you have
modules on both sides that you want to communicate what you're sending with it's an IO message
there's a management message so you can have your own header and you can have your own data
you send header through a vector and you send the data through an SG list which you can create
out of a buffer or anything and this is the general handshake protocol you have the
not very important because it's it's all being handled internally it's a connection to
question response and then in voice exchange in between thing yes so one more important thing is
RTRs internally has its own heartbeat mechanism which lets you make sure that the connection is on
all the time even when it's you're not using it and when it's supposed your IO fails what it's
going to do is if you have multiple parts internally it's going to fail over to the other part so
you won't even know that something has happened is it's going to whatever in flight's IO you had on
the other part it's going to stop sees that there's a path as prepared it goes to fails over to
the other part and it also initiates a reconnect mechanism for the failed path if you had an
network disruption it comes back your path would be basically established again successfully
yeah in the second part I will take over for the second part so
RTRs support different path path policy so we have round robbing so just circle around
from the process available and we also have mini flight which is RTRs track feels the active
request and we'll pick that pass for next IO and the server is related to it so it
uses internally this heartbeat mechanism to tracking the minimal latency for the
fact is passed so when there's so happy to detect there's a failure on the
pass it will automatically pick the healthy pass so it'll feel well as filled out directly to
the healthy pass and so customer won't notice there's a failure
and when the pass is recovered it will automatically reconnect and so it gets to
so multiplas both multiplas works again yeah we have an important
configuration knob which is a tradeoff between performance and security caught always invalidate
so it means for every IO the server style video invalidate the buffer and so
and it's indicated by a message caught RTS message or RK response and it defaults as
on so it's more safer and this has its own performance default but you finish your
transfer you just call this RTS client close to close that session
here from the first side is mostly similar just different structure so there's RDMA
event callback and there's a link event callback so so in RDMA feel there's different
events generated so just you need to handle them RTS does handle it transparently inside
so there's some when you so you first need to define the structure and then call RTS
server open call and you will get back this RTS context for for handle the data transfer
yeah link event is simply handle the event generated from the underlying low-level
data event and you'll finish the call you call RTS server close to close the session
yeah we get to this be different into the IO pass so you're basically you need to define the message
and defines the confirmation callback this is message our confirmation callback and we use
send the request it is because we reserve the resource on the server side memory resource to hold
this IO so you need to get a permit so don't overload buffer and you define
the type of connection there's different type called IO connection or admin connection so the
main is mostly for the amendment commands and always general IO it also have different mode you can
wait for the running mode and false all-down servers in our data center and you can also see
the performance graph is skills pretty well the number of jobs increase or number of device increase
is mostly line nearly but when there's multi-numa architecture there's slope not so deep
yeah so basically the KU's case for RTS is managing give a general
internal library to do RTS may do RTS may and you can also use it for AI machine learning
scenario to transfer data and because mostly generic so we currently use for
for R&BD there's a set of module R&BD and it can be reused for other modules too
we just call out others to get familiar with the code in can just run and test also reused the
code as needed I think almost done and if there's any question you can ask here's our context
in the email and thank you
hi thanks for the talk I have a quick question is RTRS available to user space through
you know you verbs interface or is it on the kernel level feature
we are it's internal right now we do plan to have a interface that basically either using
either you know something or maybe a user space blocked device through you block
to start using RTRS and kernel but right now it's not available to use a space and before
for break do you rely on an infinite button or something else it doesn't matter you can use
infinite button you can use rocky it doesn't matter whether you're using
nalenox or a broad gone it just needs to be available to use through the I the words
okay thank you any other questions anyone okay
I may be I miss something does this library designed to export block device at all I can
share read or write any block from this device yes so this was designed in
paired with another kernel module called RNBD and you would see that so basically when you use
RNBD it internally uses RTRS and through that you can export block devices through RDMA it's
very similar to NVM NVM or the differences that if you so for app form and test it shows that it
performs better than NVM or web basically so but if that other module is also available in the
kernel if you want to export block devices across RDMA network you can try that it's RNBD RTRS
okay thank you
okay thank you very much round of applause