Well, so I'm Felix, this is Vaseos.
We're going to talk about the work we've done in reframe with open source tools to support
for basic performance testing.
And so what do we do at the video?
We are both in the team called Applied Systems at Nvidia.
And what we do is we build internal supercomputers with the latest GPUs and CPUs
and meta-nox hardware.
And we have created internal clusters like EOS,
it was number nine on top 500 in November 2023.
And we do that to enable internal users so that they can run
that deep learning work load or the HPC workloads.
And as such, we also help people from our customers
to build their own cluster at a later point.
And our internal clusters run benchmarks,
such as well HPL and MLP of training,
if you're familiar with that.
And also, okay, of deep learning research,
such as training the megachron model on the cylinder computer,
that was the generation before EOS.
And so that's the kind of things we are doing in our team.
And we've been using reframe for a few years now
for basically, if you're familiar with reframe, it's very useful for the
performance validation of your cluster and regression testing.
So reframe is an open source project,
developed initially at CSES by some people around here.
And you express your test in a declarative manner,
you express your test in Python,
and you can express dependencies, constraints on your test,
and you define what you're expecting the output to be,
and you define performance targets for each test.
And reframe of generating the scripts,
launch starts using slum automatically,
you don't need to care about slum,
and it will gather the results,
execute everything concurrently as much as it can,
respect the dependencies,
and then you will get the results green,
passing, red, it's failing, obviously.
And yes, you open source, it's as great documentation,
you can take a look.
So platform and testing, but does it look like in reframes,
it's a very simple test where we run the stream benchmark,
and you declare methods, which are the decorator,
performance function, and you say,
I have a metric called copy bandwidth for stream.
So I'm going to execute stream.x,
the binary cost stream.x,
and I'm declaring a regax here,
and the first matching group,
I'm saying the first matching group,
basically cast that to float,
and that's going to be my copy bandwidth,
and same for triad bandwidth,
because stream as multiple benchmarks.
And then for the performance targets,
you define dictionary of dictionary,
and you say for the system,
for the default system,
you can say by system at different targets.
I want my copy bandwidth to be between 23,000 megabytes per second,
minus 10%, plus 13%, so you give percentage bounds
around this middle value.
And that's what you usually do for performance testing in reframe,
and when you execute reframe, it looks like this.
That's in a terminal that doesn't actually get lapsi-i.
It looks like this.
You have the name of the test.
reframe tells you what it's doing.
It's executing this.
As I said, okay, the green minute,
it's passes, and then you have this summary again.
And where reframe is very useful,
is that you have really the choice,
or you send the logs.
You can send the logs to a plastic search,
to a gray log,
and you can also configure exactly the format
where you output so here.
We're defining a format for the output,
which is CSV.
I mean, separated by pipe, but still CSV.
And your output file here,
by reframe after executing this,
will be, so timestamp,
the name of the test,
the metric we see,
like triad or red or right.
I had to read that some of that,
but that was a parallel value right here.
And that was the performance we were targeting,
and minus 3% plus 3% is the performance
bounds around the value we were targeting.
And how do we define those performance bounds
usually in reframe?
They have to be fixed a pair system,
but so basically we're going to write
on a few machines.
We're going to see what the performance looks like.
We're going to mean the deviation,
and we're going to see,
okay, how do I define my bounds?
And the problem we're facing,
if you bounds are too narrow,
you might get spreaders failures,
because you didn't really test the footwear
population initially,
and you're going to get for a positive,
and admins do not like that,
so as I will tell you,
can you please increase the bounds?
It's failing by the very, very small margin.
We need it to pass,
it's not a really performance problem.
If it's too large,
the problem,
if it's too large,
here I showed normal distribution,
because it's often actually normal distribution,
for performance results across the population,
if it's too large,
obviously like free sigma,
free circulation,
you can get,
obviously, a regression
that while it's still within your bounds,
and that happens,
you know,
actually quite a lot.
Thank you.
And so,
a common problem we,
of course,
we face is that we need to validate
our clusters,
the performance compared to
one week ago,
to weeks ago,
or if we install a new software,
we want to have confidence
that we have the same performance
as before,
otherwise users will report
problem,
as they're going to take us a lot of time,
because they're going to report
problems,
it's very complex application,
as we go through much harder
to find what's going on.
So, we need a very robust test suite,
and performance bounds are,
honestly, it's a great.
We find a lot of bugs with free frame
and performance bounds,
but for tests that have a wide,
variation,
you tend to have wide ranges,
and you're going to miss some regressions.
So,
people usually doing their own tools
on top of free frames,
they wear developing their own
pandascript,
like we did,
or they wear sending things to
splank,
or elastic search,
and build tools on top of this.
And we wanted to see,
okay,
but that's not very portable,
that's really depends on your architecture,
on your cluster.
What can we do in
reframe
to have that built-in
baked in?
And Vasileo is going to take over
and talk to you about this.
Can you hear me?
So, yeah, as Felix said,
yeah, people did,
oh, okay, sorry.
As Felix said,
yeah,
it's been,
users that have been using
free frame that have been using
ad hoc solutions,
one lies that is often
do historical analysis.
So,
I,
we thought that could be useful
for other users in the community as well,
to be able to compare past results,
inspect past,
test results,
get performance metrics,
aggregate performance across
different characteristics,
like,
not least,
test parameters,
time periods,
also be able to compare
performance between runs,
between different configurations,
between the current and versus,
historical data.
And, for example,
also different time periods.
And,
we also wanted to,
as a key goal,
to store as much test information as we can,
because,
experience shows that,
you later regret the information you haven't collected.
So,
it's better if you have the information already,
all the test information that you can have.
We still want to allow,
want to allow external post-processing,
because you never,
we will do the post-processing that everybody else would like to do.
So,
it is to give a basic,
analytics,
let's say,
layer.
Also be backward compatible,
we didn't want,
like,
users over frame to come back,
complaining out,
you change this option,
you broke that interface,
you broke my test,
so we want backward compatibility.
And,
also,
we want to provide an easy,
command line interface,
intuitive,
to be able to do some basic analytics.
So, we consider two options.
As Felix said,
one way of storing performance data is what we call,
in the refrain perflogs,
which are those usually CSV files,
that they contain the,
essential performance data of tests.
But there is,
although they are compact,
there is two disadvantages to that.
Important test information may be lost,
because,
yeah,
they don't carry all the whole information.
And information is really bound to the users,
to that log format that the user defined,
and basically select what information is important.
So,
then the second option,
which refrain does internally,
stores a full test case information
in a JSON report,
which then can dump to a file.
But the advantage of this is that
it contains the whole test information,
the test parameters, test variables,
where they turn and so on.
On the other hand,
it's quite verbose,
and it's also unstructured data,
because every test may have different variables.
Those that they have potentially used refrain,
it's test can define its own parameters,
its own new variables,
so,
which could be important to,
they are usually important to the performance you get.
Nonetheless, we select the option too,
more complete.
And this is,
I'm going to now describe briefly,
a bit of the designer architecture of this feature.
So, essentially,
it's layered with interfaces between its layer,
so that we can choose and along the next different
implementations for its layer.
So, on the top level,
there is a new CLI interface,
where we added some new command line options.
There's a couple of list stored test cases and sessions,
which will list present data of previous
or specific test,
in a tabular form.
There is its counterpart,
described stored test cases and sessions,
which retains raw data,
in JSON,
which then you can ingest elsewhere,
and process post-process yourself.
There is a new option,
performance compared,
that compares past results,
and there's also two other utility options,
to attach new information to the session,
with session extra,
all control the,
the data format.
The analytics layer does,
essentially,
the test case grouping,
the performance aggregations,
and the performance differences,
and retains either tabular data,
or JSON data to the layer above,
and at the bottom,
there is a storage layer,
where stores the results,
the data base,
and is also responsible for
creating the raw results,
out of the database,
and doing also the filtering,
based on the various criteria,
and then gives the upper layer
some JSON data.
Now,
some of the implementation details,
so reframe,
that's already in the reframe report,
is a big JSON file that reframe
produce with all the details,
and this is its structure.
So the structure is,
it's a bit hierarchical,
so you start with the session,
which is essentially a reframe,
minus run invocation,
and it has a session info.
Now, the session info,
it has a unique identifier,
plus an information about the session,
which includes also an information
now passed with session extras,
and then session contains runs.
Now, if you run,
if you run a frame,
your test may run multiple times,
and that depends on the actual options,
that you pass.
For example, if you have max retries,
your failing,
your failing test will be retried a couple of times,
or if you want to just rerun the test multiple times,
that's why there are multiple runs within a session.
Now,
within a run,
there is a set of test cases,
which is actually the test have run,
with all the information that your test has,
but for variables,
performance reference,
threshold,
actual performance,
that you got,
and so on.
This is like the information we need.
Now,
we store the results in a,
we started light,
so we started using it,
and I calculated the basis,
but the deal with the layers is that
if the need comes up in the future,
this could be easily replaced.
So, essentially,
we do index test cases and sessions.
So, practically,
we store,
in the database,
the full JSON blog of the,
of the report.
And then, we index the session,
also, by their UID,
and their time.
So, then,
we can do easily time-based queries.
And also, the test cases themselves,
we gain index time,
with,
by their completion time,
and also by a judo,
UID,
let's say,
the session,
you,
unique identifier,
the run index,
and the test index,
inside the session.
So, then you have the unique coordinates,
in a specific refrain run,
and you can retrieve all the test case information,
and then you can apply filtering stuff.
So, time-based queries,
use this index to retrieve,
the sessions of interest,
then the test cases are decoded,
and then filtered,
and they return to the upper layer,
for analytics processing,
and similarly,
for sessions,
where only the information of the session is decoded,
to save space,
because you don't want the whole session information.
So, going on a bit with the syntax,
the general syntax in all those options,
has three parts,
select,
spec,
and aggregation, spec,
and the columns,
spec,
those like presentation,
spec, somehow.
So, this select,
spec defines which results,
we want to select.
So, it can have different forms.
One is like timestamps.
So, you can say here,
from 25th of January,
and the 31st of January,
give me all the results.
Or with abbreviation,
there is also the last seven days,
till now,
or by UUID.
So, you just say,
even the results from that specific session,
or you can have through session properties,
which usually you start with session extras.
So, here,
it says,
all the tests that they have
run with a driver version,
576,
on that host name.
Then the aggregation spec,
defines how we want to group
and aggregate the performance results.
Oh, yeah.
We default,
there is a default grouping by name system,
partition environment,
and the performance variables and units.
And we can use custom groupings,
and they're set of available aggregation that you can use.
Then there is the column specs,
we define what to show,
but if all those are the fields that they are,
we have grouped our results by,
but you can add additional fields,
or you can completely use custom columns.
One, I think,
is that some common filtering options from refrain,
like minus n or minus e,
they are used,
they can be reused when you do
different formats in a latex query.
So here,
I have some examples.
So, for example,
here,
it's list me the main performance of a specific benchmark,
like stream code for the last seven days,
this is how you can do that.
Then imagine you have a parameterized test,
where your test has a mode,
different modes,
and also parameterized over the GPUs on the node,
and you say,
the main across all GPUs on the node,
and I want all nodes that I have tested,
and for all modes.
So here is a query,
when we can get like,
for a specific driver version,
we can get the information we want.
Then, okay,
I want you to compare all the benchmark data that you have
between two driver versions,
and yeah.
And then,
there are some also examples here,
I'm going to skip them
of getting like some information.
And if you want to get like the raw JSON report,
yeah,
you can still get it with the describe storage session,
and then you can post-process the way you want.
Or just list it as CSV,
and just get the information that you need.
And here is an example.
So this feature is available in refrain4.7,
which is the latest version.
It's by default disabled,
so you have to enable it,
and then you can also customize where you want the results
to be stored.
And here is an actual query,
and you see how it shows up
with performance table.
So you have the value A,
your first set,
your second set,
and also the difference between the two.
So you can easily spot,
you know,
regressions that are smaller than,
especially within the thresholds.
And we have like two or three minutes,
if you are very quickly.
I'm going to really quickly describe all use reframing
how we use this feature.
So as I said, it's very important for us
to check each hour with component,
because if one of them is behaving anomaly,
it can slow down your old HPC,
or the planning training.
So we need reframing test,
basically,
running on HPPU,
which is available in each HCA,
HPU memory,
everything SSD.
So every box,
basically on the HDGX-800 diagram,
needs to be properly checked for performance,
and stability,
and that's what we are using.
That's why we're using reframing.
So we're using SLAM to reframing SLAM,
and we have our own container on time,
that's also open source called EnRoot,
and Pixys is the SLAM integration
for this container on time.
And we use a lot of open source projects
for the testing,
the black nickel,
and then we bandwidths for HPU memory,
or adMAPF test,
or famous stream benchmark,
FIO for disk,
and we have single notice
that we are going to test,
as I said, each component,
and we have kind of eye-o-level test
that are closer,
maybe to what users are running,
but are very important for performance
prediction,
and also,
things that are multi-node,
because you cannot test multi-node,
obviously the network,
on just one node.
So we have two types of tests,
and actually the only people
in our team that use the reframing SLAM,
because reframing SLAM is me,
and Vasilios,
and our users actually use the GitLab CIY,
and they say,
I want to run on this cluster ABC,
I want to run the single node flavor,
and I want the short version,
so like 30 minutes,
and they click run pipeline in GitLab CI,
and boom,
they get the per node run,
and they get a different node fail,
they can click at it,
and click on it,
and look at the reframing log.
So that's the way we integrated reframing SLI tool
into something that our admins can use,
without adding them to no reframing.
And reframing even supports a drag unit export,
and the lab CIY also supports a unique export.
So you can click on the node,
and say,
oh, and actually this is the reframing log directly,
you see the reframing log directly to the user right here.
And I think that's the last slide,
and right on time.
And I think this is great,
because that allows us to have more insights,
and we can,
when we run in GitLab CI,
we populated that base,
and then when people ask us in our team,
hey, can you compare this between this driver version of Nvidia,
and this driver version to verify everything is fine.
We just run one command,
and we give them the table in as key art format,
but obviously it's our next,
a lot of next steps to get more insights into the statistics,
to get more comparison of the statistics,
and also a big open is automatically,
we make that more accessible to users,
we did for GitLab CI,
and also the query of the,
the latency of the queries is still a bit slow,
so that's something that's,
if I see those, we'll work on.
Thank you.
Any questions for feeling sensitive?
Yeah.
Do you think,
kind of reframe the public base,
where your CI might not have identical notes,
so it might be a worse CPU on one,
like,
figure,
you know,
in this case,
I said,
oh yeah,
do we support,
can we support,
in reframe the case where we have multiple types of notes?
Yeah.
We know we do support this use case,
in reframe you can have multiple performance targets already,
saying if the node is this type,
you get this performance target,
just this type,
this performance target,
and for this feature,
you can add,
arbitrary metadata,
to the database,
saying,
I want to run this workload,
and I'm going to add an arbitrary tag called,
RWA1,
and,
and then if you have a different node,
you'll say,
I want to run with RWA2,
and then you can ask,
this feature,
you can say,
compare all the results just on RWA1.
So, yeah,
we can add,
anything to add?
No,
from,
also from reframe test side,
you can support like multiple clusters at the same time,
and then you can in your test,
you can have constraints for your test,
say for example,
this test is for GPU,
and then reframe automatically,
will only select your test
for a configuration that has,
for example,
TPU.
So, yeah,
that's it.
What are the questions?
Yeah, any more questions?
One,
one,
and then,
thank you very much.
Thank you.