Okay, second talk is Christian, um, efficient histogramic for high performance computing in T++ with Yoda.
Thank you, Christian.
Thank you very much.
Good morning.
Um, thanks very much to the organizers for having me.
My name is Christian. I'm a research author engineer at the Center for Advanced Research Computing at University College London.
And I'm also a particle physicist, analyzing collision data from the Large Hadron Collider or LHC.
And today I want to talk to you about a tool called Yoda, which is a statistics library that we use in particle physics.
But before we get into the software, let me just set the stage a little bit better.
The LHC is the world's largest particle accelerator.
Um, it's housed in a 27-kilometer tunnel beneath the Swiss French border.
You can see this indicated in this aerial view.
And if you squint your eyes, you can probably make out the city of Geneva and the Alps in the background.
Inside this tunnel, we accelerate particles to nearly the speed of light.
And then we smash them in four interaction points, which is where the LHC experiments are located.
We have Alice, CMS, LHCB and Atlas, which is the one I'm working on.
So the LHC generates petabytes of data annually.
And just last year in July, the Atlas and CMS experiments each surpass the one exabyte threshold.
Which is the equivalent of about 3,000 years of uninterrupted Netflix streaming, also CHPT tells me.
Um, but even that is only about 10% of the total data set that we expect to record over the lifetime of the LHC.
So it's a big data set.
Um, this data isn't just the experimental results.
It also includes simulated data.
Um, for coming from Monte Carlo Venture generators, which are used to model the processes that we expect to observe in nature.
Simulated data allows us to compare our experimental measurements against theoretical predictions.
And they pay a crucial role in understanding and interpreting our data.
Um, yeah, at the same time, the scale of these event samples of the simulated event samples presents a major challenge for efficient processing and analysis.
If you've ever created a histogram with NumPy and Python or even with Excel, then you might expect a workflow where you sort of hold all of the memory and data and create a histogram from it.
But this simply does not scale the data sets at the scale of the LHC.
What we need is a tool that allows fast, in-loop analysis to allow us to summarize statistical information while keeping the memory usage minimal.
And the solution there is the updateable, um, constant size objects that allow you to summarize, um, to keep track of summary statistics such as mean and variance and stuff like that.
Um, with minimum, with minimum memory usage.
And that's where Yoda comes in. So Yoda stands for yet more objects for data analysis.
And it's designed around memory efficiency and high performance, and it's used in particle physics for over a decade now.
Um, it's, um, written in C++, um, but also provides Python bindings, as well as a whole suite of command and tools for, um, output, um, inspection manipulation from a conversion and things like this.
Um, although Yoda originated from a background of Monte Carlo, then generated analysis, it's designed as really general purpose, and it's not tied to any specific application.
Yoda was designed around, um, several key principles is a differential consistency, a real choir, because histograms don't just count occurrences.
It also represent probability densities, it's a level. Um, and so you want to make sure that these objects scale correctly if the binding changes.
Um, continuous aggregation allows us to process the data incrementally, as it comes in rather than waiting until all the data's there and analyzing it later.
Weighted statistical moments are being stored, um, which allows us to also calculate uncertainties correctly in the case of weighted data sets.
And for more, for essentially all of the modern Monte Carlo engine rate, this weighted data sets is the norm.
And integral consistency allows us to, um, project higher multidimensional histograms into lower dimensional histograms without introducing bias.
As a couple more principles to do with usability and robustness, we want to separate style from substance, meaning that the statistical data should remain invariant.
Um, regardless of how it's plotted, so your choice and visualization shouldn't influence the design of your statistical underlying framework.
We also want to separate the binding from the bin content.
And that allows us to have a clear distinction between live and a nerd objects, meaning active statistical tracking and finalized representations.
I'll show you an example in a couple of slides, what I mean, well, that exactly.
Yeah, and finally, um, user friendliness. So here has a clean and intuitive, um, API.
Um, and as well as, uh, it's got minimal, it's got zero dependencies, in fact, the core library.
We've got a couple of optional dependencies, mainly to do with desired output formats.
I think HDF5, um, but we'll get back to that when we talk about IO.
So let's talk about a bit more about some of the, um, more fun features that we introduce in a background to make this all happen.
Um, we've introduced, um, a new access, flexible access class in Yoda 2, which was released at the end of 2023, or the latest major version.
And this can handle both continuous and discrete access.
Um, a continuous access by far the most common type.
This is the thing we have end bins defined by n plus 1 edges.
We've got an underflow and an overflow in each end.
A discrete access, if you want to contrast that is, um, really useful for categorical data, such as counting particle multiplicities.
Um, and this is the case we have one explicit bin per category, and then you have one other flow that, uh, catches all the outliers.
And here are the main differences that discrete binning don't have a bin with associated with them.
Additionally, we introduced a binning class, which manages one or more of these axes.
And efficiently maps between the global bin index and the tuple of local indices along these axes, uh, supporting slicing and marginalizations.
Um, the bin content in Yoda is templated, which means it's quite flexible.
Nevertheless, we provide two primary content types to handle raw data collection and find a nice statistical representations.
For the live bin content, we have this dbn class or distribution class, which tracks first and second order statistical moments dynamically.
Um, um,
and so I wanted to say about this. No, I don't think so.
Uh, I, and this, this was already available in Yoda one originally in Yoda two.
We've just, um, generalized the star of a tree dimensions.
Uh, contrast that with the inert content type that we introduced an estimate class, and that really represents a finalized value with an optional error breakdown.
That supports both correlated and uncorrelated and certainty components, as well as their respective treatment and arithmetic operations.
We have a bin wrapper class that takes the template to content and links to bin properties.
And all of that is bundled in at the back end into a bin storage class, which is essentially, um,
a main, um, histograming object, a bit object.
And that can handle arbitrary data types, including nested structures.
So you can play around with storage of storages, which can be quite fun.
Um, we support both index-based lookups and coordinate-based lookups.
And for the latter, we also optimize between linear and logarithmic lookups, when it seems.
Um, and finally we also support bin masking, which allows us to disable specific bins, and that can be useful to create visual gaps and distributions as well.
One of the main features in Yoda is its ability to handle on the fly updates to the binning and for that we introduced, um, this object called the fillable storage, which is a pulling morphic, um, abstraction error.
The users have filled adapter to manage to bin updates efficiently.
The fill operation will return the index of the bin that has been filled, and that allows active tracking,
uh, tracking of active regions.
Um, and that's a feature that can be useful for, for example, machine learning, um, applications,
where adaptive binning might become, uh, necessary.
When the analysis is done, Yoda allows, uh, reductions from life to a note object.
And we have specialized, um, zero-dimensional cases if you want to just store, um, key statistical objects, without creating a dummy binning around it,
such as simple counters, and on the estimates.
You can also reduce both the life and the inert objects to a simple scatter objects, um, which is very useful for plotting.
And finally, all of the user-facing types that you see here, they also inherit from Yoda's analysis object base class,
and that thing introduces, um, metadata storage as well to provide additional context about the object.
All of the analysis objects in Yoda can also be serialized into a vector of doubles, uh, and that makes it possible to distribute them efficiently across MPI ranks.
And at the same time, it's also enabled sufficient, on the fly or rather, in memory merging of different histograms across different MPI ranks, without, um, having to rely on expensive,
and expensive operations.
And this design is also, this makes it interesting also for machine learning applications.
In terms of, um, IO, build a support multiple output formats.
We have a compressed ASCII output format, which is, um, metadata friendly and human readable.
I put a little snapshot here for some dummy objects.
Essentially, what this would look like in your editor, um, two histograms, um, with some metadata at the top,
and then you've got different columns corresponding to different parts of the summary statistics, one row per bin.
Alternatively, we also support an HDF5 based output format, which is clearly more suitable for HPC applications.
We also provide a Siphon-based Python API, um, for scripting and workflow integration, um, and we also have a built-in plotting mechanism, which is, writing out,
MATLAB-based Python scripts, uh, that are completely standalone, in the sense that you don't even need your installation to execute them.
They contain all of the numerical data, and the plotting interface for those fully customizable without changing the underlying data structures.
Um, and this ensures consistency and reproducibility makes it quite nice to share with your collaborators.
So just to summarize, um, you'll as a small but powerful statistics tool, um, it's build around, um, a couple of key principles, which is,
uh, a clean intuitive API, uh, self-consistent plotting and minimal dependencies, and robust and scalable statistical data analysis capabilities.
You're the two, the latest major version is a significant evolution in terms of usability, flexibility, and performance,
and if you need, if you're working with large data sets and you need robust statistical analysis, you're as a great tool to have.
Thanks very much for your attention, and if you've got any questions, hit me.
Okay, thank you for the talk, that was great. I have two questions. So one is, I understand that this is more of a library, less than in application,
and the other question is, uh, can you offload to GPUs, or can you make use of like parallel STL that could be offloaded to GPUs?
Um, we haven't had a request to offload to GPUs, don't think in principle, we probably could, um, we've never tried, could be done.
There's interest, and we, I think we'd be happy to look into it.
And for the, the main thing, and, and particle physics where this originated is that we need something that we can do in loop, and the way that this is now designed is that,
solve this problem at some level, and so if you look at typical applications where this is used,
histogram doesn't even show up in the profile in plots anymore, because this essentially solves the memory issue.
And if, if there were becoming a problem that you need to have with the GPUs, we'd adapt and look into this for sure.
Thank you very much for the talk.
Um, what I wonder is, do you provide some Nampai compliant API to, to use a Yoda basically or start using Yoda before diving into the details?
Yeah, so the question was whether we should repeat them.
The question was whether we, um, provide some Nampai compliant API. So we do.
And, uh, sci-fi base, some Python API uses the C++ code underneath the hood, but if you say asked for an array of all the,
so we're not, we'll check, do you have Nampai available in your environment if so will return it to you as an, as Nampai array, and if not,
if we'll fall back to the default Python stuff. So yes.
Any more questions?
Yeah, so the other question that, that just came to my mind is in the beginning, you mentioned you have these like super large datasets.
And then you also mentioned that this basically becomes to like no more like identifiable in the profiles.
Is that because it's so fast or because the other stuff is so slow.
And like how large is the data that like this actually deals with, right?
Um, it's, um, the, typically the the other stuff, the event reconstruction that you have to do.
And we calculate your observables are going to the Instagrams that takes a lot more compute than actually maintaining the objects.
Obviously, you can stretch that and you can go for very large objects with very fine binning.
But typically the kind of histograms that we tend to measure and publish don't have that many bins altogether.
What, what is that many in your use case?
Like, because I have no idea.
It depends, it depends.
But let's say like if you have, if you book a histogram with 50 bins in that, that's not much really in a grand scheme of things.
If you push this to a thousand bins, so I'm sure. Yeah, that's a different thing. Thank you.
Any more more questions?
Don't fall asleep.
Okay. Thank you.