Hello, everyone. I have a really great pleasure to invite Matt Tophol to talk about
Arrow. Matt is my friend. We meet quite often at Apache Software Foundation, a conference
as a meet-up event. We have the opportunity now for Matt to talk about Arrow in the context
of AI. So, it's all yours. Hey, everybody. So, as you all said, I'm Matt Tophol,
a PMC member on the Apache Arrow and author of book in memory analytics with Apache Arrow. So,
so small confession here. AI and ML is not my expertise and I hope that doesn't bring out
the pitchforks and torches on them. But my expertise is more in the data analytics pipeline
area. So, I've talked with a lot of people and a lot of AI and ML engineers and people focused on
it. And I've been told that my little diagram here is surprisingly accurate from a lot of people.
So, the point here is that when you think about it, a lot of AI and ML can be thought of as data
pipelines. And you have all of these libraries and tools and utilities that exist in the ecosystem.
You've got on X-JAR, you know, Jax, PKI's, GDUF's for formats. You've got TensorFlow,
PyTorch, Lama, Rapids, Jax. You have all of these things and then you have all of the
sources you're pulling the data from. You have the your varies the frame, libraries,
your future stores, parquet files. It's an enormous ecosystem that keeps growing. And of course,
there's all these different pros and cons to the various libraries. So, when you're looking
in building out all these machine learning models and inference pipelines and just building out
everything out, potentially controversial, it's a specialized version of a data pipeline.
And that's where I came at this and I thought about it a lot. And when you think about it
in those terms, in terms of AI and machine learning models as a specialized version of data
pipelines, you come up with interesting ideas. Because now I'm aware that this doesn't represent
every, you know, the entire area of miles. But a lot of miles are basically, you know, a
pre-processing step of your inputs. And then you send it through various layers that do
transformations, that do interesting computations. Eventually you get your output of a prediction,
maybe with your probabilities and so on. And then you do a little bit of post-processing and
send it out somewhere else. And in many cases, you're passing tensors between those layers and
across, you know, you're pre-processing into the layers from layer to layer all the way through
to your prediction, possibly even all the way to the post-processing output. And then at that point,
you convert your tensors into whatever you want to actually output to your user, to your inference,
to convert it back to reasonable texts, so on and so forth. And of course, with this entire giant
ecosystem of tools and libraries, everyone wonders, which one should I use? Because it's expensive
to tie them together. But there's all pros and cons, all them have different benefits,
they have different drawbacks, they have different functionality, different algorithms that the
library supports, that the framework has different performance characteristics, and the cost of
linking these things together to kind of mix and match what you need can often be very high.
They may not have the same layout in memory, so you have to copy and convert from one format to another,
the passive between the libraries, or you have to, or one library may only work on CPUs
and the other may support GPUs, or even just, you know, the library doesn't expose
a way of getting the data out while keeping it on a device model, possibly, because of streams and
events and other very low-level manipulations. And then you have a lot of other
new tools and utilities coming up. You know, you've got the building of LansDB for a vectorization
database, and a new format. You've got nimble as another option for improving upon like the
parquet formats and optimizing towards ML workflows and point lookups and random lookups. And a lot
of these systems are consolidating around utilizing the Apache Arrow format internally for how they
represent their data in memory and on the devices. Now, this is the first point where I'm mentioning
Apache Arrow. So the point I'm mentioning here is interoperability is being able to get data between
all these tools, the all these libraries, without a high cost. You know, you want to be able to mix
and match everything without having this like expensive slowdown. Currently, when you're passing
your data between TensorFlow and PyTorch or from hugging phase to TensorFlow or everywhere else,
you're typically going to use NumPy. You can use pandas, do some data frame manipulation,
pre-apac, for your pre-processing, maybe Coupai, Dail Pack is an often big one. XG Boost is another one
like where you're finding ways of passing your data in reasonable formats between systems between
libraries. You know, and then you have these intermediate scenarios where you have your data source
of CSV files and parquet or just raw JSON data because you hate yourself. You know, so
the result is that there's a ton of unnecessary copying that happens because the underlying
tools and libraries can't operate on the data as it is where you're starting from and you need
to copy it and transform it to useless use it. So before I continue going forward, which obviously
you can see where my answer is going to be going, let me actually explain what Apache Arrow is in case
you're not aware of it, you may not be familiar with it. So Arrow is an in memory
column-oriented data format. As core, it's a spec. It's a spec that's been implemented in
the whole mess of languages. You know, you've got C++, Go, Rust, Python, or Python,
C++ libraries. You've got Julia Matlab. You've got Java. It's implemented in a ton of languages.
And the point here is that by consolidating around a column-oriented in memory format,
you eliminate a lot of the cost of serialization and decirilization passing it between libraries,
passing it between nodes, between systems, transport over the wire, and so on. You know, transporting
the data over the wire is identical to the bytes in memory, which means there's no serialized
or decirilized step whenever you cross systems, or even crossing process boundaries, or even just
one time library boundaries within the same process. Let's not get to in a moment.
And because this column-oriented, it's highly efficient and friendly for vectorization and
Cindy. And the chances are, you're probably interacting with arrow or utilizing it under the
hood somewhere and don't even realize it because it's already everywhere. If you ever use lib
kudyeth in video rapids and that whole ecosystem, you know, kuml, kuspe, all that entire ecosystem.
And video rapids and all of the related system kudyeth is using Apache arrow data format just on the
GPU. It's the same Apache arrow data format. It's laid out the raw bytes in GPU memory.
All right, identical to the way it would be laid out in CPU memory, but because of the efficiency
of a calm-oriented data format and the benefits of passing it around without the cost of serialized
and decirilized, it becomes super efficient and Nvidia adopted that for that entire ecosystem.
If you're using Polars, the Polars data frame libraries and pandas or rust. The entire underlying
backing memory representation for Polars is Apache arrow. It's just arrow. Which means that you can go
from libraries, for example, to pandas, to Polars and so on, zero copy because they both have that
format. Pandas has a pie arrow back end. You know, earlier on, we had to talk from from a developer
hugging face. If you go under the hood, hugging faces data sets library. You load the data set
downloads a bunch of parquet files and then hugging face has a caching system locally so that you
can operate on larger than memory data without blowing out your RAM. How to do that? It stores it
as a series of Apache arrow files in your cache and then memory maps them so that it can load only
the bits it needs into memory as it reads it, but you can interact with your large data set
that's larger than memory without requiring a large amount of RAM. And because of the properties
of Apache arrow that allow you to write out arrow IPC file and just memory map it to improve your
data and only actually have to allocate memory for the bits that you use. And then we go to the
compute side of things, the actual computational things. You know, dot db's internal memory representations
about 95% identical to arrow. Which means that in dot db implements the interfaces so you can
get zero copy, cost, results from dot db in the arrow format. Snowflake has an arrow interface for
getting API data. BigQuery has an arrow output implementation. If you're using Apache data fusion,
data fusion started as a sub project of the Apache arrow project, a top level project of its own,
is entirely built under the hood as Apache arrow for its memory representation.
And as a result, all of these system support arrow in an arrow out for interoperability. Any tool
that can output arrow can be used to then send it to one of these systems as ingestion. And any tool
that can receive arrow data can zero copy superficially get data from those systems into it.
Now, the point when I'm talking about all the zero copy and interoperability and
manipulations is that it's often very hidden where these copies are. When you're dealing with,
like I'm going to bring up the hugging face data sets library again, you can see here that
when you load the data set, the underlying dot data for the data set is a pi arrow schema
and the underlying actual data columns are pi arrow chunked arrays.
But when you actually index into the data set top part to retrieve your column,
it converts it to a Python list. That's a copy. Because it has to convert, it has to be in the
Python memory, not the raw underlying series, it's a plus memory, there's a copy there.
You may not realize it and it's just very simple hidden situations where you don't realize that
copies are happening simply because the libraries you're interacting with are not necessarily
already utilizing arrow. So with everything I'm talking about, everything I brought out,
the meat of one, the meat of this is called about what exactly am I proposing? Why should all of
you care what I have to say here? Well, you have the multiverse of arrow systems and
universes and libraries, you've got the IPC format for interrupt interrupt data, for interoperability
between libraries and systems. You've got a C interface which is a stable ABI to pass the data across
run times in the process without having to copy anything. It just passes pointers to the buffers.
And you've got this implemented in every language under the Sun has, well, not every, but most
languages have a implementation of the arrow spec. Not only that, but arrow also specifies
interfaces and protocols for data transport and database connections with ADVC instead of ODBC
to get a column-oriented communication of data. So with all of this connectivity and all of this
interoperability, how can we benefit from this? How can the AI and ML pipelines and data
tools and libraries really utilize this to their fullest? And by doing so, the result of doing that
is going to be an increase in the ability to mix in match libraries for getting exactly the right
tool for whatever operations, computations, calculations, you need in your model.
Now, when you actually think about leveraging that zero copy, you can see the performance benefits.
There was a white paper published last year by members of one of the straws I mentioned earlier
about plan, which develops a Python sidecar runtime, highly utilized for AI and ML workflows.
And they were testing in the, as one of the pieces of the white paper they published, they were testing,
reading data frames on a, you know, C5 9x large AWS machine. And depending on with a source of
the raw data was coming from. Now, the interesting part of this was because you're, if you're building
a distributed system, how do you get data from, from node to node, from piece to piece? And if
you're just, and so a common way of doing that is you write the resulting intermediate data to
a parquet file in S3. Another way of doing it commonly is writing a parquet file onto a shared
SSD because you don't want to, because you want to bypass the network cost of going to S3.
And you can see those comparisons here of doing it with, you know, 10 million or 50 million
rows of, you know, six gig or 30 gigs of data. If you utilize one of the frameworks that the
arrow project specifies, arrow flight, which is a GRPC based protocol for building a framework
on top of system communication built around streams of arrow record badges. And you can see right
here, you know, there's, you still have that network overhead and also arrow isn't as comprise
and compressed like parquet. So that may not be the best, but that arrow IPC, if you write the arrow
for arrow data to share memory and then just memory map it into your system, which is like I said,
exactly how hugging faces caching works, look at the difference in the performance.
You know, right there you can see what, where this removing of unnecessary copies results in
enormous potential for performance gains, which therefore leads to enormous potential for expanding
model computation and inputting more operations without increasing the massive amount of cost.
Now, one thing you might wonder about, if I keep talking about arrow, I'll keep talking about
how we can use like, one thing you might talk, you might wonder about is, okay, can it support
everything I need to represent? And the answer is most likely yes, you know, you have all of your
standard primitive types, you know, you know, floats, float 32, float 64 integers of all
signed and bit width and so on, decimals, strings and so on, but you also have complex types,
maps, trucks, lists, all efficiently represented in column oriented organized memory.
Now, the arrow spec also includes a piece for extension types and an extension type is just
an already existing arrow primitive type and then layering some metadata on top of it so that
an individual library or individual runtime or instance can register a custom class to add
semantic utilities to that data. So that, for example, you could register a JSON type that's just
a string column, but when you read in the data, it will wrap it in this custom class that will
give you extra semantics and extra semantic functions on that string column because it's supposed
to be JSON. Doesn't change the underlying storage and also it means that the data can flow through
other systems that support arrow that don't have to care about the extension type because it's
just an arrow data with some metadata keys. And so the community got together, especially with
this AI and ML work and put together some canonical extension types, specifically to help out
this interoperability with ML workflows and thus create a fixed shape and variable shape
tensor types of extension types for arrow so that you can represent, you know, tensors of any
amount of dimensions, permutations and so on with the arrow format in a standardized way to make it
very easy to work around. How does it work? Well, I can give credit to a couple of my colleagues
who recently came in and said that sat down after their pie arrow talk, rock and a link up that
gave a big talk and also pushed through a lot of the stuff on how the arrow tensor system works.
So you can see here, like, utilizing the way that you have, like, say a NumPy and DRA and the
way it would be represented in the raw memory is just that continuous values buffer of all the
data in one chunk and then you have a logical, you have a logical handling of it that will then
allow you to interact with it as an actual list of multi-dimensional tensors.
But the raw underlying bytes in memory are a continuous chunk of bytes, the way you would expect it
in say NumPy or DRA, or any other way, or any other library, or the way that you represented.
And so you can get this highly efficient constant time at random access to your individual
tensor you want, but you also have this list of tensors that you can pass around very efficiently.
And as I mentioned, there's also a variable shape tensor. Now, while most, most AI
in the ML probably is utilizing fixed shape tensors, there's also a lot of usage of variables,
variable shape tensors, especially when you start talking about LLMs, where your input data of
tokens of your strings and your variables and your values and your text is not going to be a fixed shape.
Each individual input may be a different length so you need to handle this variable shape
of the tensor and still, and still handled using the underlying arrow format, raw bytes,
continuous, easy to utilize, and interoperable. And just so you know I'm not blowing smoke,
and so I'm not lying to you, some proof. So if you look at the actual code here,
I generate a random NumPy array of data and I flatten it out so that I can have my nice list
of 32 by 32 tensors. So, and then I create an arrow tensor array from that.
And then for the app for the NumPy array, the arrow tensor array, and the
and the NumPy underlying result, when I go from the tensor back to NumPy, you can see that
one, the shape is retained, you know, a thousand 32 by 32 tensors, and two on the left-hand side,
you can see that all three versions, whether I'm accessing it through arrow,
whether I'm looking at the original NumPy array, or if you're looking at going from arrow to NumPy,
the actual memory address of the raw data is the same address.
That through these conversions of how you can interact with it, with arrow, with NumPy, with the
extended tensor type, there was never a copy of the data. All three of them are still pointing
at the same exact location in memory, not, not no point actually copied it.
Another example that I got from, from another colleague from my colleague is
storing tensors in parquet, which can happen sometimes when you want to store the weights,
when you want to store the result of tokenization, when you start to want to store input data,
you want to store it in parquet, and it is extremely easy to do so. You know, you have your,
you know, create my little shapes, do it, my extra transformations here, in this case,
I'm loading up geotyph data for geospatial handling, do some manipulations for the inputs,
and then I store it in the parquet file as a whole bunch of tensors representing the geotyph
data. And by converting it to a fixed shape tensor pyro, I can then use the parquet library that's
part of pyro, and just write my table to parquet, and it maintains the tensors, just going to rot
it right table, and then I can read that back out as a pyro table, and now I have my tensor,
my, my list of tensors again, extremely efficiently, and then I can go through it,
read the table, two numpy and d array, guess what, there's no copy, there's zero copy,
pass the resulting tensor list of tensors, p-process them, put them into my model for prediction,
get the result, pull up from a numpy and d array, make my, make my pyro table, and then write the results to a parquet file,
the same way that I can then read the tensors back easily. Okay, so many of you might be familiar with this lovely
slide, and the big question is, okay, so, but we have the alpac, we have numpy,
why we need another interoperable standard here, and the big thing that came to me as I investigated,
and as I researched, was the complex types of board. For example, the alpac numpy,
only support numeric data. You, the alpac has a little thing, you can put opaque stuff in, but you
lose your semantics stuff. You can't represent say strings using it, or null data.
This becomes this habit of using N-A-N to represent nulls, instead of actually having nulls,
via arrows believe it map. You can see that, you know, a while back there was a big
talk with a gauge showing a huge performance benefit of building a feature store with ducted
DB and arrow flight. The other thing is that, like I mentioned before, QDF and when I already
are arrow on the device, which means that if more libraries implement support for the arrow device
API, you can do pre-processing on a GPU, and pass your results to your model without having to
copy it back to the CPU and back to the GPU. You can keep the data on the device for whole workflow,
whether it's for training or inference. The pre-processing and the post-processing can be on the device
and not force those copies. And that's all that's it. You were removing the unnecessary copies.
Now, I had a contrived quick demo, but I'm being told I have no time. So I'm going to skip that for
now, but if you want to see it, come find me. And the ultimate of the goal here is really just
mix and match picking the best tools for the job as you need them. So if you want to learn more,
you can go to the arrow documentation, you can go to the arrow websites, you can leverage the
second edition of my book, which actually contains a whole bunch of stuff including the example I was
going to show of using arrow with GPUs and so like that, among other things outside this top context
also. So thank you very much, and I hope to see you guys around.