Okay, hello everybody, nice to see all of you interested in the topic.
My name is Alenka, I'm a software developer, I like to be, I'm also an engineer if needs
to be.
Data engineer, my friend Rok is also here with me, we're all interested in error working
on error as commuters, mostly Python, Rok is also involved in C++ and Perkate.
So before we start, I would like to also get to know all of you a bit, with exercising
our limbs.
So if you're involved in data, if you work with data as an engineer and list whatever
can you raise your hand up.
Okay, keep your hands up and raise the second one, if you know or if you use Pyro in any way
as pandas directly, you are very high, okay, that's good, thank you and all of the others
that didn't raise your hands, is it correct for me to assume that you're here because you
heard of error and you're not really sure what it is, yeah, okay.
So our goal here is to, if you know Pyro to know it better and to see what you can do
with it, extra and otherwise to kind of have a glimpse of error, like what is it about?
We hear a lot about it and we don't use it, let's get to know it a bit better.
So I'm going to take the introduction, as I think it will be necessary, in its essence,
arrow is a format, is this specification for a column or format, so if you have tabular data,
it defines how you have it stored in memory, column by column versus roll by roll, so it's
column, right?
It works well for analytics and some other complicated words like send the operations
and all that, but it's really, really good for analytical workloads, it's not designed
for fetching rows of data, okay?
So here you have a diagram to compare how it's stored in memory, like roll by roll or the
second one column by column, so it's together, columns are together in memory.
It comes not only with the specifications, so the implementations are done in multiple languages,
plus plus, a Python has a binding, the C++, you have Rust implementation, etc.
And you can just move around between languages in the same process and also you can move
around outside through the wire and so if every part of your workload understands arrow,
it's fast and it's efficient and it's zero copy, because not necessarily if you go through
the wire, but it's really efficient if everybody knows arrow.
So yeah, the implementations implement the spec, but also other functionality built on top
of this spec for efficient storage, etc.
So we got a bit of an understanding what arrow is, so in essence it's a way to store data,
so it's more efficient.
Python, Py arrow is a Python binding to the C++ implementation, so whatever you do in C++,
we take it, we connect it, and we use it in Python, with maybe some extra stuff.
It works really well with a NumPy, with pandas or with built in Python objects.
And it also the same as other implementations, it has other functionality.
For example, most of you probably know who use Py arrow, I would assume you use it for
a copy, probably.
Okay, so let's start with the first topic that we have, like we will share knowledge about
four topics, of the Py arrow, one is array interchange, one is storage, the third one is compute,
and then the final will be flight. So the first array interchange, let's start with what
array is, it's a building block in arrow. You have array, the same data type for all the values
and non-climbed, so it's the array stored together in memory, and then you can put
array together in a record badge, which would be like a data frame if you know a Python data frame
libraries. But you can also stack together arrays into a chunked array, and then get those chunked
arrays as columns, and you get a table, which is like a data frame, but chunked.
Okay, so the arrays are the smallest building blocks.
Not the timer? Okay, this is a diagram of how an array would look like in memory.
An example is for Boolean and string data types, which is not the simplest example,
but just to see the various variables, the difference. So for every column, every array,
you have different buffers that live in memory of your computer. For Boolean, you have two
bitnaps, because in an arrow that's a special case, it uses bitnaps for Boolean. If it would be
a primitive, it would have one values buffer, and then a bitnap buffer to indicate null values in the array.
The string type has an extra offset buffer to know where the string ends or begins and ends.
Okay, so that's kind of a general view of how arrays are stored in memory.
If you want to learn more, there's a link with all the different diagrams for all the functions for
sure you know from it arrays. Yes, anything? Okay, so let's go to the interchange,
done by an arrow. If you have an unpy array, you can go to pyro array with just creating an array
out of an unpy, and you can check that it has the same pointer to the buffers, the value buffer.
So that means that nonpy holds the same data, the same part of memory, and pyro. So there's no
zero, that means zero copied. You can just use nonpy, you can use pyro on the same data without
any extra copying. So you have a nonpy array, we turn it into a pyro array, and then we
went to pandas, why not? Okay, so we got the pandas series, we check the buffer, so the pointer to the
buffer through the array interface is the same. We can go from pyro to polyards.
Again, check the buffer, it's still the same. So all this Python libraries operate on the same
data, and you can use all of this functionality from different libraries. You can also use
a data fusion, not an array, but you can stack it, stack it together in data frame, and move it
to data fusion, or duck DB, you can go to nano-arrow to narwhals to arrow three, which is a Python
binding to arrow rust implementation, et cetera. So all zero copied. The previous
interchange is built on top of C data interface, it's kind of a way that arrow project
supports other libraries, moving data, or getting access to the data without copying.
Okay, but don't you also have other protocols, for example, DLPAC, and you can also go from pyro
to DLPAC through DLPAC to torch without copying. Okay, so that's kind of a short view of arrays
in pyro, and how you can move to different libraries and use all of those without copying.
The next part is storage in pyro. Thank you. Yeah, so of course data does not always live only in memory.
So sometimes you want to store it places, and of course, having column more into data than
it's nice to have a fitting column oriented serialization format, which is what
per case designed to do. It's designed to be efficient for storage and retrieval, right, both,
and it provides efficient compression and encoding schemes to handle data in bulk.
It provides various other things, such as encryption, and it comes with decent amount of
metadata to optimize certain optimizations. And pyro provides an implementation or rather pyro
wraps the C++ implementation of parquet, reader and writer. Here is a very tiny image of
how data looks like on this, when it's serialized, unfortunately, it's unreadable, but basically
the blue parts are metadata, and then you see the boxes on the left are pages of data stored on this.
And here's an example of how one could use parquet to store images on this. Now, we recently
introduced to arrow and also pyro a fixed shape tensor format. And here we have an example where
we read some images. We take their numpy representations. We zero copy those to a table,
right, to well, first to an array of tensors. Then we put that into a table. Again, a zero
copy operation and we finally write everything to parquet. Okay, here we don't compress, but we
could. And then we read the table back, and the table is still the same. Of course, in this case,
we wrote one, so we serialized and then we serialized back. And we hope that this kind of in the
dayonage of people doing a lot of deep learning and stuff, they might start using this.
I think hugging face actually uses fixed shape tensor for some of their internal things.
So here's another example of what you can do with parquet, because there's made it attached.
You can check it, check whether it fits certain Boolean conditions. And if it does,
you read pages that do and you emit reading others, which can locally, of course, it helps,
but especially it's useful for remote reads. So in this case, a reader of a table that was
starting to onto disk will emit certain rows, right, and this is actually building on top of the
first example here. It's only reading the second row of the date, and you can see that's the result
indeed. It can really be useful if you read from remote, it will reduce your loads.
Further, for reading and writing remote, this is a working example of writing to an
history bucket. It will work with other object storages and self-hosted ones too.
What you need to provide is AWS some sort of credentials for your remote system, and then you
basically just write to the URI and read back. Yeah, and here, right, because of the network traffic
optimizations with metadata come in really handy. Further, I mentioned encryption earlier.
I hope this is approximately, yeah, I guess in the back you can't read it. But basically, this is an
example of working encryption. So everything is on this slide. Sorry again, it got kind of big,
but it does work, I promise. So the way we implement a modular encryption in Perkay,
is you have a key management system, so that can be an open source example, it's Hashikar
core vault. It keeps your keys, right, you provide the KMS connector, a key management connector.
You create an encryption configuration that tells you which keys to use for which columns,
and which keys to use to encrypt the footer, the footer only contains the metadata,
but it's important because you have statistics there and you can leak some data out of it.
Then you have a cryptofactory that takes the key management system and it takes the configuration
properties, so the mapping of keys, and finally you pass everything to a perkay writer and it will
encrypt your data to disk. So for instance, with all the past three slides, one could make like
training, one could write an encrypted data set to S3, right, and read it back, and data at rest would
be encrypted. And here is then reading back, this data, this is luckily a bit smaller,
because you don't need to deal with encryption configuration. So you read back data that was encrypted.
In this case, this example only encrypts one of the columns, the other one is not encrypted,
so that means that you could also just read the non-encrypted one back and ignore the encrypted
column and that would still work. Right, compute. Okay, back to me. The third part we want to show you
is the compute module of pyro, again binding to C++. With the module, you can do basic operations
on arrays, on scalers, you can do stuff with tables and data sets. If needed, for example,
a group, there are aggregations, joins, and filtering. I'm going to show examples of basic,
basic compute functions, but you have a reference for what we support here in the link on the slide.
To note, the pyro is not meant to be a library to give you compute functions for every need.
You have it's more like basic things. If you really need to do some minimal stuff for other
numerical operations, you use other Python libraries to facilitate that. So first, a small example
is using some minimum or maximum of an array, so we're back to arrays. So if you have an array
and you just want to see the sum of an array, which is a column, you can use this compute
basic compute function as the same for min or for other available. You can also do some transformations
on arrays. For example, if you have two arrays, and you want to create another one. So another
column that checks the equality of those two arrays, you would use equal function of the compute
module. You could also multiply a column with a scalar. So one number, multiplying all the
elements in an array, or you can multiply two arrays to get another one. So like basic
examples of how would you use the compute module in pyro. So this is the third part we wanted
to show the functionality built on top of arrow format, and the last one is the flight for
loins or the wire. Thank you. So flight RPC is kind of a specification for
or how to use it as a specification, or is it like an implementation, a specification, okay?
Arrow flight is a specification for sending data through the wire using GRPC. So it's kind of like
thin wrapper on top of GRPC that gives you convenience methods to send columnar data on the wire.
So GRPC brings some benefits, but also some complications. So we want to abstract those away from you.
And basically, as it's an RPC framework, it gives you an option to define vocabulary for
for custom services, right? You can write, you can make your own endpoints, and call that,
it comes with an ability to put everything under TLS. So your traffic will be encrypted,
and the idea is that it would be used for either native, ironative storage services,
connections to databases, etc. And then there's another thing built on top of that.
It's arrow flight SQL that enables sending SQL requests, so it reduces the functionality a little bit,
but already defines the SQL part. All right, so this is just a diagram right about the difference
between an RPC and a rest point or a rest system. So instead of having this predefined
words that you can use to pass data, you can actually write your own vocabulary for passing objects.
It comes with its own downsides where you can really get into trouble, but it gives you a lot of power too.
Right, so the way that communication works is a client would connect to the server,
to get back the information about the endpoints that are available there, and then the client can
start to get command and then gets data back in record batches from the server.
I think it's possible to also paralyze that so you can have multiple servers returning the data.
So I didn't want to include the example here because it would not fit in a single slide,
but you're welcome to look at the cookbooks. We have multiple cookbooks for different language
implementations, but this is the Python one. So there's there's an aflight example there,
and some other stuff we covered already. So yeah, we're an open source project. We're currently
where a community driven peers are very welcome. Also input on what people want,
exactly come open issues, ask. We have a bi-weekly call also where computers meet and discuss stuff,
so you're welcome there as well. What else should we say?
Questions, yes, questions. Please, we're lucky to have one here also. So if there's a complicated question,
he can help. So do you have any question for Ake Alenka? Yes, I'm going to run.
No, yeah. Let's far up.
Hello. Okay, yeah, thanks for the talk. One question about a here. Arrow IPC,
so on the process communication. Can you maybe explain a bit? Let's say if this scenario
where we have Arrow data in a Python process for example, but we also want to access from Java or
from Rust, are there ways or also to kind of get zero copy in those situations? Yes, so you can
basically IPC is just the way that the data is represented in memory, you just send it as a buffer
as a IPC, right? And you can indeed do use it for inter-process communication.
We didn't cover it here much because we had to limit the topics.
And inter-language communication is pretty easy and I believe there are examples on the cookbooks.
Another use for IPC is also if you want to, you can store it to this. That's now called a
feather file or an arrow file interchangeably a little bit and it's basically just dumping the IPC
directly to disk and gives you the benefit of being that you can memory map it so you can do
stuff that's larger than memory. Yeah, I don't know. Cool, thanks. Thanks for the questions anything else.
Any other questions?
Hi, thanks for the presentation. My question is about the static type analysis tools because like
all the spy arrow deal with them because when I was working with pandas, it was a huge headache
constantly. Yeah, yeah, as far as I understand there's people who want this and there was some
POC in progress, yeah, I think there's actually a PR in there. We just need, you know,
people to help and get it to the finish line. Yeah, we want that also. Yeah, yeah, indeed.
Thanks so much then. Yeah. Thank you. Thank you. And if it goes, we're out of time I believe.
Thank you so much. Thank you.