Who's going to be talking of the next segment around OVVC take an arrow to the net, not
probably so far.
So, I have a team up here.
Thank you very much.
I agree.
Greetings everybody.
Hi.
And then one who's watching on the streaming.
So, like you said, this is OVC takes an arrow to the net, going to be giving a talk,
talking about utilizing the patchy arrow in context of database connectivity.
Before we get going, just a little bit who I am and why you should care at all what I
had to say.
So I'm at that topo.
I wrote the book on patchy arrow right here, actually the publisher sends me copies and
ever go to conferences.
So if you can find me throughout the conference, I have one copy of my backpack every
day that I will give to someone and sign it whatever because I can.
I'm also a member of the patchy arrow PMC.
I primarily work a lot on the go implementation, but also contribute to PIRO, C++, ADBC itself,
and I'm also a committee on iceberg and developing the go-lang information iceberg.
So if you're not familiar with patchy arrow, quick primer, patchy arrow is an in-memory
column-oriented data format, it's a spec.
The idea is that you have this in-memory performance column-oriented data format that you
can be identical on the wire as it is in memory, which enables you to transport data between
different systems with no serialization, desirization overhead or little overhead.
It also has implementations in a truly ridiculous number of languages.
C++, go, Steesharp, Julia, Matt Lab, JavaScript, all have implementations of the arrow spec
and many more languages that I didn't mention.
If you're not familiar with the term column-narr or column-oriented, you're very simple to understand,
typically when you're talking about representing the data of a struct of a table in memory.
So the often way of doing that is row oriented, where the data for each individual column
for a given row is collectively in order in continuous memory.
In column-oriented way, you have the data for each column is contiguous.
Now the reason for this is that it makes it much more efficient for friendly to vectorization
and SIMD operations.
If you're not familiar with vectorization SIMD, vectorization pretty obvious and the idea
being that to do vectorized operations, the data has to already be contiguous in memory.
So by having your general in memory representation of your data already be contiguous by column,
you can perform vectorized operations with no transforming it or manipulating it to make
it work.
So if you want to do something, say, reading data, you want to read only one or two columns,
not all of the columns.
Well, because the data is contiguous as column-oriented, you don't have to read the extra
columns.
This is one of the reasons why Parquet is very efficient at the ILO level, because Parquet is
column-oriented so you can read just the columns you need.
If you are performing operations and doing aggregations or other computational situations,
you get to use a lower memory footprint to make it work, because again, you only need
the columns that you're operating on directly.
You can reduce the ILO of what you have to read in and out and so on.
So now that we've established what Arrow is and column-oriented and the whole idea is there,
we're all on the same page, hopefully on Arrow, at least enough that you understand the
Russian talk.
So now let's focus on the actual bulk of the talk itself, which is the most common way that
most applications and systems interact with databases nowadays is ODBC and JDBC, right?
And you know, you add your client API and you have swapable drivers.
So kind of looks a little bit like this, you know, you have your application, submits
its query via JDBC or ODBC via the interface.
That goes to the API, which you've loaded a driver for, a driver, then will translate
that query into whatever form the database needs it to be for whatever specific protocol
the database is using, the database then executes the query and result and the results are
sent back in whatever database specific format their return does.
The driver then translates that into the format that ODBC and JDBC use for their APIs
and then the application can now iterate those results that came back from the driver translating it.
Standard protocols, standard, very straightforward, most of you are probably familiar with all this
and it makes sense.
So there's pros and cons, JDBC and ODBC have been around for decades, they're not going anywhere,
they are standards, they make sense and they're ubiquitous, everyone is using them.
For almost any database system there's probably an ODBC or JDBC driver that exists.
So you have nice and stable, the problem is that conversion cost.
The majority of analytical systems nowadays are all column oriented, which means that we're going
from a column oriented compute and then transposing all the data into rows for ODBC and JDBC
because their row oriented APIs, yes ODBC does have a column oriented interface but it's
not great and you have to pre know what the schema is beforehand, which isn't as good for
single queries but stay with me here. The row oriented APIs. So we have our analytical data systems,
our column oriented and we transpose all that data into rows to pass it along ODBC and JDBC
and then half the time the application is going to transpose it back in the columns for your
visualizations, your BI tools and other compute. That transpose to translate from columns to rows
is expensive. It's a copy, you're copying all of the data and like I said, nearly all
analytical data systems nowadays are column oriented, whether using snowflake, druid,
treno, diqueery, ductDB, all of them are all column oriented internally.
So if you want to avoid that cost, that conversion, you end up having to integrate with specific
SDKs for each and every system. You lose the benefits of the abstraction that ODBC and JDBC are
giving you. So you can see where I'm going with this given the nature of the talk, there's another way,
ADBC, arrow database connectivity. Conceptually same idea as ODBC and JDBC. Single client API
swap out the drivers for what individual data system you want to communicate with. The difference
being that all the data communication through the system and through the API is built around the
patchy arrow in this column oriented nature, which means that your column oriented data system
can now just pass through the data and not have that conversion cost. But it also means that
other systems that are not currently column oriented like Postgres or MySQL or whatever,
you can have a driver set at the application level. Your application can only deal with
arrow column oriented data. We have bindings for it in a host of languages, Java, Ruby,
Rust, Python, Go, or have bindings and the drivers themselves are written in multiple different languages
because at heart it's just a CAPI. So let's take a look at that workflow again. But this time
let's think of it using ADBC. Once again, application submits a SQL query to whatever system
using the ADBC API. You know, execute query. The driver will then use whatever protocol needs
from the database to submit the query to the database itself. Database, execute it and ideally
returns the data already as arrow data. If the database returns it as arrow data, then step 4 is
just pass it through. There's no copy, no translation, no transformation. You take what the database
gave you and just literally pass the pointers to the data across. If the database did not return
it in arrow format, then you convert it to arrow format and pass it to the API and now you're
still getting arrow, record batch, stream of data out the other end. And so everything from the
application perspective is always arrow no matter what your underlying data sources. And like I said,
the ADBC is essentially a header file. It's a CAPI which means that anything that can call a C function
can use it and then we wrote the bindings to make it just easier for each individual language.
It also means that anything that can expose a C interface can be used to write a driver,
meaning whatever language you want. And because it's utilizing arrow, arrow has what's
called the C data interface. It is a way for systems to pass data across a C boundary within the
same process with zero copy. You're passing a little bit of metadata and just the raw pointers.
Two of the data buffers across the C boundary. You know, super simple, straightforward,
an arrow can support all the data types you like, you normally need and also supports complex
data types, validity, nulls and the whole of nine yards, all the complexity you need.
Now we have drivers that already are in the ADBC repo for snowflake, BigQuery, Postgres,
SQLite, already all had drivers. db, implements the interface. You can load,
lib.db.so, as an ADBC driver. And we're still building and working on more drivers.
And it's, you know, semantic release, so major minor, versioning, the whole nine yards, you expect.
And we maintain that's going to be ABI compatible across releases. So adopting the standard is something
that you can be sure of. It's not going to get ripped off from underneath you. And remember,
it's part of the Apache arrow ecosystem. It's a sub project of the arrow project itself, which means
it's all Apache 2.0 license. So where does this fit in the general area of applications?
ADBC is not looking to completely replace ODBC and JDBC in all use cases. It's not going.
It doesn't want, we don't want to. It's not going to. If you're, if you have a use case that is transactional
and truly row oriented or small amounts of data, you're probably better off just using ODBC
and JDBC unless you're already using arrow elsewhere in the stack. Because if you're already
using arrow elsewhere in the stack, then the conversion cost to and from arrow is going to be
a benefit to using ADBC to avoid that. If you're not already using arrow elsewhere in the stack
and you have a truly row oriented workflow, then you might not get much benefit. It's not going
to be better or worse. There's not going to be much benefit to the work to convert. But if you're
doing bulk analytics, if you're coming from a column oriented source, if you're doing large amounts
of data transfer, that's where ADBC will shine. It's vendor neutral and it can use all the different
protocols and because it works by plug-able drivers, it can be ultimately any protocol that's necessary.
Now, one thing that some people do bring up is they've heard of arrow flight, arrow flight
SQL and they wonder what is ADBC in that context. It's the difference between a protocol and
API. ADBC is a client protocol. It's how the application will interface with the data system.
Arrow flight SQL is a protocol. The network protocol that will be used by say the arrow flight
SQL driver for ADBC to do the actual network traffic and transport to the data system. However,
that means is that if you're building the database, you're building the data system, an API,
whatever. If you expose an arrow flight SQL interface as your interface or as an
available interface on your system, that means you get all the client sites more for free.
There's already a ADBC arrow flight SQL driver. There's already arrow flight SQL clients
in many languages and so on. So if you are exposing an arrow flight SQL interface,
you don't have to do any extra work for your clients to be able to access your system,
they can just use all the existing open source stuff. So for example, influx DB and Dremio
and Apache Doris all expose arrow flight SQL interfaces. You can use ADBC arrow flight SQL right now
and use that to connect to execute queries and get your data results from all those systems
super fast, like significantly faster than leveraging ODBC in JVC. That said, there is arrow flight
SQL drivers for JVC and currently an open source one for ODBC is being finished. So if you
expose the arrow flight SQL interface, not only are you going to get the flight SQL clients
in the ADBC driver for free, you also get anyone who wants to use ODBC in JVC, get it to access
it for free too and you don't have to do the extra work. Instead, you can just help contribute
to the projects themselves. So an example, I mentioned that DukDB implements the ADBC API.
So DukDB while back when they first did that, they did a test. They just did a simple benchmark
to insert million rows and then you select star million rows of a TPCH scale factor one.
They use the ODBC driver to do it and then they use ADBC to do it. Now, what's really cool
about this benchmark in this test is because DukDB is an improcessing on the system.
There's no network overhead that or network in state instability that's interfering here.
This is effectively showing what the cost is for that amount of data of that transposition
because that's really the only difference here is the conversion from arrow data to ODBC versus
passing some pointers through. So some code examples. Nice straightforward simple way of
how you can use ADBC right away. Python, we have ADBC drivers that are exposed to the
epicellin libraries. We utilize DBAPI to win a interface and then add it and we add a fetch arrow
table or fetch record batch functions to it so that you can retrieve arrow data directly from it.
But you can also use any of the other DBAPI methods and it will convert them as necessary
that's what you need. We already have, you know, flight SQL, snowflake, postgres SQLite or all
right there. Nice simple connection interface. There's also the lower level driver we expose there.
With the driver manager, even if we don't already have a packaged ADBC driver for Python,
you can use the driver manager to load any arbitrary ADBC compatible shared object that you want,
which is what DupDB did. If you pip install DupDB, you can then import ADBC driver manager
DupDB, DupDBPI and it'll work. Because the driver manager is a simple wrapper that allows you to
load the arbitrary shared objects, which means that you can share these drivers across any languages.
In fact, many of the drivers in Go because it really is easy to compile Go to a static
we linked shared object that has almost no dependencies other than lib C. It makes it really
really great for distribution and passing around. So like the snowflake, big query flight SQL drivers
are all written in Go and then expose a C interface and are loaded and packaged with Python,
R, Ruby, and so on. And then we have the bindings for all the individual languages.
With the Go with the Go library, if you don't want to use, you don't need arrow directly,
we have a wrapper for the standard database SQL package that wraps around ADBC and returns,
and you can use it the same way as you would, you know, the scan library.
You can use it in R, nice simple and straightforward. In enable systems that have an ADBC driver
to for free get utilized in other languages that we have the bindings for without having to make
specific ASTKs for the individual languages. And then conceptually there's a lot of overlap
with the systems, you know, whether you're using ADBC connections and flight SQL, so it's
easier reason about. If you're used to reasoning about things at that level, the concepts are
pretty much the same. So, what would an ideal scenario? Well, ADBC all supports bulk ingestion.
If you have an arrow source coming in, you can ferry it into ADBC to do inserts,
which also means that you can then take your ADBC driver for snowflake, get a stream of arrow
record batches, feed that stream to ADBC ingest for ductDB, and release stream data from snowflake
into a ductDB instance, and you didn't have to materialize everything locally. It's streaming
it, you have a little memory footprint because you're producing an arrow stream, consuming an
arrow stream on both ends. And that will work for anything that has ADBC drivers. It also supports
partitioned results sets. Flight SQL allows you to do partitioning of results sets so that you
can have separate endpoints on how you fetch your streams of data. ADBC exposes an API to allow for that.
So, if you're a system, your database system puts data on different nodes so that you can
parallelize the fetching, you can expose that through ADBC's API. It's there, it's available.
You know, we all support transactional, you know, always standard database transaction,
workflows, and even supports using substrate plans if you're utilizing things at that level,
you can send substrate plans to the system and send it to ADBC if you're driver supports that
if your data system supports executing a substrate plan. So, that's pretty much the talk there,
that's ADBC for you. If you want more information, you want to learn more about Apache
arrow or ADBC, or any of those things, you can go to the websites, you know, the arrow dot
Apache dot org, and arrow Apache dot org slash ADBC. The documentation is all there, the bindings,
the information, the GitHub repose, everything you need to get started and to learn more.
And then, you know, not so shameless plug of, you know, I recently had the second edition of my
arrow book came out, which includes an entire new chapter that is on ADBC. Because when the first
edition was written, ADBC had just begun as a project. It was just starting. So, it's only
up one paragraph. When I went to do the new edition of the book, I expanded that into a full chapter
with code examples and insights and usage using examples and C++ go and Python. So, lots of
information and not just about ADBC, but also everything in the arrow ecosystem in general.
And, yep, that is hit.
And there's also lots of big sticker.