All right. I'm going to get right into it. I'm climbing again. I work at DuckDB Labs. I contribute
a lot to the DuckDB iceberg extension and I'm going to be presenting DuckDB in the cloud
a simple powerful SQL engine for your lakehouse. So let's get started. What is DuckDB?
Learn MIT, license, open source, analytical database. We have zero dependencies. And we can
install on any major platforms, Windows, Ubuntu, macOS. It's a super easy install. If you
like your CLI, you can just curl and you're off to the races. We also have a Python client
along with 15 other clients. Java are Rust. You name it. We probably have it. So what does
that all mean? Well, it means we're super portable and there are a lot of different deployment
methods. So you can just install it on your computer, start querying your favorite part
k files, your favorite CSV files, and dump it into a DuckDB database file. On top of that,
if you need a little more resources, you can deploy it in the cloud. Favorite EC2 instance or
whatever, if you need more memory, more storage. But since we have zero dependencies, and we
have a small binary, you can also scale it down to a serverless function. So you can put it
in a Lambda and then you can start reading files on an S3 bucket, whatever blob storage,
you prefer. On top of that, you can have multiple readers from a DuckDB database file
and your favorite blob storage. But what we cannot do is multiple read writers on a database
file and blob storage. So you might be thinking, why should I use this in the cloud if you
can't have multiple read writers? Well, because we're in process database, if we have multiple
read writers, there's a lot more fine-grained locking content and that kind of goes against
that would hurt our ability to do great analytics. Thankfully, a couple of years ago,
a lot of people started putting a lot of their data just in blob storage on par k files and S3 files.
And from that, you had open table formats get developed. I'm talking about iceberg, talking
about Delta Lake and what's great about those is you get these basically database table
like properties just in file storage and multiple databases support reading from these open
table formats including now DuckDB. So we now have, you can now read and write to these data
open table formats in these blob storage files with DuckDB. So this is great. This enables
multi-player updates on object storage. Unfortunately, it's only limited to a single table,
which hopefully I can get back to. So what does iceberg look like in DuckDB? Well, it looks like
you're pouring just your regular CSV or par k files. If you know where your metadata.json file
is, you can just do a select star from the iceberg scan of the metadata.json and iceberg does
all of the merging of the schema information and all of the data files and what you get out
looks just like a regular table. This is just line item from tbch. A one problem with the iceberg
format is that it is like all files and if you have fast-moving data, there is a lot of
contention around, okay, who gets the insert now, what is the most recent version of the data.
So what happens, what has now happened is there is now a catalog layer that has been introduced,
which is the way of saying, okay, this catalog, this database now controls who has the most
who gets to update this table with what data. So if any of these clients want to update your
fast-moving table that exists all in blob storage, they get to write all of the data and then
they get to ask the catalog, hey, update this table with my new data and the catalog will say,
yes, okay, now this table has been updated or the catalog will then say, no, it hasn't been updated.
And some of the popular catalog providers now are Lakekeeper, Polaris, Nessie,
AWS glue Amazon S3 tables, these are all in RCI for ductDB iceberg. If you've been paying
attention, oh, and this is what does it look like in the ductDB iceberg extension, you can create a secret
using our secret, yeah, using our secret functionality, you can attach to, in this example,
it's S3 tables, your S3 table Amazon resource name, type iceberg endpoint type, and then you're
off to the races and you can dump all of your CSVs and park caves into an iceberg table.
If you've been paying attention, you know that I've mentioned iceberg OTS open table formats
and a lot of table state exists in these arrow files. If you've known that, I just want to
mention duct Lake has recently been developed at ductDB, which takes a lot of that metadata state
and now puts it into the catalog into the database. And that's a lot more efficient.
It's still a little bit under development, but I'm happy to announce a touch statement,
super easy. Happy to announce that we will be releasing a major version next month, February,
and that's, yeah, that's my talk. Thank you so much.