We're actually receiving a lot of basic advice for us.
We're going to go a lot of different things that a lot of intros might miss.
We're going to make them soon and soon we can call them soon.
For those of you who weren't talking about the talk,
for any of my names you can find,
I'm a developer advocate that's so nice that I exclusively focus on
how to talk to experts, so I'm glad to see you today
to share experts and methods.
So the key things off, what is I've heard,
why should you actually share?
You probably heard by experts, right?
It's the main way to do this.
It is an open game.
What about that?
That was surface built to handle large scale.
These are the effects in the data lake.
When you think about cuttings, we're in the past.
We've got data lake now.
We've heard from the past we had our wonderful modelists,
our data warehouses, right?
The data lakes were kind of response to a lot of the things
that weren't quite growing well.
We've got data warehouses, mostly in the past,
having that modelists.
But there were some mistakes about data warehouses
that we didn't like.
What the expert needs is to turn the answers to the concerns
of the data lake and concerns of data warehouses.
And we're combining the best things of those into a simple context
with data lake.
What's so clever with our needs, right?
And so what it is, what I forget is,
it adds extra layer of metadata on top of the files
because we already want to be storing in our data lake.
And by doing so, it allows us to select that data into
an organized and more structure paper.
So effectively we're adding some more structure
that people must set on top of the data lake.
So we get some benefits on that.
I mean, it's one system we're building on top of the lake.
So we still get some of the benefits
of if being a single person true,
like we were just doing the data warehouses.
We get the consequences and most cases of the storage.
And then we get the flexibility to use whatever performance
that we actually want for our data.
But we get governance and initial structure,
and obviously we're moving into those data.
So it also means that you know what's a vendor
or lock-in that you would effectively get a lock-in.
The warehouses, you could take your ice per table,
wherever you want, you would have a tool
that you want better or better, right?
Because that's actually like things up.
So it's really about to go through it.
So I learned isn't to be a really technology
I was there trying to unlock the power of a data lake
or you may have heard of the type of security
or not the lake.
But we're going to be focusing on experts
in the talk of the tool and things that you can get
in the benefits of ice search.
And we're even more into these features
as we go through more of this talk.
But I want to be able to face a tool
of what we expect to be able to do.
Okay.
So for that, we're going to focus on these kinds of experts.
The very basic, thank you for the two people
that sent in a little bit.
So since we asked for some of these,
the very basic thing you need to know about ice-creamable.
And to those of you who work here this morning,
you've got that crash course on the architecture
where you take it a little closer to the sun
and actually take a look at what actually
makes ice-creamable.
So I think that ice-creamable is the layer
of metadata on top of the files we want
to be storing in our data lake.
Okay.
So we have some data files.
And what we're going to do is you're going to track
with metadata.
So we're going to work from right to left here.
So first of all, we have our mammoth files.
So mammoth files are going to point to the data files
and also, as we read files,
we're probably going to come to these more
written rows here.
And more importantly, mammoth files are going
to have two types of specific about the data files
that they're tracking.
And the point thing to note here is that a mammoth file
is only going to keep track of the subset of the data files.
All right.
So it's going to be a number of mammoth files
for a data set to make sure that it keeps track of a lot of similar data files.
And so by doing this, by having me just to take
the arms off of these columns that were stored in our data files,
you're able to layer on very, very plain.
Effectively, it's going to be an off data that's not available.
All right.
So we can track with a maximum of the data that you keep out
actually, okay.
But we can hear a little more questions with our theory later on.
Next up, we have a mammoth list.
An example of a family list, a mammoth file.
But it's a little more important that each mammoth list
is going to list out a mammoth file that we want to use.
I think I need to have a snapshot of the data
I gave them from time to time.
And then look at your device.
What's your name?
I think none of the microphones are working today.
So this is not a surprise to me.
We're turning it off.
Okay.
No idea what just happened, but hopefully it's good.
All right.
And then from there, we have metadata files.
So the metadata files are going to track the snapshots,
which then point to their associated manifest files.
So we'll be keeping track of multiple snapshots of this table
as it evolves over time.
And then we're also going to keep track of some non-data information
here, like the partitioning scheme that we're using,
the actual schema of the data that we're storing as well.
Okay.
But that's not all.
So that's the entire metadata layer.
And how it interacts with the data layer.
But there's one more key component that's needed to make this
a little more database-esque.
And that's the catalog.
So the catalog here is going to keep track of the individual
tables that we are storing.
And specifically, they're going to maintain a reference
to, for that given table, where is the latest metadata file
that we need to be aware of?
Where does it exist so that we can then access this later on?
And the catalog is also going to coordinate updating that pointer
to a new metadata file every time we want to evolve the table
and add more data to it.
Okay.
So speaking of catalog, there's actually, it's pretty important.
Okay.
It might sound like an easy component to add on there.
But it really is worth thinking more about iceberg catalog.
This is a whole.
So they're keeping track of, again, that most recent metadata pointer.
But they serve a purpose beyond that.
So at the most basic level, they're there to make sure that we can find it
and interact with all the iceberg tables that we're storing.
And more importantly, ensures the consistency with the multiple
engines that might be trying to work with those iceberg tables
at the same time.
Okay.
And they do that with atomic operations to actually update that metadata
pointer every time we want to update the tables.
So it's simple, but they're incredibly important to make sure that
our tables are updating in the right way and functioning properly over time.
And an important thing to note here is that the iceberg project
doesn't actually ship with an implementation of a catalog.
That might sound problematic.
But they do provide an interface, thankfully.
Okay.
So along with several implementations of clients that can talk to catalogs
like Hive, JDBC, and more.
So that's what you'll see a number of options out there.
Some of which are listed on the right of different catalog
implementations.
So as per the interface, the only real requirement
if an iceberg catalog is that, as we already discussed,
it provides a way for you to give back that current metadata file.
So we know where that is.
And then provide a means for us to interact with the tables,
like listing them out, creating them, dropping them, and more.
And I don't know if we're taking, are we taking questions in the middle
or at the end?
Does it matter?
Up to me.
What is your question, sir?
We should slide it.
We don't know if it's called like files,
but if it could look, it's a sulfur component.
Okay.
Yes, that would be a service.
Oh, I'm sorry.
The question was on this slide here.
All of these, the components on the right and the metadata
and the data, those indeed are files.
And the catalog is an application, the separate application.
It would be okay to summarize the function of the catalog as being
a kind of database for your metadata files.
It is not storing any of these metadata files,
effectively.
It is providing pointers to those files.
Yes.
All right.
So, back we were on catalogs.
They are necessary part of iceberg as a whole,
but iceberg doesn't ship with one.
So that might sound a little problematic.
So you might want to choose one of the other catalogs out there.
And how do you choose?
That is a great question.
As with most problems in life and software,
it depends.
But I've conveniently put together a couple questions to ask yourself,
so that you can help narrow down which catalog to actually choose.
So the biggest consideration,
the biggest question that you should ask yourself is,
how lazy are you feeling?
And by that, I mean, do you want to commit to running a catalog
that requires additional software or external resource?
Or if it does require external services,
is there a managed version?
So how involved do you want to be an actually operating this catalog
in the long term?
The next question is, speaking of other software,
the next thing you should consider is,
how are the catalogs integrating with other tooling that you want to be using?
Other environments, compute engines that will get into shortly?
Using certain catalogs can corner you into using certain cloud environments,
for example, or compute engines.
So you might want to consider that when you're choosing one.
The next question to consider is,
does it handle the basics?
Hopefully your answer is yes.
Does it do the thing that you want it to do?
This might come as a surprise to some of you,
but there's atomic transactions that we like about iceberg,
aren't a given for each of the catalogs.
Some catalogs don't handle multi-table statements very well,
or multi-statement transactions.
And finally, where are you running this?
Meaning is this a toy use case?
Is this a thing you're playing around with?
Or do you want to actually ship this out to production?
Some catalogs do to the aforementioned limitations
aren't really recommended for production use,
or they come with a big asterisk with them.
So, down that we've seen at a high level,
what the key components of iceberg are,
our metadata, and our catalog.
The next thing we could focus on is how to actually work with our data,
and we're going to do that with some sort of query engine.
So, conveniently, the Apache iceberg project
maintains some implementation of open source compute engines
with built-in support for Apache Flink, Apache Spark.
Both are great options out of the box,
if you just want to hit the ground running with iceberg.
But what you choose would ultimately depend on what your text act currently looks like,
what your familiarity with either of these technologies actually is.
But as we'll see later, there's a lot of really great features,
especially with Spark, with the iceberg actions that are built in there.
Beyond that, the iceberg system ecosystem is very healthy.
There's a number of other query engines,
and tools available to you to actually interact with your iceberg tables.
This is not an exhaustive list, and it's ever, ever growing.
All right.
So, that's sort of the basics of setting the stage for iceberg.
But you might want to actually get started with it,
and you do so by using queries.
It's as simple as writing a query to interact with your iceberg tables.
And for a motivating example,
because we all love those, we're going to say that we're keeping track of data,
related to Arctic cruise bookings.
Isn't that fun?
See, that's so fun.
So, our create-table statement would look like this,
and it's going to feel very familiar, right?
It's a sequel.
I note that we're going to be using these syntax for the Spark sequel,
that ships with iceberg.
So, we'll keep tabs on here some standard information
for our Arctic cruise booking.
The booking ID passenger, how much it actually cost,
and the timestamp at which we're creating this booking.
And an interesting thing to call out here is that
the partitioning statement at the end of the statement here.
You might recall from earlier that I mentioned hidden partitioning
as a cool feature of iceberg.
So, what this means is that iceberg can use a transformation
on top of an existing column as the partitioning field.
So, you'll see here we are extracting the hour of that booking time stamp,
and we're going to partition the data based on that.
And it's going to come in handy later when we talk a little bit more
about selecting the data.
So, if we have a table, we should then insert some information.
And again, this is sort of a standard insert statement.
And then finally, what we got to go up,
want to go and select information from that iceberg table.
We're going to make things a little interesting,
and we're going to filter by a two week period.
We want all of the results, all the rows from that two week period.
But you notice here that we're filtering by days,
on that booking timestamp.
And so, the cool thing about partitioning is iceberg,
is that because we are partitioning by a transformation
on that booking timestamp, and we're partitioning by that,
when we actually want to go and select that data later on,
we can take advantage of that partitioning,
and transfer information on top of an existing field.
We're not creating a separate field,
and the compute engines know to make that transformation
when you're playing any filtering on top of that timestamp.
So, you can take advantage of that.
So, that was the very basic of iceberg.
You can leave now if you want, but I hope you stay.
Because I want to go a little bit beneath the surface,
and scratch that it's, see how these things actually work,
how these queries are operating when it comes to,
under the hood, with some of the files that we saw earlier.
So, when we look at our create table query.
So, for this example, we're going to imagine that we have a fresh instance.
We have no other files hanging out and no other data files.
So, when we issue that create table statement,
there's nothing here, right?
We just have our running catalog instance,
and we're running some engine of our choice.
In this case, we say it's Spark SQL.
So, when we issue the create table statement to the engine,
we're going to first parse that query, obviously.
And next, the engine is going to reach out to the catalog,
to fetch the metadata for that table.
That table doesn't exist, right?
So, we're not going to actually get any files here, any pointer.
And now, we can start to actually go through and create that first metadata file,
related to that table.
And remember, the metadata file keeps track of the partitioning,
spec that we're actually using, the schema,
that we're specifying based on those columns in that create table query.
And we're going to store that information in the metadata file.
We're also going to create a unique identifier for the table.
And yes, so now we have a metadata file.
We have no additional data files, right?
We haven't actually added any engines to the table.
So, we have no manifest file or manifest list to actually track those data files.
We just have our metadata file here.
And then, finally, the most important step is to update that pointer.
So, we have created this table.
And so, we want to reflect that in our catalog.
So, for our bookings table, we're pointing to our metadata file.
Cool.
So, when we go ahead and add data for an insert,
we're picking back up where we left off.
We have this table, but already created reference in the catalog.
So, we're going to submit the query to actually add the data to the table.
First step, we reach out to the catalog.
Find the latest, you know, where this file actually exists.
The metadata file.
And then, with the location of that file on hand,
the engine can go ahead and read the metadata file.
And I know you're thinking here is that we are adding data files, right?
Why do we have to think about the metadata file that currently exists?
Well, we want to be able to keep track of what this schema is,
based on that create table statement,
and make sure that if we're changing that, we can evolve it properly over time.
And also see what that partitioning spec is for that existing table.
So, we do want to make sure we read the metadata file to get that information.
Okay?
And then from there, we can go ahead and write our data files.
So, this will be written as parquet by default,
although you have flexibility to change that.
And this writing will be done according to the hourly partition spec, right?
So, if we were inserting a number of rows with across a number of time stamps,
we would be breaking it up into those partitions and writing them to separate data files.
And at this point, we could also define a local sort on top of the file
that we wanted the data that we want to be writing as well.
All right.
From there, we are going to work backwards and create that manifest file,
to track the new data file that we created.
And at this point, again, if we were only adding one row here,
but if we were adding many, many rows,
we would also compute the statistics on top of the columns that we have inserted.
So, we have the maximum and the bounds of all those columns,
and we'd be storing that in the manifest file.
And of course, working backwards again, we have our manifest list
that then just points to this one manifest file,
and we roll up those statistics into the manifest list as well.
And they also keep track of more statistics on how many rows were added
or removed during this operation as well.
And then from there, we're not going to touch the metadata file that we already exist,
but now we're going to create a new metadata file
that has the same information as the partition spec,
our schema as well.
But here, we're going to keep track of this first snapshot,
because we now have data that's actually stored in our table,
and this snapshot refers to that manifest list.
And then finally, most important step,
we're going to automatically update that pointer in our catalog,
so the booking's table now points to this specific metadata file.
So, next up, our select.
So, similar sort of process, we submit our query to the engine.
We're going to reach out to the catalog to see where this metadata file exists.
We know it is this one at the bottom with that snapshot,
so we're going to go read that latest metadata file.
And when we're reading the metadata file again,
we're going to figure out the schema,
so that we can prepare to actually send back data according to that schema on our engine.
The partitioning ski, so we can start planning out the query,
and actually accessing the proper data files,
and then the latest snapshot, right,
where are we actually reading from,
or what version of this data that's available to we want to read from.
So, knowing the latest snapshot,
we only have one here,
and it's pointing to this manifest list,
so we can go ahead and read from this manifest list specifically.
And so now from there,
if we had more data in this table,
we'd understand all of the manifest files that are belonging to this snapshot.
And then, and also at that point,
we can scan as part of the query planning,
understand which data files we actually want to be accessing
to receive the query for the proper filtering, right?
If we're filtering here on that two week period,
and if there were more data outside of that,
we could prune off some of the data files, okay?
And then we can finally access the relevant data files here,
scan the data and select the correct rows,
and send that back to the engine for their computation.
And so, since we just saw it actually in action now,
so this is where the hidden partitioning comes into play with that select query,
right?
We're filtering for that two week period,
but we have actually partition the data based on our.
So, just a little bit more on it here,
is that the engine is aware of how the data is partitioned,
and so it can conduct that transforms
when we're actually doing that filtering,
and then that is passed along.
So, it's hidden partitioning in that it is hidden from the select
that we don't actually have to be aware of that, right?
So, I wanted to go a little bit deeper into him partitioning
and what it actually means for the users,
so if you think about how you might partition data
in the past, a lot of the partitions fields
are going to be derived from an existing field.
So, if we wanted to do an hour partitioning,
based on a timestamp, we would probably actually do that transform,
and store it in a different column,
then partition on that and have both of those columns kind of co-exist.
But when it comes time for the person to actually select that data
and make use of it, the analyst who doesn't care
how you partition that entire, that table,
they're not going to know that it would be way more efficient
for their queries to use, you know,
one of those fields over the other, right?
And so what happens is that they ignore that,
or don't, or not aware of it,
and they're not getting any of the gains that they might get from
knowing of the partitioning.
So, icebergs, in partitioning,
eliminates that issue,
and also brings some additional functionality.
And I want to note that iceberg can do this
because the partitioning isn't tied to any physical structure
of how the data is stored.
In other words, it's not relying on that directory structure.
So, what do we get about,
get out of this partitioning, this isn't partitioning,
there's a lot of things.
The biggest thing is that we can partition the data
based on a transformation of these existing fields.
And again, they're going to be applied by the engines during
clearing planning.
And so what this means is that also,
we're going to save space when we're actually storing the data
because we don't have to have a separate column
that stores this transform data.
It's done on the fly.
And so, we're reducing what we're actually storing in those.
Another win is that we're protected from another class of bugs
where the derived value isn't written properly or updated properly
or stored properly.
And finally, the business users don't have to play
guessing games on what they actually have to,
what fields they have to query in order to gain the benefits
of that query printing query.
But that's not all.
We have another partitioning trick of our sleeves
with partition evolution.
So this means that if you want to change
how you're actually partitioning your data over time,
you can't.
I'm just store the current partitioning spec.
We're actually tracking all of the history of the partitioning
specs within our metadata files.
And so multiple partitioning specs
can coexist within one table.
So in the case of a multiple partitioning spec,
we'll just have different the filters of the predicates
will be applied differently based on how those files
are actually partitions.
And again, this is hidden from the user.
You don't actually have to think about that
when you're querying the data.
So before we wrap up queries,
another cool feature of iceberg is time travel capabilities.
Meaning that we can query the table as of a certain time
or snapshot.
Because in those metadata files,
as we actually evolve the table over time,
we're keeping track of the old snapshots.
So you can actually go back and see what that table
looked like at a certain point.
And so the first thing you have to do
is actually see which snapshots are available.
And you can see that by querying the system metadata table.
And so you'll see a lot of information on the snapshots
when they were last made current.
And then from there, you can actually select
either of these things when you actually want to query the data.
So as of that timestamp or as of that specific snapshot,
ID.
So how this happens under the hood and here,
we can assume that we have a couple more snapshots
that are relevant to us.
So if we are going to use the snapshot ID to query,
again, that's going to be parsed by the engine.
We read from the catalog.
And we know that this is our latest metadata file.
But suppose we're actually accessing our zero snapshot.
Okay.
So from there, we know that we are going instead of our S1
manifestless, we go back to the other manifest list
and read from there and collect the correct data files
that are then sent back to the engine.
And I promise we're rounding out queries now.
Bonus one for you.
So now that we know how selects work and how inserts work,
we can have a merge and up-sert functionality.
And the traditional example here is that we have a staging table
for our bookings, our active cruise bookings.
We want to periodically merge those into our bookings table.
And so when you think about this sort of functionality,
this query is effectively a few in one, right?
We have our select to see what data we actually have
in our bookings stage table, then it's an update for the relevant rows.
And then it's an insert for the other ones, right?
And so the select and insert are handled exactly as we discussed earlier.
But the update is a little more interesting.
For updates, we can either update things in place, more or less,
or we can issue delete and then reinsert the data.
So these are effectively your options in iceberg and which you choose
will affect how efficiently your read and write operations
will actually perform later on.
So this is going to take us a little bit deeper
and I suppose some additional considerations that you might need to think about.
And so an important part of how your iceberg tables behave over time
is handled with copy on write and merge on read.
So at a high level, these define how your iceberg tables handle updating
and how it deletes data.
And so data files in iceberg, they're immutable.
There's no surprise there.
So when we need to make an update, we can't just access that file
open it and add a new row or change an existing row.
We instead need to make a new file.
Some sort of new file.
And we can handle that in multiple ways.
So the first is with copy on write.
It's what it sounds like.
During the write process, if we're updating rows in a file or adding new rows in a file.
During that write process, we are going to make those updates
and copy the entire file over to a new one.
Okay?
If that sounds inefficient, that's because it could be.
It could be. That's the question.
We're going to copy everything and update only the rows that need to be updated.
If we want to do some sort of update or delete information.
So you might be rewriting a 500 megabyte file.
Just update one row, which, well, that doesn't sound very efficient.
But for, you know, for updates and delete, it can be efficient and efficient.
But for reads, then that's really efficient, right?
We just have to read that new file and we're done, okay?
So it's copy on write is good if you're handling multiple,
both deletes often and you want your files to be nice and neat for your readers later on.
Okay? If you want to do more efficient sparse deletes,
this is going to be handled with merge on read.
So here, we're going to add delete files into the mix,
which tells us which rows to ignore.
And so with this mode, we have a two-step process for our updates.
We first issue a delete in a separate delete file to say ignore that row,
and then we create a new data file that says,
here's the, we're inserting that new row, okay?
With the, with the updates to it, okay?
And then when it comes time to read, well, now it's a little more complicated,
because we have three files to resolve against each other.
One was the new update, the delete and then the existing file, okay?
So it's faster for our rights, but then we have the trade-off
is that we have to put in a little more effort for our read time, okay?
And so the other thing that we have to think about here
is that now we're inserting a lot of small files, right?
And so file IO is the biggest killer of your query performance in iceberg.
And so small files, we don't really want them around, okay?
So I wanted to cover here just a little bit of the considerations
around small files.
We just saw that there's a potential downside of that,
especially when we're considering deleting data.
And so, but it's just a natural thing in iceberg,
we're going to have small files at some point,
but we want to be able to reduce them.
It's not just a problem with more done read, okay?
But generally, we want to reduce these files,
and so the solution is here, has fewer small files.
Great, it's simple, just do that.
And so how do we actually do that?
Well, you can use copy on right, generally,
that's going to help reduce the number of small files you have floating around.
Wonderful, but that's going to kill the performance of your rights, right?
So to reduce the number of small data files that are going to be created from the get-go,
we could also try to be smarter about our partitioning, right?
Because if you have a lot of partitions,
you could potentially be creating that many data files every time you insert
do a bulk to insert on your iceberg tables.
So you might want to be smarter about your partitions
and reduce your partition size,
but again, there's trade-offs there.
Okay?
So we can actually fix the number of small files that are floating around by doing compaction.
Okay?
And this is an important part of just regular maintenance on your iceberg tables.
So the thing with compaction is that we're going to read in a bunch of these small data,
these data files, and then we're going to write them to essentially larger files,
again, just from a single partition.
Okay?
So this compaction mechanism ships with iceberg has a lot of configurations available to you
for what the size of the output files should be, how the tasks are actually operating,
what compaction strategy you use.
So there's a lot of different levers that you can pull here,
just to make the resulting iceberg tables just a little more efficient for your reads and writes.
Okay?
I need to say this generally something that you should be doing regularly,
as part of your maintaining your iceberg tables.
And then also we have a rewrite manifest files, sort of compaction,
just to reduce the size of those manifest files as well.
Okay?
So I know I probably have like 10 seconds left,
and so this is the end I got to get into it.
So I have a QR code that links to my link tree,
and then there I have a bunch of resources related to iceberg and just my blogging and where I am.
But the most important thing on there that I have linked to my link tree is iceberg summit.
Okay? If you want to learn more about iceberg, this is your chance.
The event is on April 8th and 9th April 8th in San Francisco on April 9th.
It's virtually.
So the virtual conference is free to attend.
So if you really want to broaden your knowledge about Apache iceberg,
I would encourage you to register for that,
and join us at the very least on the 9th if not in person on the 8th.
And then the other thing to make note of is that the call for papers for iceberg summit is still open.
So if your company is doing something interesting with iceberg and you want to talk about it,
or if you know someone who should be talking about it,
please consider submitting to the call for papers.
It is open until February 9th.
I mean it, no extensions.
So yeah, so I hope to see some of you submit there.
And at the very least, I hope to see many of you join us virtually on April 9th.
So with that, I am done. I promise. Thank you so much.
Thank you so much.
The next session is going to start in 10 minutes.
I'm going to have you out of your mind.
We're going to be doing a session for accelerating questions.