So, welcome everyone. It's quite a sizable audience for a database benchmark talk on a Sunday
afternoon. My name is Gabber Sarnyos and in the following 30 minutes I will talk about database
benchmarks and what I learned about them while running a benchmarks standard organization.
A little bit about me. I spent a decade in academia, then I was building up a benchmark
non-profit called the IDBC and now I work at a database vendor. An overarching theme of my
career is doing database benchmarks and using them. Today I'm here in my benchmark non-profit
head and I will try to share the lessons that I learned as such. The talk is five parts. I will
talk about the need for database benchmarks. Then I will talk about two benchmark organizations,
the TPC and the IDBC. Then I will review some popular benchmarks in modern times and I will
leave you with a bunch of takeaways. So, let's get started. Why would you benchmark databases in the
first place? It turns out that databases and benchmarks are a pretty good match. First of all,
users usually care about performance. They spend a lot of time optimizing their workloads and if
the database can do this for them or if the database allows them to go further, they are quite
happy about that. So, the vendors are incentivized to have good performance or at least to claim
good performance. Databases are also mostly deterministic. Benchmarking the database seems reasonably
straight forward because you get usually the same query plan and some similar execution time.
So, it's not like benchmarking a large language model which is the lot more stochastic than a database.
So, at this point you may expect that you have a data set to define a bunch of queries,
you throw it at the database and then you write down the numbers of the databases performance
and this will somehow magically make database benchmarks work. Unfortunately, reality is a bit more
complicated than this. If you just do data set and queries, you will not really stress
updates in the database which is a crucial part of any database system. If you just run the same
queries over and over again, you get to limit it understanding of how the optimizer works.
So, you should also plug in different parameters into those queries and you need to somehow
tie all this together into a workload definition. The database is of course a piece of software that
doesn't exist in isolation. You have all these layers under the database, the five system,
the operating system, hardware, natural components, maybe a container, an emulator, some virtualization
layer, these all have interplays with each other that can have an effect on your results.
And finally, performance is important, but correctness is also important. You have to verify that
the results are correct and usually pricing is something that users deeply care about, so you need
to assess the pricing of the system. And speaking of being deterministic, yes, databases are most
deterministic, but reproducibility is still difficult. It's quite difficult to hit
reliably the same code path in a database system. And you have all these extremes of different runs,
cold runs, which are incredibly difficult to do because you basically have to restart the computer,
restart the database and then run the query to get a true cold run. And hot runs, which are
easier to achieve because you just run the same thing in the loop, but maybe they are not that relevant.
So when you are measuring systems, you are kind of in between, lukewarm and warm runs in a
system, and there can be orders of magnitude differences between these runs. Another surprising
difficult thing is that once you have done your benchmarking and you maybe have 100 million
little entries in your benchmark report, you would need to write them up in some sort of a summary
agregust statistics. And there is an excellent paper from Torsten Huffland Roberto Bell,
either they explain how to report these in a parallel computing system, which is of course
every computing system these days. And I cherrypicked two rules for you. Rule three says that
you should only use arithmetic mean, the average for summarizing costs. If you are summarizing
grades, like transactions per second, you should use harmonic means. And if you are summarizing
ratios, first of all you shouldn't, you should go back and dig out the original values from
the base data and then summarize that. But if you only have these ratios or speed-ups available,
then you should use the geometric mean. So we basically use all the Pythagorean means just to report
on some sort of an average in terms of costs and execution. Another thing is what to measure.
Microbenchmarks are quite popular in research. You measure one join operator or maybe just a simple
hash-proving. This is very interesting if you are implementing that operator, but for users,
it's more interesting to have macrobenchmarks, load the data and run a bunch of queries, or even
application-level benchmarks that measure the database systems performance and to end with complex
updates, which may be a more intertwined workload. So, this is all quite difficult even if you
have the best intentions, but let me tell you in the 1980s, database vendors did not have the
best intentions. In this period, relational databases were still kind of immature and had major
performance problems and something called benchmarking was quite common. This means that benchmarks
were misused, vendors were even implementing their own benchmarks and surprisingly they've owned
their own benchmarks and they both did about these results. An interesting chapter in this story
is 1982, their professor David Duitt measured a bunch of database systems, including Oracle.
He published those results and that didn't really make Oracle happy, so they put a close into
their license saying, you are not allowed to publish results unless they are officially sanctioned
by the company, and this became so popular that this is called Duitt close based on this little event.
But by the end of the 1980s, it was kind of clear that everyone doing their own benchmark is not
really doing any good, and there is a clear need for independent authority and the benchmark standard,
and thus the TPC was born. The TPC, somewhat confusingly stands for Transaction Processing Performance
This is a non-profit company that is based in the United States, and it was founded in 1988
with the mission to make standards for benchmarks and then to make official results for those standards
available. TPC's membership has fluctuated over the years, at its peak it was around 40 members,
now it's at 21 members, and you can see it's heavily dominated by companies based in the United States
and in China. With regards to the type of companies that are members, that's also worth mentioning,
you have the hardware vendors like NVIDIA, AMD, and Intel, you have these server-bedding companies
like Lenovo, Dell, and HPE, and you have cloud vendors, you have database vendors, so it's kind of an
interesting mix of companies. And what did they do? I picked TPC-H, which because this is their most
influential and most popular benchmark as a good example. TPC-H was released in 1999, and it
captures an ad hoc analytical workload for a whole sales supplier. If you look at the schema,
you have a supplier, you have parts up, you have line items for the entities that you are selling,
and you have 22 queries that do analytics over these tables. It also has some refresh operators
for inserting and deleting new data. The pillar of TPC-H is the data generator, the data generator
can create data sets in different scale factors. A scale factor is defined as the size of the data
set in GBBIC, so when we say scale factor 100, we mean it's 100 GBB of data. This is actually a
really good way to measure it, because CSV files are kind of timeless, it's really easy to understand
that it's a plain text serialization, and even though these days we are mostly using binary formats,
this is still very easy to understand. The original data generator was written more than 25
years ago, so it's written in C and it's not threat safe, but luckily a bunch of people last year
rewrote it in Rust, and you can now just pip install it and run it basically anywhere through Python.
And this is a really neat piece of software, and I think this is worth mentioning, because if you
ever need like 100 GBB of data at your disposal on your laptop, you can just run these two commands,
and if you produce somewhat meaningful data set in less than a minute, so this is even interesting
if you're not into database benchmarking per se. The queries of TPC are obviously quite involved,
I don't have time to go through all of them, but here is a really interesting example TPC-H
query one. This is a rather simple query. It uses a single table line item, it uses some simple
filtering, and it uses a bunch of simple aggregates like some average and count, and I like this
query because this is something that realistically every analytical database should be good at.
It doesn't have anything complex, you can implement it as a benchmark query in maybe a few minutes,
and if you just start out building an analytical database system, you can start benchmarking
it with this query quite early on in your journey of building the system. The way TPC benchmarks
works is that it has three distinct faces. You first load the data to your database, then you do
something called a power test. The power test means that you take the 22 queries and the refresh
operators and run them sequentially. This allows you to measure whether the database can do
inter query parallelism, whether it can exploit multiple cores for the same query. Then you compute
the geometric mean run time of these queries and you get a power score. Next up is the throughput test.
This is concurrent, you run queries and updates concurrently and try to achieve maximum throughput.
And then finally, these are put together in a composite metric by taking a geometric mean of the two.
And this weirdly named metric QPH is something that we used to track the performance of
analytical workloads. I dug this out of the official TPC results and it actually shows a really nice
trendline starting from 2001 until 2020. And what you can see is that basically the trendline
starts around 5,000 QPH points. And now it's above 5 million. So this is a thousand X or 41%
year on your improvement in terms of analytical performance that is at your disposal when you buy
a state of the art database system. TPC H has been so influential that people try to do some reverse
engineering on it and break it down on first of all why it is such an efficient benchmark.
And second of all, how you should excel on it if you're a database vendor. So if you read this paper,
it defines these choke points, these kind of well-defined bits of difficulty that the benchmark
in codes. And you can see how the improvements on those choke points would improve your results on TPC H.
So for example, if you improve aggregation, it will have an effect on all of the queries
and the more outside the fact on some of the queries. If you improve joint performance,
you will not impact queries 1 and 6, but you will deeply impact queries 9 and 18.
Another benchmark from TPC is TPC DST and I brought this more of as a cautionary detail.
This is a more modern and improved version of TPC H, so it's for analytics. It models a retail
product supplier with multiple sales channels, it has dimension tables, fact tables and it's more
realistic than TPC H in general. The problem with TPC DST is that it has 99 queries and on average,
they are almost 2,000 characters per query. So it's 200,000 characters almost just for the query definitions.
The deletes are also more complex. You have these cascading billetes from one table to another.
If you delete something from sales, it will scatter into the returns fact table, so they are more
difficult to implement efficiently. As a result, even though TPC DST was released in 2011,
the first audit took seven years for someone to overtake. It was done by Cisco and it was the
scale factor 10,000 result. In the next year since, there was a scale factor 100,000 result by Oli Baba,
and in 2021 there was a record breaking result on TPC DST by Databricks on the 100,000 scale
factor. This actually triggered a subsequent benchmark war with Snowflake, with Databricks and Snowflake
putting out press releases, blog posts and social media on why their system is better, more efficient,
and better in terms of price performance. This is really interesting, great, and I'm also
happy to chat about this after the talk. So I said that TPC acts as an independent authority and
they have official results. The way you achieve it is that they have an auditing process that you
have to undertake. If you want to audit your database, then you become a test sponsor and you commission
this audit from the TPC. You have to run the experiment yourself and you have to write it up
in the full disclosure report yourself, and then you hand it over to a certified TPC auditor
who validates the result. They have pretty high powers, they can not only rerun your experiments,
but they can do additional checks. Back in the day, they could fly to your server farm and
plug the server out of the socket to see whether the durability of the server is really there.
And they do all sorts of these checks to make sure that there are no benchmark specials in your
system. What is a benchmark special? Well, vendors are incentivized to do well on TPC benchmarks,
or sometimes they built in operators just to do a TPC query well. And the way the auditor is
trying to catch this is, for example, by Gorbling some of the queries. So this is Query1.
And this is Query1 with all the column names and all the table names replaced with some random
encoding. This prevents systems from using hard coded highly optimized operators and query plans
if they use this method of detecting which TPC query is running. An important part of TPC
benchmarks is pricing. TPC pricing is a specification that's almost 70 pages and it captures
how you should price the total cost of ownership for a given database system. And this has three major
components. The first is a software license, which can be zero if it's free and open source.
The three year cost for the hardware and the cloud service if you use a cloud vendor. And then you
have a three year maintenance with enterprise great support. Now I personally have two main criticism
with this pricing. The first of all is that I believe the enterprise support is very strict.
It's quite common, especially in analytics to run database systems without 24-7 enterprise support.
So more of a next business this day support would be sufficient and it would keep the prices
more realistic. But my bigger problem is that this is really an outdated way of thinking about
databases. People who run databases in the cloud where the storage and compute are disaggregated
and you can elastically scale the compute depending on your needs. Don't rent out computers for
a fixed three year period. And you can actually see the effect of this in the pricing that
pricing's that are reported. For example, the Databricks audit that I have mentioned from five
years ago, they run it on 256 machines and because of this, the TPC pricing specification just
assumes that you would run this out for a three year period and this yields a half to some of
five point two million dollars. This is obviously not realistic number for doing data processing
on 100 terabytes. 100 terabytes is not that much in these days. One interesting thing that I noticed
is that I put it out the TPC number of audits in the last 15 years or so and they have a moderate
negative correlation with the US federal funds interest rates. This is because the closer we are to
zero interest rates, the more money is available for venture capital investments and when there
is money in the VC market, the vendors are rushing to get TPC audits and try to one at each other
in terms of performance in the hope of getting more venture capital. So that's kind of an interesting
observation, but of course it's not the full picture. All right, so we arrived to LDBC. What is LDBC?
Well, it is a non-profit founded in 2013, registered in the UK and its goal is to accelerate
progress in graph data management by facilitating this sort of pro-competitive work, where you
collaborate on standards and then try to be the best one at implementing that standard. The name
LDBC reflects this focus on RDF processing back in 2013, so it's the linked data benchmark council.
Back in 2011, when the idea of LDBC came up, it was very similar to the 1980s of relational
databases. Graph databases had major performance issues, there was also no standard graph
curve language and there was just no common understanding of what a graph database system should do.
So some vendors and academics got together and created this organization. Today in 2020
we have 18 members, you can see that again is dominated by companies based in the United States
and China and we also have cloud vendors, we have software vendors database vendors and consultancy
companies. LDBC is heavily inspired by TPC, so we do complex application level database benchmarks,
we define data generators that can do different scare factors up to tens of thousands of
terabytes and tens of terabytes and we design our benchmarks even based on the TPCH chokepoint
methodology, so we took the TPCH chokepoints, we extended them with graph specific chokepoints
and then we made the benchmarks such that they covered these chokepoints. We have a stringent auditing
process and we have adopted the TPC pricing specification. Let's take a look at our most popular
benchmark, the social network benchmark. This uses a data set that's capturing the Facebook data set
in terms of distribution and in terms of the number of edges a given person has. It's actually
based on a 2011 paper. Of course we not only have people in this network, we also have the content
that they create, this is in the form of messages and this forms this nice graph where the left
side is a network and the right side is a bunch of trees. So what can we do in this? A very simple
simplified example query is this, we have two parameters a name and the day and what we're
interested in is that starting from a person with a given name, if we do two hops along the
nosages, friends and friends of friends and then we check the messages that those persons
authored. What are the messages that were created before a given day? So let's go through this with
an example. Let's say we are interested in ban and messages before Saturday. In this case we do
one example along the nosages to get to Ada and Carl, one more to get to Finn, Eve and then
we hop over to these messages and we get a filtering operation to get the ones that were created
before Saturday. So visually it's very simple but for the database it means efficient lookups,
joins filtering, so it's quite an involved operation. And of course updates are also interesting
and in the graph space this is even more so. We can do simple inserts that's not that special,
we just add the nose edge between Eve and Gia or we decide that Gia is creating a reply to message
M3 on Sunday. This is pretty run of the real stuff but where graph databases can be interesting
is when it comes to deep delete operations. So for example if you decide to delete the person Eve,
then that could be only deleting the node and the outgoing edges but if you are privacy
conscious and you need to delete all the data that person ever created that also means removing
all the comments that she ever wrote and maybe the replies to those comments and the likes to
those replies. So it can be kind of a replaying effect and we just removed one node and ended up missing
almost half our example graphs. So deletes in graphs are quite heavy eating operations.
How does this all add up into a benchmark workload? Well we have a preprocessing phase where
you can tweak the data to your liking, then you load it into your database, do a 30-minute warm-up
and then a two-hour benchmark run. The goal of having this two and a half hour period is to have
some realistic representation of an actual life system where the database is running for a
prolonged period of time and not just running a bunch of queries. The workload mix composes of
three different types of operators. We have complex reads like the one I have shown you,
we have short reads like looking up a certain message or getting the friends of another person
and then we have the update operators, the inserts and the deletes. And the way we replay this
is actually quite interesting. So in the generator we simulate a three-year period of the social
network's activity. We write down the timestamps and then we replay them. We try to compress time
based on something called a total compression ratio. So for example we have four operators here,
along some logical assimilation time. And then if we have two threads to run the benchmark,
we can compress them by putting updates one and two on thread one and update three on thread two.
It's actually more involved than this because there are all sorts of data dependencies. For example,
you cannot like a message before a message was created. So those have to be tracked by the benchmark
driver. But once you're done with the tracking, you can do the scheduling. And then we schedule
everything else to follow these update operations. For example, we do maybe 10 updates between
each query one, 15 updates between each query two and so on. And then the queries themselves also
send off kind of a short read repo. This is to simulate that you have written a message,
but now you're maybe looking around in the social network to have something more to do.
If you start running an actual database and start hitting it with this workload,
what you will notice after some time is that maybe the database is not able to keep up
with your workload. So the start times of the queries will start drifting back in time. And obviously
this is not great. What we want is that the database should be able to keep up. So we have this
on-time requirement that 95% of the executive queries must start within one second of their schedule
time. Otherwise, it's an invalid run. All right. So we have the data set the queries, the updates,
and the workload mix, how do we do all that? Well, as I said, we had strict auditing guidelines
similar to TPC. We have a list of certified auditors that you can work with. And unlike TPC,
here the auditors are always rerunning the benchmarks, and they are writing the full disclosure
report with all the details. One sign of success that we have noticed a few years ago is that people
told us that they have seen these RFPs, requests for proposals for internal critical procurement
in companies that necessitated official ADBC benchmark results. So as a benchmark organization,
you sometimes get to become the arbitrator of whether a system gets picked or whether another system
gets picked. You can see a list of audited results on our web page and just create a similar
plot to the CPC-H plot. Here is how our transactional workload ran on SCF Actor 300,
the 300 give you by data set over the last five or six years. What you can see is that it started
from around 4,000 operations per second, and it grew to about 30x to more than 130,000 operations per
second. What you can also see is that these are all vendors based in China. Apparently China has a
very healthy venture capital ecosystem for funding projects like this, and these companies were basically
one-up-ing each other in terms of performance. So this kind of driving competition thing
really works with benchmarks. We did quite a few audits, so I have a few lessons to share about this.
One thing that we have noticed is that performance variability is huge, especially in the cloud.
We had issues because of limited disk space, we had issues that you get a different machine,
and the performance of the IO is widely different, but we also had issues where it was a
machine that was good. It was enough space on the disk, but on the second run it just had an
inexplicable performance drop. This is obviously something that the auditor needs to take into account.
We also noticed that sometimes never misaligned expectations because the vendors expected the auditor
to finish the implementation, and in general the audits always took much longer than expected,
sometimes three or four months. We tried to mitigate this by having an elaborate questioner
of dozens of questions that we do during the pre auditing process, but this is still a
difficult challenge to get the audits done on time. A few words about organization, so obviously
running such an operation really needs some sort of an organization structure, some sort of
financing, and so on. ADBC had four distinct phases in its lifespan. It was originally
an EU project with almost three million euros of funding. Then it costed for a few years,
most living off of favors of vendors and academic institutes. Then we actually started collecting
money from our members, maybe $2,000 from regular members, so we had small fees, and from last
year we up this, and now we have had the 150,000 euro per year income. This looks nice, but it
actually has quite a few problems, and the problem is banking. We were kind of doing rounds at different
banks, and ended up settling on one of these new FinTech nail banks that you use from your phone.
Neobank one kicked us out without explanation. We were moving our funds to Neobank 2,
and they blocked our account, and they triggered an urgent and extensive KYC,
Neoyor customer check, because they said we have an unusual structure, or they didn't say that,
but the reason was clearly this. What this means is that they really don't like organizations
that have 25 directors from more than 10 different companies. We passed this check, but then we did
the restructuring of the organization, and we were happy for a few years, until they said,
we cannot accept money from your member X. In the last couple of weeks, I tried to migrate to Neobank
3, they denied our application straight away, so you can see it's kind of difficult to do. Maybe
we will go to London into a traditional brick and mortar bank and open an actual account,
with an actual person instead of fighting support. That is to be seen.
So it is a difficult thing to run a benchmark organization, and it's also difficult to have a value
proposition. We found that vendors join us to have a seat at the table when the benchmarks are designed,
and one particularly interesting reason is the defensive reason they want to prevent the benchmarks
to be used against them. So they are there to prevent a new benchmark being defined that puts them
at a competitive disadvantage. Interestingly, many of our vendors say, yes, benchmark for a nice,
they have maybe achieved their goals, but now should focus more on queer languages,
graph schema, and de-emphasize the benchmarking part, and just keep some data sets and synthetic data
generators. To reflect this, LDBC has actually rebranded as GDC, so we are now called the
graph data console, and we still do benchmarks, but it's more kind of overarching theme on graphs
instead of just a benchmark organization. Alright, popular benchmarks. I have two popular benchmarks
to talk about. One is clickbench. This is something that's a macro benchmark for analytical
database systems. This is something that's designed and maintained by Clickhouse, and this is designed
to be an easy-to-use benchmark for analytical workloads. The data set has a single bite table of about
100 million rows, and 105 attributes, so 10 billion plus sales in the table. It's 75 gigabytes in CSV,
and it's webinars coming from the index metric data set. The queries focus on scan operations,
aggregations, and lookups. You can see a bunch of queries starting from the simple
comes star through some filtering, through more group-by-aggregate queries. Clickbench is
philosophy, as much as I can tell, is to strive for simplicity, so there are no scare factors,
no career parameters, only simple type, and very few restrictions on the implementation. The whole
benchmark is kind of fast-paced, and you should be able to do a git clone, go to your directory,
do the benchmark, and finish within 20 minutes, including downloading the data set.
Then you can open a poor request, and they usually review it within a few days.
This allowed Clickbench to scale up in terms of the results quite quickly. They have more than 75
different systems, with 125 different configurations, and I think this is a massive success in terms
of stimulating competition and just getting people their results out there. They have somewhat
standardized hardware, so there are a few common setups that are at the center of the competition,
but you're welcome to bring your own setup maybe on premises, laptop, GPU, you name it.
If you open Clickbench, they have this elaborately, their board, and if you zoom in, it allows you
to compare systems. I picked my SQL and Postgres, because none of them has a reason about
chance of being good on a fully analytical benchmark, and you can see that you can compare their
performance, you can compare their system sizes, the database size, that is, the size of
to loading, the load times, the individual query times, and so on. Another benchmark that is reasonably
popular is called H2O.AI. This was originally created by Anguretsky, and funded by H2OAI.
Then it was abandoned, and now fork is maintained by dark db labs, where I've worked during my
day job. It has two workloads, focusing on group buys and joins, and three scare factors, 0.5,
5, and 50 gigabytes. You may have still floating around on social media, it has this really
recognizable color for squares, and these long bars representing the first runtime for the code
run, and the second runtime for the vorm run. This is a synthetic data generator, and the workloads
themselves are really simple. So, five basic groupings, five advanced groupings, switching between
low and high cardinality, grouping on different types of strings, and so on. And there's a joint
workload, it's five different joint periods. We are the maintainers of this benchmark, and we keep
getting proposals for extending it with different functionality like window functions. This is actually
becoming a dilemma for us. What do we do with it? If we keep as it is, then it's slowly going to
become a stair benchmark. If we extend, then people will say, well, you just accept the proposals
that make your system look good. Again, you know, an independent benchmark authority would be nice,
it's kind of difficult to maintain a benchmark as a vendor. So, to kind of run it out,
the takeaways. I think benchmarks do carry tremendous value, they capture a common understanding
of your field, they drive innovation, but they are difficult to finance, and even though vendors
use them extensively, a lot of that is internal use, and it's difficult to get funding for it.
Here's my idea of benchmark in one slide. If you ever do that, maybe it's worth visiting,
and I would just like to leave it the fact that you maybe shouldn't bring your own build your own
benchmark organization from scratch. AdPC or GDC is open, please reach out if you would like to learn
more. Thank you.