Please welcome our next speaker, it's Alexander Valerking, CTO in Victoria Matrix, and he's
going to speak about working with file systems in time series database, welcome please.
Thank you for the introduction, let's start.
Let's meet, I'm Alexander Valerking, it's a registered and I'm also known
on the my nickname, Valerking, which you can find Google in Google and know more about me.
I'm found of writing a hyperformance code, and this code is optimized for low resource
usage, and I also created a few time series databases, optimized for performance and low
resource usage, and this database of course opensource, this is Victorimetric, Database
for Matrix, and Victorimetric for the database for logs, and as I said before, this database
is a fast and cost efficient, and let's, okay, they are not written in the rest, they are
written in Go, and let's talk about specifics of time series databases.
This database is contained at time series data, and that I see this is a series of samples
and each sample contains time stamp and value, and value can be numeric, in this case,
this database contains metrics, and value can be arbitrary blob, plain text, or some
structured blob, in this case, such time series name talks or events, and every time
series can have arbitrary set of key value labels, and these labels are usually used for filtering
and grouping, for simplifying the filtering and grouping, and the number of time series
and in typical database ranges from a few thousands to billions, and the typical data
ingestion rate for time series databases is up to 10 millions of samples or rows per second,
so you can see that such time series databases must be optimized for high data ingestion
performance, and typical queries in time series databases usually need to scan up to 10 millions
or 100 millions of samples, so obviously you need to have high query performance, and the number
of typical samples in time series databases is measured, not in billions, not in millions,
but in trillions, and the size of the data stored in time series databases is measured in
terabytes and petabytes, so obviously such size doesn't fit memory, and how to achieve
high performance for such databases, data base performance is limited by disk IO, if the size
of data doesn't fit memory, because you need to store this data persisted data to memory,
and so if you want to achieve high performance, for the database, you need to reduce the number
of disk operations, and you need to reduce the amounts of data read and read from disk,
let's talk about how to achieve high data ingestion rate for databases, any ideas,
just reduce a sync frequency, why, let's talk about why, because when you store data into files,
this data isn't stored in files actually, it's stored in memory in the old page cache,
by operating system and operating systems usually stores this data in the files in background
when they want to store this data, and the order of storing this data in files in disk is undefined,
it doesn't match the order used right as data to files, and when you unplug your computer from
the power, page cache data can be partially written to disk, and this means that you get
easy data loss or data corruption, so you need some system calls, I just have sync,
which forces storing all the data from all-page cache, written data to all-page cache to disk,
and that's why all the production rate that the basis must use a sync, the exceptions,
let's talk about the data, why you need to reduce frequency as I said before, because a sync is slow,
and it is slow not only on HDDs, but on SSDs too, why, because when you write even a single byte
to SSD, disk under the hood SSD writes and a much larger size of block, such as up to four
megabytes or even more, which are named to erase blocks, because SSDs, as it's hardware works,
it can store you one single byte and SSDs, so if you, that's why, expect only thousands of
fcps, even on SSDs, there are exceptions such as enterprise-grade SSDs, which are more expensive,
and they have special capacitors, which allow you to store the battery data, which is not persisted
yet when you power off the SSD, but on regular SSDs there are no such capacitors.
And the basic frequently used technique, which bypass all the page cache and directly write data
to disk, they are direct IO, but it doesn't help too much, because in reality, this direct IO
just replaces a sync, you just simulates a sync by yourself, so it takes the same amount of time and
resources, and also, additionally, direct IO doesn't work well on many popular file systems,
it has bugs and various issues, so how to reduce frequency for a sync at that base,
the first approach is to send stored data in the base in big batches, when you store data in big batches,
the client expects the success response when this data is persisted, so we,
per each batch, you need to issue only one f sync or multiple sync, so big batches,
small number of fs, and you have to, you can achieve higher data injection performance.
Another approach, commonly used in databases, is to merge concurrent inserts in a single
data block, and store it with a single fsync to disk, so for example, you can run fsync every 10
milliseconds, 100 fsps, so every insert can be delayed up to 10 milliseconds, while collecting data
from concurrent inserts, and then storing this data in a single insert in a single fsync.
The advantage of this method is that it is limited to 100 sequential insert per second
from a single client, in this case, of course you can run thousands of concurrent clients and
achieve much higher injection performance, and as a popular approach is to use right-of-head
block, how it works, you set stored data in a just a data in a right-of-wall, a sync it, and after that,
stored data in the ground in other data structures without a sync, and if a power of occurs,
you can replace this wall of the restart, and do not lose any data, and many of the
databases use this technique, but it doesn't reduce the sync too much, because you need to make
this fsync after every storage of data into wall, in order to prevent from power off,
and that the basis uses various data tricks for reducing the same set wall, for example,
they can buffer data in RAM, and then periodically rises data to wall, with a sync, of course,
or they can write data to wall without any of the sync, and periodically run a sync on this wall,
and both cases are bad, because they can lead to data loss and power off, and also they
even worse, they can lead to data corruption, if power off happens in the middle of a sync,
and in this case, you are on a way, which parts of data are persisted, and which are not persisted,
and you need to protect from this with some checksums, so there is another approach,
it just admits that there isn't just a data can be lost on power off, and there are practical cases,
when it is a K, for example, when you work with observability data, such as metrics,
logs and events, if you admit this, then you can buffer data, listen to ingested data in memory,
and periodically flush it to disk with a single sync at the end, and if the data is synced to disk,
then it is safe from loss and corruption on power off, but listen to ingested data can be
lost on power off, and so let's talk about a sense of also that the basis and querying pass,
they provide you ability to quickly find the needed data by the given prefix, okay,
and that the basis you use indexes for this, and indexes, data structures, which map keys,
to locations of the data on disk in database cases, and indexes usually provide
or look and see complexity, where n is a number of interesting indexes, and the most frequently
index types in databases are based on B3s, B3s is a tree, that structure where every note contains
thousands of entries, and notes are usually aligned in size up to from 4 kilobytes to 64 kilobytes
and stored to disk at once, and when you add new entry to B3, it usually requires updating multiple
notes in the tree along the path, where you add this note, and note the entries usually can
be three entries contained, not the data row data itself, but pointers to the data and this data
stored somewhere else in other files, and another popular index type is based on LSM3s,
this is the structure contains files, every file contain data sorted data, and such files
periodically merge in bigger files in the ground, and this means that these files usually always contain
entries sorted entries, and they contain values itself instead of pointers to values which
are stored somewhere else, and let's look at some practical cases, the first case is a high data
ingestion rate, when you use B3, then you need to update B3 notes, multiple B3 notes,
per each data ingestion ingestion entry, this means that in a worst case you need to make a
split, since you're ingestion ingestion entry, and also since when you use B3, you are relying
on wall, and you need additional disk bandwidth for wall, and when you use LSM3s, then you just buffer
multiple entries, thousands of entries in memory, and store them in this at one F-sync,
and you do not need any wall, and save for this guyo, the second case is reading a well in the
index order, which is frequently used, in B3 case you need to read data from many different
locations on disk, because in B3 doesn't contain usually rows, as I said, row data it contains pointers,
those data can be scattered across the disk, and in LSM3s, as you remember, the data itself
is stored in order, in order, and this means that you just read data sequentially with a
much smaller number of disk operations. The next practical case ingestion big data volumes,
which do not fit our memory, in B3 case, is increasing significantly the amounts of disk
operations, because if some B3 nodes are missing in memory, then this means that you need to read them,
in memory, and if you have no final space for all the B3 nodes, then you need to read some
this B3 nodes and store them to disk, so you increase significantly the disk write operations,
and read operations, and in LSM3, the frequency of this write delete operations doesn't depend
on the size of data base. In other cases, in B3 case, you need some complicated HMS based on
NDCC and transaction IDs, which need a lot of time and efforts, and in LSM3s,
snapshots are instant, for any database sizes, why? Because LSM3 files are immutable, and this means that
you just make hard links for all the LSM files and get snapshot, and it is instant for any size of data.
In other cases, network attached storage, storage is such as NFS or object storage. In B3 case,
it doesn't work at all, because you need to update files and all object storage, and network
file systems, but for updating files, and in LSM, it works perfectly because files are immutable,
you just write files, and delete already mere files, and do not update them any time.
As for this space user, in B3 index, isn't compressible, usually, and row values are compressed
not so well, because the order of this values are coherent, and this means bigger this space
usage, and in LSM3s data is compressed much better, because it's sorted, and sorted data
compresses better, and you get all of this space usage, and of course, you get
a discrete bandwidth in LSM3s compared to B3, and this means higher query performance,
for queries which need to read the robots of data, because you need smaller amounts of data,
and package them, and then process. Let's talk about row oriented with column oriented storage,
row oriented storage, you store rows on this one of the other, it works great for select star,
type of queries where you select all the row fields, but such queries are not usually used in practice,
and row oriented storage has very big overhead for typical production queries, which select
some subset of fields, and perform some filtering on subset of fields, and this results in low
compression rates, as for column storage, you store data per each column separator in separate files,
and this gives you big overhead for select star queries, but they are not used frequently in production,
and but it has much smaller overhead for production queries, because it reads data only for the
given columns which are mentioned in select and where, and it doesn't need to read to other data,
and per column data can process much better than per row data, because per column data values usually contain some constants,
or small number of unique values, and this means that you get faster performance and to let this space.
Let's keep this because I don't know time, and let's go to the conclusions,
as the first conclusion is thin and slow, and the best way to reduce a thin crate is to admit that
the recently ingest of data can be lost, this doesn't apply for typical all-lob data, all typical
databases such as mySQL and Postgres, but it applies well for time series databases, typical time
series databases which throw metrics, locks, and events, and just admit it, and let's increase use
this, let's list this kio, let's invest this, and column rate storage uses list this space,
and this kio, then row oriented storage, even if you include the ground merges overhead,
and when you combine it, let's invest this column rate storage, you get the best performance
for big data bases which do not fit memory, and this popular data bases which use LSM-3s
and column rate storage, a kioly house, of course, DAGDB, and of course, vector metrics and vector
locks, and I recommend you investigate in the source code for these data bases, because they are all open source,
and let's say thank you.