Okay, as you're getting settled in, we, next speaker, we have Nicoleta from Fresha,
this name of the company, and then, should be talking about query federation and all
all right, we'll get started, okay, no, hello, no.
Hello, I am Nicoleta, I come from London, and I will talk today about query federation
in modern, modern all of the databases, right, so I want to go through a journey together
with you on what query federation is this, okay, so, yeah, okay, all right, so today I will
talk together with you for a journey of what query federation is, why it is important to
be able to have a unified sequence access across various data sources and why query federation
is very compelling in query, the data lake, lakes, and in a way that allows us to build
the unified lake of architecture, so let's start simple from what query federation is, and essentially
it's just a way to query various data sources without needing to replicate data without needing
to copy over anything or bring everything into a single place, it's a way to simplify data
architectures, so without query federation and if you have multiple data sources that you need
to do, what you'll have to do is to just query the first data source, for example,
query the second data source in just, during them manually, either in the back and up or
the data analyst in some sort of spreadsheet, but at the same time another possibility is to
as I was mentioning is to just bring everything into a data warehouse, right, and I think
this kind of diagram is quite familiar to a lot of people because it has this idea of bringing
the operational realm from data sources into the analytical of state into some ingestion
there and from their analytic engineers would take over and model the data.
The problem with this is that the data stays within this internal format of the warehouse
and if the warehouse doesn't support itself some use cases that you require you either need
to get the data out and process it elsewhere and build even more data pipelines that
need to be maintained and so on.
Oh, sorry, can I will do, okay, is it better now?
Sorry, okay, I will just keep it like this, so sorry about it, I can repeat just the introduction
so essentially with query federation we want to be able to not copy the data over the in a
single warehouse and we don't want to also to either query it in each data source individually.
Now this engine, the federation query engine allows essentially to do this work for us, right?
So it will just be able to split of federation query into sub queries, send those sub queries
to the individual data source is created a data plan that an execution plan that allows
this separation and then collate the results within it internal engine.
The user in the end gets the final results.
Now there are some challenges because nothing comes for free.
So when you federate across various sources you have the problem of performance which
depends on the slow source.
Also, every data base comes with their own data access patterns, security policies and also
network limitations when it comes to softening data when joining in between nodes.
So for example things like LTP sources or open search or data leaks.
How it actually looks in three of them from query execution perspective.
So when a client would fire a query, it will reach a coordinator, that coordinator will
analyze the data and will return then the planner will return in the best possible execution plan.
And result will be scheduled on various workers.
Each worker will be tasked to via the connector, connect to the source, get the results,
join them, shuffle the data if needed, join the results and return the final.
Now, I mentioned all of the databases and there's an example I gave start.
But I'm not entirely sure how many of you has heard about start up in the past.
Okay, two, three.
Okay, so it is an all of the database.
It is open source.
It is essentially a clone of Doris.
And it has MPP processing engine, vectorized.
And what is compelling about start up is this my protocol compatibility.
Right, because it's very easy to integrate in the existing ecosystems.
It is also federated query engine and it shines in data lake house integrations.
Now, in start of saturation is very similar in a sense with three.
But the rock couple of differences.
So clients also federated query, the front and cluster of nodes, which function as a coordinator,
would get the query, would determine again, will parse it, will determine the best possible plan.
And then we'll send the plan to compute nodes for execution.
Those compute nodes will also get the data, but while returning the results,
most of these results will be kept in an internal data cache layer.
And then the results will be able to be used in the subsequent query.
Now, in a nutshell, if we want to put together these two paradigms, the idea is very simple.
There are two different strategies for doing query federation.
And a trainer will optimize for as many data sources as possible.
Well, and this is very useful for external exploratory federation at whole queries and so on.
But if you need more like production, all up, and to put these workloads in front of users,
then probably optimizations that the all-up engines do by default,
would be very useful and necessary.
Why is the duration important specifically?
It's also because it is a compelling case for querying data lake houses in addition to other internal formats of the databases.
So, for example, and this actually enables a compulsory separation,
because a computer storage is just in cloud, in very cheap cloud storage,
but computer provision to have the data process, the data on demand, and only when it's required.
This led in the recent years to have this lake house architectures that are built on top of open table format.
Now, why I will use the iceberg, as an example, but any open table format might be used.
So, in this case, iceberg, it has a very wide support in the industry,
mostly because it has an open specification and has been adopted by very big companies at the moment, at least.
But, indeed, it's just a series of files and manifest files that go into some data files in the storage.
And readers need to actually rely on statistics, understand those statistics in order to have to be able to query the query them.
So, there is this separation between metadata plan and data plan, and in order for the query and engine to be able to read those data files,
they need to navigate across all those metadata.
Now, separating iceberg can be done by both of these in both strategies in both patterns, right?
The difference would be that we will have to scan tables directly in objects storage,
and that's why you will pay this price of opening the files at very time.
But, startups on the other hand, because it's an all-up engine, it can bring the external data internally,
and then cache results, cache metadata, optionally build materialized views on top of queries,
and this essentially leads to minimizing remote reads.
Now, the federation to iceberg from Starbucks is very similar.
The only addition is that you need this iceberg catalog to be able to say,
OK, I have a table name, where do I find the metadata?
And once I have the metadata, I can get statistics, I can determine exactly what files I can plan,
I can do partition training, and then I can just say, OK, I only need to scan very little files from the system.
And then I need, I can return the results of the user in a very performant manner.
Now, how it looks in practice, I gave here an example, just to see how simple it actually is, right?
Because in this query, which is really a query on top of a view, there is this joining between an all-up table,
which is internal to startups, and an iceberg table, which is staging the data link,
and then there was a drink, but the iceberg scan itself is very quick.
In this example, it's like 2 milliseconds, just because it relies heavily on the data,
on metadata, cache and data cache.
Now, and underneath all these view, there are more tables that lead in the end to the data sources.
Now, how star, how star ox itself addresses the challenges that I was mentioning before,
I already mentioned that this put everything in context.
There is the object latency, storage latency will be solved by data and metadata cache, cacheing, data freshness,
the raw strategies, like, hot called data separation, which I will explain in a moment.
File fragmentation is solved by just cacheing, and materializing some of the view,
such that you don't need to rely on compaction so much on table maintenance, essentially.
And schema evolution is just by default, because star ox can, as incredibly refresh metadata,
so once the table gets updated in iceberg, it will be updated in the example from Kafka,
or maybe if we need to do stream processing, we will do the stream processing using cling,
then ingested into star ox, for example.
And federation stays in the middle there to say, okay, if a user does a query,
that needs both very recent data from, did it happen like seconds ago,
but also data that happened like that was ingested like a couple of years ago,
that's very easy to do, right?
You have query federation that can query directly the called storage.
The internal tables are already there, but also we can bring into the engine,
as well into the user.
We can also bring other sources, for example, open search in this case to say, okay,
we have a query that where we want to search to do full text search based on users, for example.
And I query first, open search, then I enrich that with whatever was saved in the internal storage,
and finally I append the historical results.
Now, if we zoom in into all these 18-dimensional layer,
we can think of this architecture that we essentially build with one single all-up data source,
and you can see here that all these components are open source components, right?
Everything can be built on top of open source components.
These data, we can get an architecture that essentially it is layered,
so we have an injection layer that goes from postgres and then data getting to Kafka,
for example, from Kafka, it can be processed with a thing,
and then ingested into various other sources,
and then also from Kafka, it can reach the real-time layer backed by start-off internal catalog.
Now, in here, the common component is essentially this all-up engine, right?
And we can change the technologies, but the idea remains the same, right?
If it's not start-ups, it can be, for example, Apache Doris, or VLODB, or something else,
but the idea remains the same.
There is a need for all-up data base that can federate across both cold data,
and hot data that's internal, and maybe bring together within the engine,
other external data sources, for example, open search, or something else,
and this gives a unified and unified query capabilities across these sources such that
you don't need a system anymore, that is essentially need separate cold data,
for example, data versus hot data, they can both be served from the same engine.
If we can go with a couple of key takeaways, the idea is that, yes,
query Federation is very useful in orchestrating these queries across,
and leaving a unified SQL access, the comes with trade-offs.
So in terms of performance cost use cases,
but the idea is that industry shifts towards cheap, usable, and universal storage,
and the query engines as well need to adapt to be able to query these new lakehouse architectures.
And thank you very much.
I think I'm open-type.