Okay.
Hi everyone, my name is Michiola Becker, and this is my colleague Gassan Mohamed,
we are both from Twintag.
At Twintag, we do what we call connected products.
Basically, we bring data to physical objects in the real world.
As part of that, we do have a need to handle data very flexibly.
We don't see ourselves as a database vendor, more of a database user,
but because of this flexibility, we do some fun stuff,
and that's what we want to present to you today.
So, we already had a very nice overview of what Arrow can do earlier today,
I think from Matthew.
So, I'm not going to go into this in depth,
but there's basically two things to remember.
Arrow is a in-memory format for common storage,
and then we have the wire protocol, Arrow West Rail Flight,
to put stuff over the wire.
So, what do we want to do in our business?
Basically, we have a whole bunch of data being stored in different formats internally,
but what we want to do is we want to give the simplicity to the end user,
basically giving them one grace-purpose to interact with all of our,
all of this data.
And that means we need to solve for the challenges in between.
So, mapping all this logical query access to the physical storage,
federating all the queries across that,
and then also the governments for the multi-tenancy and stuff in between.
So, today we want to show you a little bit how we tackle this in our approach.
This is the same picture with a little bit more detailed,
and so on the way on the left you have some of the access patterns.
We have SQL query like connectivity for our customers as well as APIs,
and connected for the common BI tooling,
which you can see here also is that it gives two different types of access patterns.
We have the SQL and the BI tooling, which is more all up related,
but then the API is then to be more transactional access patterns.
So, we kind of have to solve for both in our stack.
Yeah, you already see a little bit of hint of how we are tackling that.
We did the site and using data fusion as our query engine.
A little bit more on that decision later,
and then downstream you see all these different sources from relational databases
to stuff in object storage,
where we basically take incoming queries,
cut them up, and federate them through those systems.
And, you know, it didn't open the slides with the Apache arrow slide for nothing.
We originally started using that for doing some data pipelines,
but now that we're doing this query federation,
we're actually slowly trying to adopt arrow at every part of the stack,
all the way from the connectivity,
where we do have SQL flight connector to data fusion,
which is arrow native basically,
and then also to the downstream databases.
Now, including TIDB, for which I have something as well later.
So, data fusion, you also had a nice introduction already in the previous talk.
What did we like about this?
While we are a little bit more up in the stack,
you know, designing services for our end customers.
We ourselves are kind of still system designers,
you know, not really database people,
but we like to write services, you know,
that the form button hole and not duct taped together with all bunch of scripts.
So, being able to embed data fusion,
and therefore have that control really aligned well with our goals there.
It's also very well articulated by this slide that Andrew Lam made in one of his presentations.
You see data fusion as the LVM of databases, you know,
really a tool that you can customize and embed in your service
to do what you need.
You are kind of a couple of examples of that,
as well as what we are doing.
Probably way shout out to Andrew,
because he's one of those gold theorem containers out there,
what he did with fostering the data fusion community to what it is today.
It's a really impressive.
Normally, I don't like to add a lot of code into slides,
but what I wanted to do here is just give you a little bit of a flavor
of what it means to embed something like data fusion into your services.
To kind of make it less scary for people,
you do recommend that you try to do this yourself.
So, the very basics of this is,
okay, how do you teach data fusion, how to find your data,
and that's where the stable provider trade comes in.
You just describe what the schema of your data sources
and how to talk to it, how to scan a record,
and that's how you plug in the new data source into the system.
And then, you know, in our earlier slide,
we need to do a whole bunch of stuff to map between the external world,
the simplicity for our users and the internal data systems,
that includes stuff like remapping the names of things,
you know, what's a table called,
might be different for the end user than what's in storage, for example.
And data fusion is really nice,
but analyzer and optimize rules that you can ship to do things like,
you know, these renaming, remapping, and all kind of stuff,
very easy to plug into the engine as well.
And finally, I think at face value,
this might be a little bit underestimated,
but this is just a way that you kind of spawn data fusion,
how you set up what they call a session state,
basically describing how one data data fusion instance should act,
so what catalog there is, which analyzers to be used,
where to find the data, and this makes it very easy to set that up,
and you can do it dynamically, you know, depending on what request comes in,
you load the configuration for that tenant,
spawn the session state accordingly,
and off you go to querying.
So a little bit underestimated there,
but very powerful for the kind of systems that we rebuild.
All right, I said earlier in the talk,
that we have kind of a hybrid access pattern,
we have some more analytical stuff,
but also a lot of access patterns are transactional,
and one thing that happens is if you try to federate a query
to a remote system,
is that if you start doing fill database read for transactional systems,
that's often very slow,
what we need is to really get very specific data from the remote,
and so we've contributed this extension to data fusion
that allows you to do just that,
and I'll just from a high level point of view,
explain how it works, so you might have a plan like this,
see on the right the plan, a couple of joins happening there,
and it just so happens that on the right the circles,
part of the plan is sitting in one remote database,
so it could be tied to be, for example in this case,
so what does data fusion Federation do?
It analyzes the entire plan,
figures out which suplands are actually provided
by one single remote database mechanism system,
it cuts that part out,
and then it replaces it with an opaque note,
and that it will take that supland,
materialize it back into SQL or any other query language
that the remote supports,
and when the plan is being executed,
it will take out of its little part
and go in query the remote database,
and this allows you to do some parts of the plan remotely
in the database management system that has that data,
so it can do it very efficiently locally as such.
Yeah, and then if you want to expose your data fusion,
using a standards protocol like Apache RFLIS,
we made a little contribution there that allows you
to just spawn that up very easily,
and with a couple of these systems,
you can couple of these libraries,
you can actually throw together a whole bunch of different systems,
here is some examples from the community that people have built,
from this kind of signal pane of glass for data,
for AI purposes,
to people doing workflow automation,
and even some bad three people,
I think aggregating data over multiple types of blockchains,
so all kind of using the same components in different settings,
also I think showing that this is not created rocket science.
Again, I would really implore people to just try this out.
So with that in mind,
here are the links to all of the stuff,
obviously data fusion itself,
the federation, and the flight disk available server,
and there's also a contribution from the nice people at SpyCi,
we've created a whole bunch of table providers
for basic files,
but also for remote database systems,
which have the support for data fusion federation,
for efficiency sake,
and then you can also see there is a put request for
for TITB,
very much in the
thought set of automatic disk prep earlier,
let's try to get these database to adopt a patchy arrow flight disk
as well,
so that we can get the arrow flowing all the way
from the site down to storage.
All right, and then we just have a little demo
combining all these things together,
and I guess I'll show it.
Yeah, hello everyone.
Yeah, so we built this demo,
based on everything we were talking about,
and this is bound to the darker container,
when it's my school,
and the other is a post-crease,
and now we're trying to run a flighty school server,
and this is the mapping that my friend was talking about,
which we basically,
so each table is in different database,
one is in the my school,
and the other is in the school light,
and the product table in the post-crease,
and we have also mapping for the column as well,
so now we run the client,
so we run the client with the first query,
which is just a filtering with the query,
and this filtering is pushing down,
using the diffusion federation,
and we have also another query
which do join with the multiple remote database table.
If you see this query,
it has like a three table,
one of the orders,
and so it's users,
and it's showing left-chowing with the orders and products,
which each one is locating in different remote database,
and that's the structure for the whole demo,
and how we run this all component together
to have this result.
Okay, great.
If you want to look at the code at some point,
it's just link at the bottom,
you can figure out the entire demo,
basically everything I presented today is in there,
and yeah, it's not in same rocket science again,
so it definitely implore people to try and embed some query engines into your services.
All right, thanks everyone.
Are there any questions?
Yeah, in the demo,
you want to do that slide.
So basically, in our stack,
we have kind of a little bit of an object model
that describes what the external facing world looks like,
and this is what we use to steer the optimizers
to do the mapping between the external user facing
or high-level object model,
and then all the internal storage behind the scenes.
Yeah, there's a lightweight version of that
in the demo we published,
obviously internally we have a more extensive setup.
So I think the question is about the Federation,
how exactly the execution happens there?
Yeah.
So what happens is you look at the entire plan tree,
and the data fusion federation optimizer figures out,
it's a simple plan, so little trees in the overall plan,
but provided by the same remote source.
And then it cuts out that part, all those little trees,
materializes them back to whatever query format
to remote supports, it could be SQL,
or something like FluxQL,
for all the index that they are something like that.
And then when the plan has been executed,
it goes ahead and does those queries on the remote,
and then days of fusion itself,
does the final job of bringing all that together
and creating the final outputs?
Yeah.
Yeah, exactly.
So there is the simple ends are being executed
by the remote database,
and then all tying it all back together happens in data fusion.
So you try to push the compute as down as close to where the data
lives as possible, and where the indexes lives in it.
Yeah, exactly.
I think the question is a little bit about the default interface
in data fusion, so if I go back up.
This is kind of what data fusion provides of the box,
what you can do here is a little bit limited.
You can get data back, you can ask which columns,
you can do some limited filtering,
and then limit how much records we want to get back.
And that's obviously a little bit limited,
and many remote database systems are very,
way more expressive than this.
For example, this would not be able to support joins.
But the alternative here,
what we're doing with data fusion federation is,
by cutting out the supply and you can actually
plug in your own serialization for that supply.
So if you're doing something complicated in there,
you can actually materialize it back in whatever format
that the remote database supports to then execute it
with the full fidelity of the query language.
Get that back into data fusion, and then it will resolve
the rest of the plan.
Wait, I will not be able to understand this, I don't.
Yeah.
So I think the diffusion is a micecule,
or postgres inspired dialect,
and we are kind of ending on top of that.
Then within the federation itself,
you can get to choose in what dialect you materialize
the plan to execute remotely.
But outside of that,
we do just use whatever data fusion uses,
which is a simple ascule.
All right, thanks everyone.