All right.
Thank you.
Are we going to sound check, too?
People can hear on the stream.
All right.
Well, I'm going to start with a confession.
So the real reason why Evan and I are here today has to come and enjoy everything that
Belgium has to offer, the good beer, the good cheese, the good chocolate.
But we're also here to preach the gospel of open source, which I think is going to be easy with
this crowd.
It looks like we have a lot of very open source enthusiast here.
But we're going to be up to talking about Apache supersets and let's kick it off.
All right.
So let's start with some interesting shows.
Yeah.
Hi.
I'm Evan and Resecus.
I'm one of the projects.
So I am here for the beer for real.
But yeah.
Yeah.
You've got more beer cred than I do.
And I'm Max.
I'm the original creator of Apache superset.
You might know me to have heard me place is on the internet.
Talk about Apache Airflow.
If you're familiar with that, assume we have a lot of data and enthusiasm in the room.
So I started at Apache superset back in 2015.
It's almost like 10 years ago.
And since then, I started a company coming to you called preset that helps kind of foster
and offer services in and around superset.
But today, we're really focused about talking about superset.
Yeah.
And of course, one of the things is, you know, global switch day.
If we can talk you into one thing and one reason to come halfway across the globe.
It's to switch from proprietary BI tools to open source BI.
How many of you are paying for superset?
Yeah.
How many folks are already using open source BI?
And how many folks are still paying or your orgs are paying for proprietary BI?
Everybody, everybody's got some Tableau licenses.
And some of what we want to be preaching today.
And the core message is that open source in this area, business intelligence, open source, data visualization,
has gotten extremely competitive due to the work of the community over the past decade or so.
And is extremely competitive and pretty much all aspects of what you might want from a business intelligence tool.
And ahead in many ways, you know, there's a lot of advantages you get from open source that you'll never get from an open source vendor.
And we'll talk about some of that today.
Yep.
So really quickly what we're going to cover is all of this.
What is the product?
We're going to look a little bit at how it's built.
We're going to talk about how you can tweak it, which is the fun stuff with open source.
And then, of course, how you can get your hands on it.
And, you know, talk your people into using it.
The start off with exactly what it is.
If what is a patchy superset, so it is, of course, open source we wouldn't be here.
If it was not open source enterprise ready.
Maybe not everyone here and works out enterprises.
There's probably students and college attendees.
But by enterprise, we mean it is very much.
So a patchy superset is an application that you install and that you can use for you and your team, right?
And open source we have, you know, frameworks and pieces of infrastructure.
We have things like databases, small libraries.
But in this case, a patchy superset is an app.
It's a web app that you can install on your network and serve your entire organization company team with very easily.
And then business intelligence, if you're not familiar with the term, even though it is a, you know,
it's kind of a dated aging term that probably started like 20, 25 years ago.
But what it means really is like data consumption, data visualization, dashboarding, right?
So superset will get a little bit deeper into it.
But superset is a tool that enables you and your team to, essentially, like connect a database,
consumer data, rights of the SQL, if you're not a writer, a sample of dashboard, share it with people.
So it's really about data exploration, visualization, creating interactive dashboards.
Now, of course, being open source, we need to talk for a second about the Apache software foundation.
This isn't a patchy project and what that really means is that no organization, like preset or any of the other people that own,
or maintain it are actually the owners.
Apache owns the repo, Apache owns the trademarks, and they provide a governance model that allows us to responsibly maintain the software and maintain the community.
And they provide all kinds of other resources like legal coverage and all sorts of other.
Yeah, in many ways, like it keeps a commercial organization honest around the open source software that is around.
So you'll see, you know, big companies at some point kind of shift their license and things like that with the Apache software foundation.
There's a really clear governance model around things that keeps everybody kind of fair and honest.
And it's all based on the meritocracy as open source should be.
Yeah, it's all about individuals, not companies, which is great.
So we have to play by the rules of engagement to be patchy way.
This is the slide about me, so I'll already introduce myself, but maybe I'll talk a little bit about the genesis of Superset and how it came to be.
So about, yeah, 10 years ago or so maybe like a little bit longer than that.
I was at Airbnb.
And a lot of the tools that we wanted to use for self did not exist on the market.
So we had tablo licenses.
We look at different BI tools, but like none of the tools that existed out there would totally serve our use cases.
And some of the issues is we had picked these big databases at the time, like Apache Druid, Presto, and Tablo did not play very well with these databases.
And we had a set of like very kind of unique use cases.
So we decided to build our own and like often an open source, like you need someone a little bit more delusional and kind of crazy and think like,
why don't we, why can't I build a new state and new thing from scratch?
That's going to what happened in the REST's history.
Just a little bit around what we do, so part of what we do is we, you know, people use all sorts of databases.
Every other data is a new database that's created.
It's a C% plays very nicely with all of the databases that exist in the world pretty much.
That means like whatever database you're using for your project, your company, you know,
superset can most likely speak to it very well.
And just being open source to, it really fosters, you know,
extensibility, connectivity, and aggression.
So we support all databases.
Next slide.
So if you use everywhere, so one thing about superset is we are one of the fairly big open source project by any open source metrics.
A lot of stars on GitHub, like lots of contributors, super high velocity.
Lots of different organizations contribute contributing to it, which is part of like,
you know, when you make a decision to commit to an open source project or software for the long term,
it's good to know that there's a really strong community around it.
And this is very much the case with superset.
So have anyone to talk about some of the components of the bits and pieces?
Yeah, obviously, the point of the tools to build highly interactive dashboards and allow people to quickly get insight into any of the data that you have available to you.
And but working from the bottom up, there's a workflow that typically happens when you're building these dashboards.
You would start obviously with connecting your data.
So like Max said, we connect it pretty much anything that speaks SQL, but even the things don't speak SQL,
you can connect through some sort of intermediary normally, like a presto, you know,
or drill something like that, some other query engine.
So we can connect all sorts of stuff.
You just put into URI here and then we drop you into SQL Lab, which is a fully fledged SQL IDE,
where you can obviously write SQL queries, but you can share those with your co-workers.
You can get a query history in case you were wondering what you changed in your SQL,
between last week and this week or something like that.
But that's where you basically select all of your dimensions and metrics and then you start to build up your data set.
The data sets, one of the key entries is superset.
We have kind of a data set centric philosophy where you really want to use SQL to kind of build up something that's going to power your videos or power your dashboard.
And we do that through either virtual or materialized data sets.
And then when you create those, we have this data set editor, you see, which is a light semantic layer.
It allows you to annotate your data.
You can add in extra calculated columns and metrics and say what's temporal or what's the primary temporal column, things like that, to kind of curate what is in the data set.
Once you have that, we go into this, which is our chart builder.
And we have somewhere hovering around 40 or so different visualization plugins.
We support any kind of visualization library that's built upon JavaScript.
We have many of them integrated and you drag your columns from your data set to populate the visualization controls.
So you select your dimensions, your metrics, you can apply filters and all sorts of other display properties of any given visualization.
And then once you've built up enough of these charts, of course you drop them into a dashboard.
And dashboards are not just a drag and drop collection of charts, but they also provide a lot of other means of interactivity.
Like if you click on a bar in a bar chart and it's a cross filter that will be referenced in that filter bar on the left.
But it also applies the filter to all of the other charts that receive it.
You can create and share filters in that left filter bar to apply.
But who, well, anything you want to any of the tabs and other dashboard components that are available there.
And then once you've got your dashboard and it tells the data story you want to tell.
Often one of the things people ask for is how can I share this on the periodic basis with my team.
So we have this alerts and reports feature that allows you to send dashboards as essentially screenshots to your Slack workspace or to an email address.
And you can do that on a scheduled basis.
Or you can write some SQL query that'll set some thresholds as like an alert.
And it'll then send the screenshot of the data for your tune to review.
Maybe I'll think it's slightly different take to if you go back to the previous slide on on this.
So I know the talk right before is a DBT talk which is a way to sequence build a dagger SQL.
The next talk after this one is an Apache Airflow talk that's also about writing data pipelines.
So it's not necessarily as linear as it may seem here.
But we assume that you have a database.
Some flat-ish data sets on data structure that are pretty much like typically, you know, large data sets with a lot of column.
Normally, I can make checks and dimensions.
Based on these data sets, you can do exploration, the explorer that we just looked at,
assemble some dashboard.
And then the SQL IDE, sometimes it's part of the workflow, sometimes it's kind of on the side.
It's totally possible to use super send get a lot of value from it without necessarily writing any SQL at all.
But we're pretty kind of SQL centric as a tool.
And if you want to talk about the API and SQL.
Yeah, so I'll talk a little bit about this.
So pretty much so because we're an open source solution and really often it's built by engineers for to support entire teams,
but also engineers pretty much anything you can do in super set through the GUI or the UI.
You can use my microphone, that's position one.
Okay, I'll bring it up a little bit.
Is that better? All right.
So yeah, pretty much anything you can do in the GUI and the UI.
You can do through the API.
So you can define and create dashboards and charts, alerts and reports, all this stuff programmatically.
So good API coverage.
There's the component.
So the building blocks of super set.
So what we use to put super set together are accessible through component libraries,
friend and library, so you can also build your own application using bits and pieces of super set.
All right, I'll talk a little bit about what is pre set.
So I think I think that Robert, a friend, Robert from Altunity gave a talk this morning about how to create a commercial entity around your open source product without selling your soul.
So we're very much in that that mindset as well.
In many ways, like, there's some really good incentives and interactions between the commercial entity around the open source project.
In many ways, the fact that pre set is selling a managed service around super set allows us to go and invest and grow the open source project.
She's really positive.
I'm going to go quickly on the commercial stuff, but we essentially offer a managed service that is kind of, you know,
so if you don't want to run super sound in your own and run your own, be on call for it.
You know, you can use a managed service.
If you want to try it, it's going to be easy.
It's going to take super set for a test drive.
Then we also provide multiple work spaces.
So usually when you deploy super set yourself, you have one super set instance typically,
but we allow you to spin up multiple of these for various reasons.
You might want to put them in different regions for compliance purposes,
or you might want to give different instances to different teams or departments to give, you know, separated,
or you might want to do different workflows like dev stage product and kind of promote your business intelligence assets.
We'll see how this thing works a little bit.
Yeah, so I don't want to go to the here.
I know the audience fairly technical, so we'll look a little bit about how works this,
this super abstract, logical model of how this, this work behind the scene is, you know,
you have a database at the bottom.
Super set is going to the center of it.
It's able to, and it's equal against your database, and it allows you to create also,
try it's in dashboard that you can publish and share with your, with friends in your organization.
Now look at a tiny bit more on the internal.
So there's going to be a line on there.
Here's the back end.
So we have a python back end with slacks and SQL off-commies, stuff like salary.
There's a little bit of pandas to do some protein in there.
There's a thing of more than a hundred and fifty python libraries that are
powering this thing behind the scenes.
On the front end is mostly a react front end react app.
That's using all sorts of data visualization libraries and kind of
bringing that into a very cohesive type place.
So that's a high level, kind of.
So we're north of 10,000 then p.m. components on the front end.
We have a Node.js side card that allows other different operations like asynchronous query operations.
There's some web socket stuff that gets built in there.
So probably not too much is going to shift into Node land, but there are a few of them.
Yeah, it's like, it has become such a large web application.
That's multi layered.
So there's a little bit of Node in here in the back end.
But it's not central to initially what we do.
And it's mostly to manage the web socket.
So that, you know, well, so I wish I had a better architecture.
I let me repeat a question.
What is in a superset?
And what is the actual architecture?
What does it take to run this thing?
We don't have a good diagram handy.
I can say it's just going to on the top of my head.
So there is a Python web server that's running a flash application.
As part of it, there is a salary, a synchronous, kind of worker back in.
There is some optional components.
So there's a metadata database.
Multiple caching layers for results sets and larger results set to export CSVs and things like that.
And then I won't get into the front end because it might take all day.
You can deploy it in a number of means.
You can deploy with Kubernetes.
There's a Helm chart.
There's Kubernetes operator that's coming out.
You can use Docker compose or you can do kind of pip install.
Yeah, the easiest way if you want to deploy it today on your laptop,
you would just clone the repo and run Docker compose up.
And then if you want to know what's happening behind the scene,
you open up that Docker compose file and see what's under there.
There would be, I think, out of the box, there would be some redness and postgres back in front
and stuff and, you know, few bells and whistles.
Yeah, and that's how you get all this stuff out of the box.
But a lot of people obviously besides running their internal business intelligence tool,
they want to, you know, tweak it a little bit.
This is the joy of open source, right?
So you want to make it what you want it to be.
And there's a lot of things you can do beyond the normal,
like, let's just run superset and look at some use case data.
You can, you can use embedded.
So we provide a SDK called our embedded SDK that allows you to install a superset dashboard
as a React component.
So you can pass a guest token from your host application and, basically, authenticate.
And it follows all the RBAC rules and everything and shows your data in situ.
And you can really make it look like your brand.
So here it is embedded in somebody else's website.
And you can feed it and all sorts of stuff to me.
Yeah, I'll say a thing too about this.
But like, you know, if you build a web application nowadays,
there's often going to be a requirement to have a portion of your app that serves analytics
to their user of your app.
So if you're building a SaaS application or any web app,
you're probably going to want to have some sort of dashboard in there.
And we see it more and more that people use superset as a way to power this part,
the analytics interactive part of their application.
And then you can do that through embedded.
Of course, it's got to match the look and feel of your app.
So you can talk about theming a little bit.
Yeah, so we provide CSS templating.
And this is surprisingly one of the most popular things people have come to our website
for is more advice on how to do CSS work, which is something we don't provide.
A lot of warranty on because the DOM of any web application is going to change over time.
But there are a lot of very safe things you can do to theme your app however you want.
You can change colors and logos and kind of go crazy with all of the little granular details of how it looks.
And you can do that one off for a single dashboard or you can create a CSS template and reuse it across your instance.
You can also create visualization color themes that are either categorical or, you know, gradient on a spectrum.
So like hop to cold kind of things.
And if you're using open source superset, you can do that in the configuration file.
And create a bunch of those little color palettes and code.
Or if you happen to be using preset we provide a GUI so you can do that, you know, more interactively.
And of course, CSS customization is not themeing.
Themeing is one of the things we're working on right now.
And it's going to be done hopefully.
And if you must.
And we'll see.
Yeah, we'll see.
But it's a lot of work to do because we're built on so many front end libraries, including anti-design.
It's our core component library.
Each part is a superset in creating a new plugin architecture.
So it will be easier to build plugins for superset with all types.
And then of course, if you want to get involved in the community and build more interesting stuff beyond just a plugin.
We have these superset improvement proposals that are kind of the bigger moves we're making in the community, including
including the extensibility and plugin architecture that we're looking at new permissions models, user preference models, all kinds of stuff.
And these are all done via consensus.
That's the Apache way.
We send them to the mailing list.
Everybody has a big discussion.
They get on board.
We vote.
And it's a very open non corporate process.
So if you're curious about what's on the roadmap of superset.
One easy way.
They're going to go look at what's happening now in the community.
As to go look at our get hub issues and filter on the sit label.
Or you're going to see we have an actual sit board with like dozens and dozens of proposals that are in different parts of the development life cycle.
Yep.
And then the other part you can get involved in people do get involved which makes open source business intelligence a really cool place to work is that we have all these different working groups.
So we have a town hall where everybody kind of comes together once a month, but we have a lot of different working groups that are very ephemeral.
So people are working on some geospatial stuff right now.
Some people are working on security.
There's software quality, developer experience, release management.
All kinds of different little working groups.
And we just break into our areas of interest and then come back and put it all together.
That's kind of beauty of having like you know a mature open source project on a mature framework of the Apache delivery foundation.
We have a lot of infrastructure or you know logistics to welcome just about anyone to contribute.
So we see new contributors every month and then you know contributions of all shapes and size and flavors.
I'm going to talk a little bit about like the future of business intelligence being open source.
I think like the real idea there is like the the future of software is open.
Actually the present of software is open source.
Like more and more you know all the cycles and bits and lines of code that execute.
They and they out are more and more open source.
We see that like in their areas where there's a lot of innovation like the LLM world like not only open source is competitive but it's ahead in a lot of cases too.
And I think that explained by that is explained by the fact that open source is just a superior way to write software and business intelligence and elsewhere.
Some things about our community.
So you know the adoption of super since been crazy we like by all metrics viewer to look at you know downloads and stars and contributors.
There's you know we're very well ranked there.
Another thing the next one is about more about the last city so we've had more than a thousand contributors today.
A lot of steady like monthly active contributors as well.
And tons of pull requests so tons of momentum there.
There's no vendor lock in so that keeps us on us at presets.
You know as much as you might want to use a commercial solution to you don't want to run it on your own.
You can go to a solution on preset by the same time if you're unhappy with your vendor you can go straight back to open source.
And we want to make sure that it's easy for people to onboard but also the off board of our management solution.
Yeah there's a CLI tool that allows you to move all of your super set assets between open instances between preset across both very flexible.
And of course you can't have a talk nowadays without talking about AI for a minute.
Like Max said our repo is very active.
We've merged somewhere north of 2000 P.R.s last year and we need all the help we can get not just from contributors but from the bots.
So one of the reasons we think BI and open source in general are going to win on open source grounds is that not only is the code available but the slack is available.
And everything's just wide open bots and AI in general have all of the context that proprietary software does not provide them.
And so they're better able to answer questions and provide help on the repo.
So we've got bots now that are closing a lot of issues on GitHub.
They're able to answer questions very effectively people say thanks bot closed.
And they're also now helping us review P.R.s and merge and all of that.
They automatically tag issues and P.R.s answer questions on slack.
It's all really great.
Yeah we're getting slowly invaded by more and more agents in the repo.
They're mostly helpful and it's kind of scary.
The pace at which is happening but we have yet to receive our first pull request fully written by an AI.
But it's probably coming this year.
I got a hand in the crowd with the question.
Yeah is there or is there.
So let me repeat a question.
So is there some form of like agent or AI integration in the product as it is today?
Is that the right question?
I think there is room for that.
So what we realize trying to build something that we build.
Some text to SQL kind of agent helping with SQL at preset but the design patterns are not super clear yet.
So it's hard to build a solution that will work for the community at large.
People might want to use certain models or certain piece of infrastructure in different ways.
We're working on extensibility and a big extension framework inspired by the VS code ecosystem.
And then we'll make sure that we have all the plugs and sockets for integration and extension.
So that's our angle on that until the patterns around rag and integrating with AI's is our clear which is going to offer an extension framework for people to build those.
And of course they can open source their implementation and other people can decide which plug-ins or extensions they want to bring into it.
We got like three minutes left. How many slides do we have left?
We're good. We're good. We're good. We're good.
Obviously the main thing is you need to get your hands on it if you haven't already.
How do you do that? The main way we would steer people toward is just go to the website and go to the repo and install the darn thing.
So like Max said, Docker compose up. That's really as simple as it is and then you can get it running locally on your machine.
It might be a little more intricate if you wanted to deploy it because there's a lot of things to secure and configure.
They're set up RDS, Redis, RDS. It might be a little complicated to really get a production instance.
But if you'd want to go on your laptop, get a clone, Docker compose up and you can get running.
You can also try on preset. It's kind of the easy way. If you go to preset the IO, try for free.
If you like it, as I mean you have to go at a commercial solution, you can go back to open source.
So you know, there's just like training wheels there and it depends if you want to be on call for this thing or if you want people to write this software to kind of run it for you.
Now it's free for five users forever. It's not a temporary free thing.
Then of course, find us on Slack. Max and I are both on this Slack along with 18,000 other people.
So your good company comes say hello and ask all the questions you want.
Yes, someone will answer your question. It might be an AI agent. It might be someone from the community or if all falls off then it's probably going to end up being one of us.
We keep the bots to a channel.
Why is it called super set?
Well, it's called super set.
Boys at call super set. We have looked for a name for a while.
And the idea was just to, you know, I think the idea was to being kind of a super set everything you want to do around around analytics and BI.
So slowly we're kind of growing and becoming a super set of everything you might want to do in analytics.
And with all apologies to those joining us on the stream. One of the reasons we're here is we'd love to talk to you all about it.
Show you a demo. Get a little hands on. We're going to be at a table for the OSA open source analytics community tomorrow during those hours.
So come on down and we'll show you in person or you can talk to us after the talk.
We'll be hanging out right around here and are we out of time? More questions?
One minute, there's more time for more questions over here.
I saw at the beginning of the presentation. One of the companies out of the organization are using super set to Microsoft.
So what argument is Microsoft has?
I don't know, but there's a really good video online.
That's about an hour long of everything they've built on and on top of super set.
They actually forked the project and added all sorts of extensions and features.
So you can watch it one hour video.
It's the big team that uses it.
Sorry, the question was, you know, why is Microsoft using super set rather than not basically power.
Yeah, so they've actually, they've got a fork of it and they've been extending it with all kinds of extra functionality.
They've got a visual query builder. The project there's called Titan.
They've got a whole team around customizing that.
I think it's about flexibility customization and like the pace at which you can customize open source.
I don't think even being internal to Microsoft.
I don't think they can really customize power BI to their needs.
All right.
I think we're out of time.
We'll take more questions outside if anybody wants to ask them.
Thank you.
Thank you all.
Thank you online.