All right, our next speaker is Evan Rusakis, who's going to present how Apache Superset
reinvented and re-engineered its world documentation.
Please give a warm welcome to Evan.
Thanks for joining.
Just curious if anybody has not heard of Apache Superset here, I'd love to see a hand.
All right, fantastic.
You've justified my existence.
All right, this is a long title, but I wanted to share some of the needs and projects
and learnings that have led to a better documentation set up for our project.
I'm Evan Rusakis, I work at a company called Preset.
It's like a managed service version of Apache Superset and a PMC member, which is a project management
committee person, and I work on the docs, and I'm tired of doing it, and I read them
sometimes, but mostly lean on AI, like everybody else these days.
So trying to make things easier for those training models on our behalf.
There's my contact info if you want to follow up afterward, but very quickly, this is not
a product talk, but I'll just tell you what Superset is real quick.
It's a very advanced BI tool for data teams that want to just democratize access to their
data and visualize it in sensible ways and explore and share the insights they find.
By GitHub Stars, the Apache Software Foundation's biggest project, it's got a lot of contributors.
There's been about 350 TRs this month, it's a very active project.
And it's got a SQL workbench, which allows you to really connect to pretty much any data
source under the sun, and write all kinds of queries, share them with your team, then you
can drag and drop those columns and build visualizations, those visualizations, build dashboards,
or do all kinds of drilling and filtering and cool stuff with.
Then, of course, preset, who was kind enough to fly me here, is a managed version of that
that adds a bunch of bells and whistles, and you can have multiple instances of Superset
for all sorts of different purposes. If you want to try Superset, this isn't easy way to do it
for free, not here to sell stuff, it's an open source conference. So let's get back to talking about the
docs. You can check out the repo here, you can check out the docs themselves, it's all live on the site,
obviously. And when we talk about rebuilding our documentation, this is one of the first questions
that always comes up. Can't you just have AI do this? And we sure tried just for sport, but in the end,
it did not go very well. You have AI that just kind of dumps out a whole bunch of garbage. Really,
it's, you get all these fancy mermaid diagrams, but all the unimportant details kind of come
front and center, and all the little nuance things that are very important to humans and administrators
of this kind of product just get ditched, you know, somewhere, very deep down in the docs or
they're hard to find. Sure, there's a lot of pages, a lot of words, but it doesn't really help people
that much. So you're the right one, you know where the product is headed, you know what's important
to people that use in the ministry of your product. So you should have a lot of say in how these things
are built. So the hot take here is that, yeah, you're the one that knows what your docs should be,
and what they should say, and machines are really the ones that are good at writing code. And the
point of this talk is that you can have AI write code to write your docs. So I wanted to share some
of the hackathon experiments that I've been working on to kind of prove the point to myself and the team.
So it all started with a little bit of road mapping. We had a project that
we had to reinvent the docs for, I'll get into it. But step one is assessing the mess. In our world,
we had all kinds of stuff that was scattered all over. We had wikis, we had, you know, emails and third-party
blogs, and just read me files in dozens of places on the repo. Everybody kind of created their own
little scattered bit of historical information somewhere, and institutional knowledge was really
what people would lean on, far too much. So the idea was to kind of clean all this stuff up,
get it all under one roof, and make it better than it's ever been. The key problems to solve are
that, you know, since if you can't find everything, the on-wrap for new contributors is incredibly
difficult, or for new users. Search is limited. AI even doesn't have that great of a singular
knowledge base to refer to when it's doing training runs. There's a lot of duplication of effort
because if there's multiple places, things are being documented, guess what you've got to do. And then,
the worst part as a contributor is whenever you write a new poll request and get some code merged,
now you have to go write the docs for that thing, and that's a total drag. Nobody wants to do it.
So this all just comes at giant mess, where the code base is a moving target, the docs can't keep up.
So what do we do? Do you let that code base determine what you should do really? You get everything
under one roof, federate all your content first, get everything cross referenced, and then you try
to get the docs to build themselves. Your code is probably full of little implementation details
and metadata and all this stuff that's really useful. So we'll capitalize on that. And then, of course,
you want to optimize for humans that actually read the docs, and then, of course, for the AI training
models that people are building. You know, these foundational models are trained on open source,
and that's whole other topic. It's great. So I wrote up this big proposal. We call it a
sip, a superset improvement proposal, and it was to build a new developer portal because we're building
a whole new extension architecture and superset, which is awesome. But for all these new features,
we want to make sure that developers are using them so we have to go make it easy to do.
First up, find the right platform. Another good use of AI is doing your homework for you and finding
all the platforms that exist and where they fall short for us that turned out that actually
docu-saurus picked all the boxes when you add all the fancy plugins. So we rolled with that.
And speaking of getting rolling, sweeping up. First thing you get to do, you've got old docs.
So go ahead and let AI switch through and find all the spelling mistakes, add all the cross links.
Just, it does all the heavy lifting of pulling your wiki over and all that very easily. So you
could just kind of get organized and have a good. Here's all my stuff version of the documentation.
And then you've got to look for the fun part. The opportunities to make your documentation build
itself. The code in many places is self-documented. So you want to look for those repeating patterns
and probably rewrite parts of your code itself so that it can be leveraged by your documentation.
AI is really great at turning metadata into pages, but not just saying here's some metadata
spit out a bunch of words that actually having it write the code for docu-saurus to render
to those pages or any other documentation tool you're using. So let it write scripts because we all
are probably using AI to write code every day. I know I hardly write code at all anymore be
an honest. So just use it to do that. So first test, we have this mapping visualization
in superset, one of a few different map visualizations where we have to have all of the
countries of the world represented and it parses a bunch of geojson stuff. And if you go and
mess with this gigantic Jupyter notebook, then you've got to go and update the actual plugin
itself to add the country that you might have added and then you've got to go update the docs.
So it's easy enough to actually have the Jupyter notebook update the code for the visualization
plugin and update the docs. And I was like, oh, that's cool. Now the contributor could just
do one little thing in a notebook and the product and the documentation just take care of themselves.
So let's expand on that. Future flags are something that we're a nuisance.
Previously we had this Markdown file on the repo and every time somebody added a future flag or
changed its status or its default you have to go and update this thing and nobody over does or wants to.
So it falls out of date all the time and led to funny bug reports and stuff.
So I went in and I added a bunch of comments to the config file. So you've got to this meaningful
stuff about what category the flag falls into, what is default status, what status it's in
if it's future flag life cycle as we get rid of things and then it builds these pages.
You've got a super long page all very organized of what is set, what way what it does, how long it's
going to be there and that's very handy and you never have to touch that documentation file again.
API docs. Everybody's got an API. Everybody's seen this thing, the swagger renderer.
We've had that in our product forever, never loved it. So Wambam, DocuSource, magic,
lots of plugins and all of a sudden you've got this very interactive stuff with code samples,
all the response objects from your API, the parameters, all the good stuff developers actually need
on a very interactive playground set sort of place. Now databases are something that we care a lot about.
Superset connects to a whole bunch of stuff and what it does as a product is essentially just
use some translation layers to send SQL to them from the database. You've got a SQL
library, you're writing the queries and then you get data back and then we visualize it. It's
actually pretty straightforward when you oversimplify it like that. But in the actual stack,
there's this top layer called the DB engine spec that sits on top of SQL alchemy dialects
and that's where we do the stuff that superset cares about like documenting time,
granularity is another little peculiarities of databases that make them special and make them work
with Apache superset. So this is our old documentation. This was hand-edited stuff on the Docs
site until just a few days ago, honestly. It was out of date, some of these connection details
were incorrect and nobody could ever answer the question of how many databases do we support
and nobody wants to go and clean up these Docs. So we also had this logo wall on the home page of
the site and just like how did we pick these databases to have logos on the logo wall? Why are these
ones important? So what I did is then went through and added with AI a bunch of metadata to
every one of the DB engine spec files that we use in superset. And that means we can all of a sudden
also take advantage of these DB engine spec details about time greens and other features and
all of the custom error messages that they respond. All of a sudden you get this lovely index page
that tells you exactly how many databases you support and you can search and you can sort by the
type they are or what features they support and all kinds of other stuff. So you get this lovely
table that if you're looking for your database you can figure out which one might be a good fit for you.
All of a sudden you get these great documents that have truthful and up to the date
information on how to connect to them and even what all their little errors and peculiarities are.
The newest one I just merged quite recently is about a re-entstory book. I don't know how to
do your front end developers are here. But in our particular product we've got a Python back end
react front end and we've got a million react components many of which are based on AntD but also
several other libraries. We've had this react story book like so many people have seen sitting there
collecting dust so to speak. Nobody ever actually does npm run story book to see what your
components do. They just kind of go into the code and figure out what they could do the best they can.
So if nobody's going to leverage it might as well build it into the docs. It turns out
there's a whole bunch of plugins you can use to make this fancy and build it into docusaurus.
You just have to have AI go and update all the real little story files.
Then you have fully interactive examples just like story book but even better you get this live
code editor. You can't do a story book as far as I've seen to just type some code and
fill it with your components. You get all the props and everything you need to know how to import it
and even links to edit the documentation that the story itself. Then of course the best thing you
can do for open source is tell the world you use it. So we have this in the wild page which used to
be a Markdown file. Nobody knew it existed. Therefore nobody updated it but why would you update it
if nobody can find it. So I changed it from a Markdown file to a animal file with the help of AI
and then a little docusaurus magic and all of a sudden we have this new in the wild page where
you can slap some logos on it. You get the little user faces from GitHub and it gives it a very
high profile page on the website and you even get a little crawling logo wall on the front
page as well. So that's a nice bonus. Then this is one I'm halfway through right now which I'm
dying to finish screenshot updates. Everybody has screenshots of their stuff in their docs and
there's such a pain to update because you're constantly changing the UI on things. So we're using
playwright to test stuff and superset right now and turns out playwright can take screenshots. So if you
actually find the right part of the DOM on your site to take a screenshot at the right time and
the right state you can just have the script run and take screenshots of all the things you need.
Copy the files into your docusaurus site and then your screenshots will always be correct and
we've added versioning to all of our sections of the docs. So whenever you cut a new version
it copies all those old files over and they'll be locked at the right place in time. And then as
you keep changing things your next version will always be up-to-date. So speaking of next, what are we
doing? Superset now supports theming so you can make the product look like whatever you want,
look like your brand great for embedded analytics and all sorts of purposes. So documenting those
creating a playground, how to build them and leaning on all the libraries we're built around so
that all of that documentation builds itself even when we upgrade all of these foundational
packages that can be done. We've got this extension effort which is kind of a big deal. We've
taken a lot of inspiration from VSCo where you can add plugins anywhere that do anything and we're
actually kind of riffing on their architectural plan of how that works. So you can add a bunch of
bells and whistles and a bunch of different places in Apache Superset coming so that's why we
built this new developer portal. And the extensions are starting to happen. So these dots are kind
of half human written, half AI written, but the real neat and potatoes of it for automation
sake is actually the extensions themselves. People are publishing them on NPM and right now we have
this little mark down table we're building because it's all very new. But obviously the extensions
when they get loaded they have a JSON file kind of like a package JSON and we can put as much
metadata in there as we want including your screenshots and descriptions and compatibility matrix
and whatever other licensing and security details we start to care about and then this page which
is right now hand edited will go away and be automatic. So as the ecosystem builds and suddenly
we go from 10 extensions to thousands of them it's all just going to show up there and be up
to date all the time. So yeah AI this is where open source has a huge advantage. It's almost
unfair really. Open source is the the best substrate for using or training AI everything about
your project. The people, the code, the design patterns, the history, the arguments that happen on
get of all of that stuff has been just sitting there on the internet and they're drinking it up.
So AI knows everything about you and now your job is to make the documentation and the public
facing stuff regarding your project is comprehensive as possible so that the next training run will
include more of it and be more useful to people. So you've got to help humans they need to know where to find
things but you know you've got to make sure that things are always current for them and
the goal is to not have to maintain as much as the code base grows. So we have a million
little helpers on our repo right now because we're open source all these people are basically
don't even their service to us for free which is fantastic and you have an AI chat on the home page
itself which is very good but all of these things are actually training on the doc site and
fine tuning constantly so the more we updated the more they know. So this stuff is already helping
and by the way there's a talk tomorrow with the GitHub thing if anybody's coming to that but I'll
be talking about these guys. They're actually starting to talk to each other which is a total trip.
So yeah and conclusion here I guess the point of this story is to not let AI just run away
and write a billion words about your product. That doesn't really do any service for AI that's
going to train on that later. Doesn't really do any service for your users that are trying to
read it and find the important parts and the nuance and the details. So yeah use AI but use it to
write code and use it to change your code so it can build the documentation and just you know
don't be too lazy about that. So ultimately if you put in the work and do these migrations
the docs will build themselves more and more and your life will become easier. So that essentially
is the crux of my time. I love some time so if anybody has ideas, questions, whatever I would love to
hear about it. I've also got stickers for anybody to do on some.
Cool. Five minutes if anybody's got burning questions or ideas or whatever.
This kind of doc's rebuilding project. See the moment you had something you could close to the world.
How long could approximately take it? You can publish something in one day. Some parts were harder
than others. Doing the in the wild page where you could just seal the faces and logos all of a sudden
took a couple hours. But doing the storybook thing where we have hundreds and hundreds of stories
and they all need to be refactored in some way. That was a lot of you know monkey in the middle
testing and nope that didn't fix it, nope that didn't fix it stuff. So it depends but you can just
chip away at it. Those were a handful of projects or projects rather that I wanted to start with but
there's plenty left and I'll just try to make as much of it as generated as possible in the near future.
So far the feedback from the developer community has been. The question was about the feedback
from the developer community was their pushback or general acceptance or excitement and so far
the it's been excitement. We have answered questions we didn't have answers to before about
what databases we support how many of them all that stuff. The optics in terms of partnership
have gotten better because now we're surfacing logos for all of these different companies and
different databases. They get a link. They get better SEO. We get better SEO because people are
able to search for these things. Does superset connect to database x? Well yes it does. The answer is
there. So being that much more comprehensive makes us much more findable as a project.
It makes the site more comprehensive and pretty and you know it's almost like the old
web rings. We just link to everything now. They link back to us. It's fruit to a cycle
and it's all growing very quickly. But the developers are stoked because nobody has to maintain
database docs anymore. Nobody has to maintain more and more parts of the docs.
It's taking care of itself. Anyone else?
I don't. You showed it somewhere where like to find out which database you can connect to
the app. It's made up of it. Mm-hmm. And then make it discoverable in some form, right?
It looked like it was stuck strictly from what you stated, but they're like they do
languages. How actually did it? Is it like a study? An analysis over like a day?
Okay. Yeah. Yeah. How has it actually worked?
Yeah. And it's a link about for instance. Yeah. So the question was kind of about how
it works. Like is it saw some type script? There's also some Python. So what languages
it in and out? How does this transformation and build process work? And when you start up
docusaurus, it gives you the ability to just run a bunch of scripts along with it. And so
half our code base is in Python, half of it's in type script, running back in. So it can really
merge any of that stuff. It's pulling in yamophiles and JSON files and Python files and all kinds of
stuff. And it's just a collection of little scripts. So for the database, the DB engine specs,
there's metadata and a Python file. For in the wild page, it's yamoph or maps, it's type script.
And just each one of these little scripts has this job to just chew through all the metadata
files, build an index, build the individual pages, put all the links and logos and everything in the
right place. So it just runs all of them in sequence. How do you set up the review process?
Like every AIDOM, you have like a review rounds? So yeah, the question was about the review process
and how we manage that. And really the pull request has a preview build. We have one bit on the
pull request. You actually get a preview build of the site. So you can just click it and click around on it.
And I'll be adding some visual regression testing as well. But there's no AID involvement in these
and when it builds actually because the contributor just edits some metadata and then what happens is
deterministic. So the AID, I guess the crux of the argument is that you shouldn't let AID be building your
docs. It doesn't in our case. It's really just that we're using AID to build more and more tools
so that deterministically the docs are scripted to build themselves from the code base.
Yeah, a bunch of import and layout builder scripts and all of that.
All right, I think that's my time. Thank you all very much.
Thank you very much. Thank you.