Open Source Project and so I was like, okay, I'll just wear this one, but I think the
luggage is arriving at the hotel, so I'm only getting it, and I have new boxes on for the
day just FYI, I did I bought those yesterday, so.
All right, good. Welcome, first time. Thank you.
Thank you. Thank you for giving their presentation, and go and unbox them, slice them up.
Yes. Okay.
What's it about how we transform our documentation into interactive sound boxes?
Let's welcome everyone. Thank you.
Thank you for having me today, it's really nice to actually see, I know we've had quite a full room
today, but it's actually nice to see so many people interested in learning about
interactive sound boxes, because I think it's a bit of a niche for documentation.
I think it depends what sort of camp you've fallen on and how you interact with your documentation,
and I'll get on with that in a moment. I don't normally do this like about myself, but I think I
wanted to for this presentation because it shows you how my brain works and sort of where I come from.
But so what for Grafana Lamps? Work as part of the dev advocacy team there, and most of the time
I spent my head in a documentation for an open source project called Grafana Loki. So this is our
log aggregation database. I do a bit with Grafana or Intempo and our other databases as well.
For me as a dev advocate, I see this is where I have like most of my impact for our users.
I see like the idea of going to toys and practical non-reference material really gives
our community an uplift, especially our new ones, where I'm focused in my job,
is bringing those new open source users in and feel like they can actually start interacting with
us off where compared to some of our sort of both seasoned professionals. As well as that,
I run our interactive sandbox initiative. I'm saying I run this initiative. It's me and
interestingly Jack that you have here. You've seen all day and a couple of others. And then in my
life I'm a big gamer, Lego enthusiast, and IoT enthusiast. And I think this is where sandbox
is for me came about. I love to follow documentation, but I love to follow documentation
where I'll be in practical all at the same time. I like to do stage by stage by stage and feel
right by the end of it. I've actually got something, not just red, a document.
So the goal for today is I want to give you sort of a level set you on my version of what I believe
dev docs are as a dev role. And then what are sandboxes? And then how we scale to 9,600
I think we're up to 11,000 now in total. I think we have about 2,000. We do it 2,200
monthly users that constantly keep coming back for more. So it's got it's getting a bit out of hand
for free people now, but we're keeping it going. And then I hope you'll be able to make your own
for your own documentation or open source projects. This is our goal of open sourcing what we've
built together and hopefully allowing you to bring it into your own docs. So for me, I kind of see
dev docs sitting in two worlds. We have our key source of truth so you can imagine it as you're
like historic book for everything that that product or project was and where it is now. And
you know for me this means like this we look at Grafana for example. I should be able to look at
our documentation for 11.1 and know this is exactly the state of Grafana in 11.1 and be
able to reference that material. The other side of dev docs is of course a navelment. A navelment
is making sure that we can on-ramp our users and get them comfortable with our software and it's
our instruction manual, right? This is I've been, none of us has ever read a car manual but it's
there. It's the sits in your car in that little boot there. But if you actually read it, it
will maybe not tell you how to drive a car, but you'll tell you what your car did. But that's obviously
gives you the enabment there. Where we sort of sit, why do a lot of work is in the enabment and I think
this is exactly for me where interactive sandboxes sit. This is about unwamping our users in complex
topics that's better than just reference static tutorials. I've made the circle purposefully
stick in the interactive sandbox bit there and more on that in a moment and that's why we're here
so that they're meant to touch. They're hugging on the key source of truth and that's where all
my memes come in later on. So did we all remember our buckets and spades for this talk?
Terrible joke. I've really tried, I've tried this joke a few times now and it doesn't land so
I'm gonna keep trying. So what exactly is a sandbox? So it's a disposable VM that's hosted online.
It's focused on building practical tutorials. So basically from start to finish you should be
able to execute a particular task and we'll see an example of that in a moment and it allows you
to execute and install applications, code to arrive at some former result. The benefits of this is
it allows people to interactively engage with what's going on. Let's imagine you're teaching
people helm for the first time. Holy crap, could you imagine just going, here's all the documentation
for helm, go nuts. When these commands on your local machine without any, this person's brand new,
they don't have to know anything about Kubernetes, they don't know anything about how to deploy
or set all of this up on their system and we say, just try the helm docs. Well what this allows us
to do is give them an environment that has these prerequisites set up and allow them to
interactively click through and execute all of the code and see exactly what happens
within this very sheltered environment. And that plays into the second one, it's risk free.
If anything goes wrong, I haven't completely ditted my PC, I can literally just cross off the
browser and then open a new one and try it again. And of course, I'd be sort of so that it accelerates
the learning curve. So you don't feel like you're constantly referencing in and out of documentation,
you have something tangible at the end that you can take away.
Are you sure on time? So in terms of our interactive software that we use, we focused on
kill a coder. We absolutely, if anyone ever use catacoder, one, one, two, so catacoder was a really cool
tool. It basically allowed you to learn a series of interactive tutorials in the same way, but
it unfortunately closed down. And I believe there's only two or three people working in
kill a coder. From basically, all of our conversations with them. And they are working hard to
basically build what catacoder is. I let you kill our coder on how kill a coder works. I think
it's really cool. There's a lot of stuff behind the scenes with Kubernetes. They don't actually
use container pods. They use VMs that are being hosted in Kubernetes. But this is not what
this talk is all about. I would definitely let them come and give a talk about how they created
this interactive sandbox land. We're talking about how we convert docs into the sandboxes.
But what benefits us in Grafana is it's free for the users to use. They just have to sign up.
The only reason they have to sign up is because he had a bunch of crypto miners come along and
uses software to basically grind a ton of data, which was not great. It's all built and marked
down, which you'll see is great for us. Grafana, all of our docs are written and marked down and
live all in every single project source code. There's not a central repository for our
docs. They all live in each project, which Jack will tell you is a high ache in itself.
There are flexible environments. You get an event of the m by default, which has Docker and
podmen, but you also have Kubernetes clusters that you can run by default as well.
And then we can also do some engagement tracking, which I'll show you. This is kind of how we got our
numbers. So, this is how we use it. We have our docs page. You jump directly into a completely
light version of that. I'll show you a demo in a second of this actually working. You can see
we basically replicate the docs over and users can start running commands directly from that
docs tutorial inside the virtual machine itself. And they should be able to see each stage of that
process as it goes. Hopefully, if this Docker pool, I tried to speed it up this video because this
is definitely not this fast. And what you'll be able to see as well is you also have access to UI
interface as well. So, all the way was a new product for us, a new open source collector. It's
extremely complicated, I'll admit it. But it allows you to take complex topics and show you know
UI integrations, configurations, and allow people to work for you. So, I was really excited about this,
I must admit. I was so excited that I went to the docs team, Agrofana, and said,
we have this really cool sandbox. I love it. Look at this. We're going to use this for the
Loki getting started. And this, unfortunately, was all the replies I got, which is to give you a bit of
context is all of the sandboxes that are built have to live in their own GitHub repo, which is
different from the GitHub repo where we keep our actual documentation. And so, the first thing that
all of our docs write is said was, where's your single source of truth then? How it what happens
when we update something in the docs is our responsibility to also update the sandboxes as well,
or is that your responsibility? And what about here, if you get hit by a bus tomorrow, who's going to
do that? So, yeah, so these are, I actually found these memes online. I didn't have to be so many
single source of truth memes, but it's quite handle X. So, yeah, by the end of it, they basically said,
you can't handle the truth. So, I wasn't uninspired. This was me as soon as the rest year as the
next one. I was like, I'm going to make a source of truth that is so single that no one's going
to be able to like bring this idea down. And that's going to make Jack. So, I came to Jack with a
problem. And so, we put together like a proposal doc and I said, the main problem that we've got
is, and the main problem that we need to solve for the docs team is we want to basically say,
if any changes happen in the primary docs, that these changes are replicated inside the sandbox
documentation. We also have the problem that, you know, one marked down doc in kilocoder land
is a series of marked down docs. And I'll show you how the structure works, but, you know,
each page of a sandbox is a different marked down doc and so forth. And kilocoder also has special
syntax as well, whether you want to make, you know, a particularly him command executable or
copyable. So, there was a lot to do. And so, what we kind of come up with was a transformer written
in Go, that all is run through GitHub workflows. And essentially what this does is basically,
it waits and looks, well, sorry, runs, it runs currently on a con job, whatever, like 30 minutes,
oh, we go. Runs on a static, so we might be lagging by like 30 minutes for an hour, we'll get there.
But basically, it looks through each of our repositories for changes based upon the sandboxes that we have.
So, whether they be low key, graphana, tempo, or mimere. And then it runs them through the automatic
parser, opens, automated PR, and writes that into the kilocoder repository. And then we get all the
changes for those particular docs. This was kind of like the goal, was basically that we don't
have to worry anymore about seeing changes appear. I mean, you know, to be honest, there are some
manual verification checks we check. We have a staging environment which I'll show you and we check
that the transformer's not done anything crazy if someone started messing with the metadata or something,
and you get some crazy output. And then we push everything to production. I think I pushed
maybe once a day just to check, but we could automate that process as well. So, yeah,
really go. I wanted it on the open source package libraries that we did use. So, Goldmark is
primarily used for all the mark down passing. And it was designed specifically for Hugo, because that's
how we build our documentation, because that's primarily the mark down that's generated for our
docs is all written in Hugo. With that being said, as long as you include the Hugo metadata or
the metadata tag at the top, it will work with any mark down file. And it is an open source project,
so we can, you know, if anyone wants to contribute a raw mark down way of doing it, please contribute
to the project as well. But so the idea here is what it does is we'll see an example of this,
it checks the metadata. The metadata for us is the name, the description, what type of
the M you'd like to host, and then you can also use some pretty crazy advanced features with
reggae. So, if there's like a specific link or something that's not going to work for how
killer code works, you can do some on the fly replacement in the transformer as well. So, we process
the mark down itself and then we divide the content, the basically the content is divided into steps.
I should say within the processing stage that we change all of our links, because we actually host
obviously all of our images and stuff in our docs, but they have like a special shortcode reference.
So, what we actually do is convert those into just a direct raw mark down reference to the images.
When I say divide into steps, you'll see in a moment you'll see how killer code outlines it steps.
And this is all broken down based upon the metadata that we include within the actual mark down file.
Steps are transformed. The key goes to a shortcodes are transformed and then we write the steps to our
course directory. So, here's kind of like the flow, especially if you're creating new documentation.
You annotate the source code, you download the transformer or you can build it from go from
the GitHub repository, you run the transformer and then you push the result. So, we do have an
automated process for doing current or you know, we'll give it this. We do have a workflow for
pre-existing sandboxed mark down files. But if you're creating a new sandbox from scratch from
mark down, the process is kind of like handcrank. You would do this basically by pulling the
repository and checking what the mark down looks like. It's by design at the moment, because if you
screw up some like the metadata, then you get some wild looking mark down structures.
We just need to work more on the validation of the transformer. And then you push the results to the
kilocoder repo. So, I think I'll give you a demo of how the transformer works.
And I'm going to do it from a kilocoder from the sandbox. So hopefully, if I come out of that,
and hopefully, the internet stays on our favor here or this whole thing falls apart.
So, here's our learning environment for the transformer. If you were going about this yourself,
the first thing you would do is this okay is the size okay for everyone. By the way, do I do
me to make it bigger? Can I get slightly? Is that better? Yeah. So, essentially, you pull the local
binary. Basically, we have a bunch of binary that we just build, depending upon your OS and
Mac windows. We actually tested it works on windows, which was quite surprised, because we don't have
everyone running windows. And then we make the transformer executable.
For us, we have our kilocoder repository, so I'm going to pull that. And in our kilocoder repository,
this is where we host all of our sandboxes. And I can quickly show you what that looks like.
So, here we are. Here's all of our sandboxes. And they kind of live in a directory structure.
So, if we look at Loki, you can see all of the sandboxes that we have. So, you know,
intro to ingesting with O-Tail, K8's monitoring, and so forth. And what you'll see below that
is the actual structure. So, step one, step two, step three, step four, and step five. And this is
kind of the structure that we're hoping to pass on marked down into.
So, if I jump back, the first thing that we're going to do is check out a new branch.
And then this is some of the metadata syntax that you need to annotate your files with,
in order to basically tell the transformer how you wanted it transformed. And so, just to give
you an idea of what that looks like. Let me jump into the editor view. What I love about
kilocoder is also how the VS code implementation as well. So, you can basically use the structure
like you would like on your local system. And if I look at one of our examples, I do a complete
doc's example. Oh, you're going to give me the, I just wanted the raw marked down.
Okay, back up, go through the VS code. So, you can see the structure and metadata at the top.
You can see that we've said, for kilocoder we want the title to be this, the description.
And in the back end, we're going to use a Kubernetes two node image that kilocoder has.
The next thing that you'll notice is the, the metadata that we do to basically break up the tasks
into steps. So, interactive page intro.md start basically says, within this, I want you to
basically separate this into the intro marked down for that sandbox. So, if I scroll down the way,
also, you, oh, yes, even of course. Is that a bit better?
Look at that. And you can see we also have m tags. So, anything between the start and the end will
be included in that intro.md. We do have other metadata. So, if there's certain things you want to
ignore, within the marked down, you can do things like interactive ignore. And then anything within
those blocks will not be included in the final sandbox as well. And we also have the ability to
not include things in our docs, but include them in the sandbox as well for those specific use cases
where there might be commands that you need only within the sandbox versus out with as well.
So, this is kind of what the marked down page will end up looking like once you've annotated it.
And let me swap back to our trustee sandbox and jump back into the editor. If I scroll down and go
next, check in time, I can grab it on with our questions. What we can do from here is we can
CD into kill a coder by yourself. Sweet, make a new directory, make a new course, we'll touch
a structured JSON, but we'll see in a moment. And then essentially what we've done here is we've
then transformed this completed docs example into a new course. And if we do an LS on this new course,
you can see a transformed that single marked down page into the structure of step one, step two,
and step three. And this basically kill a coder attaches onto your GitHub repository
for this. So our as is graphana fast kill a coder and we'll generate the sandboxes on your
behalf, no need to do anything there. And right, since I've got five minutes left, I'm going to
speed run this last bit. Just a you guys can get up and running and we can get some questions in
I even had that sightseeing time is running out. So here's a transformer repo. This has everything
that you need. It's under tools. If you'd like to try it yourself, there's also a bunch of
examples and documentation on how to use the transformer there as well. So yeah, definitely give
it a go. See what you think, leave us any issues. There's definitely things that we know about
that we're trying to add slowly over time. But yeah, definitely leave us issues if you find them
for your own documentation. This is how this is that sandbox I showed you to running the transformer.
So if you like to try it yourself and learn how to use a transformer, you can follow to that
sandbox there. And lastly, I just wrote for my own self-worth here. Here's a one of the tutorials
that I worked on in Loki and this shows you kind of how we host it and you can see how our documentation
is replicated over to the kill a coders. And lastly, this is kind of how it's going. These are our
stats. So the project itself is in my expectation. It's been pretty wild actually. I thought we'd
get maybe sort of 500 people a month, like people that didn't feel like they could do everything
on their own laptops. But I haven't got updated this image, but we're up to 2,200 now for the end
of January. So it's pretty crazy. So we have returning people come back and the fact that they
have to sign in and then purposefully click on a sandbox proves that they actually want to take
part rather than just clicking a link and going, I didn't really get there. With our app being said,
there are things that we want to do to monitor further. We want to know how far they get if they
get stuck, etc. But all that's to come. We're going to think about how we can do like a monitoring
agent for that. And yeah, that was my speedrun to the end. Thank you for attending. Do you have any
questions? Yes. Are you running killer coin under self or you're using killer coin or like
it's an open source project that's has like a sound version or how does it work?
Yeah, so I've got just to repeat your question for the, so basically you're asking if we're hosting
killer code ourselves. So killer code are is hosted by a bunch of great developers that they wanted
to replace catacoda. They have a fantastic blog on how they hosted themselves and running it.
We work regular with them. The way we obviously want to, as a company, contribute to them more.
At the moment, the way they do everything is you as a creator, as a matter of if you're a company
or if you're just a one person, you all contribute in the same way. So I think we're one of the
largest contributors now of killer coder sound boxes. But you know, the only way we can pay them
is through like a subscription to say, hey, here's 999 a month. We really rate what you're doing,
but we would obviously like to support them better. But their usage model works by
they do a lot of training for Kubernetes certifications in CNCF and that's kind of their approach.
They charge for, you paid, yeah, on that.
But it's also open source. So if you want to, you could run it yourself.
In terms of the, our transformer itself is open source, killer coder itself. I don't know
specifically. They give you kind of like the background of how they've done it. But I don't know if
killer coder itself is specifically open source. But yeah.
Yes. Yes. So they host them. And so we basically just go directly to their site and it spins them up for us, yeah.
Yeah. Yeah.
Jack, could you like to answer that one?
Sorry. Sorry.
So they don't can make two.
Yes. So are you asking is the documentation source file format restricted to mark down
in a specific form at the moment? Or yeah. Yeah. So at the moment it is marked out only. Oh,
sorry, you might need to say this. It's marked out. But in theory, going back to the earlier
ASCII doc talk, if there's a specification for your language, there's a parser for it. We could
apply the same concept. The actual transformer tool came together in days. It wasn't, it wasn't a
huge effort. So we could add transformations for ASCII doc for sure. And we would love to extend it
to make it more useful for other people. Yeah.
Yeah. So at the moment, we talked with the idea of saying, sorry,
so I keep forgetting to repeat the question. But basically, the question was around the
metadata that we host in the source documentation. Is there something that the writers have to
apply in order to transform their docs? And the answer is, yeah, and we kind of toyed around doing
more automatically, say, could we break each step out into like a H2 and mark down? So we said, like
step ones are for every, you know, H2 that we have. But what we thought was, you want to be more
explicit than that. So we said, at the moment we say, everything within this, these given barriers,
step one is step one and then everything in these giving barriers are step two. But yeah, in general,
we could probably speak to, is it too much over heads at the moment? I think once you've done it
once or twice, then it's just a matter of copying from a previous Mark file, Mark down
file that you've done before. I think the only thing that I would say we give as a drawback is,
obviously, now you've got a bunch of HTML annotation inside your Mark down, it does convolut
a little bit if you've got loads and loads of steps. So there are some best practices to make sure you're
not over, you know, lurking your, your raw Mark down documentation. But it's I think to think about,
we could look at doing it on a pair, head of bases. Yeah, yeah, yeah, yeah, yeah, yeah, yeah, oh,
sorry, sorry, sorry, there you go, yeah. Thank you very, very much. Thank you so much.
I would specifically like to thank the organizers who are here, Jeff, Chris, and Chris Thuff,
and Robert Connecology, who was here. Thank you for being part of the two of the Doctor
Room, and spread the word, bring presentations next year, before the organizers next year,
So yes, thank you very much.
Oh, yes.
Yeah, this was the final really big news.
You