You're not going to talk about red fun, will you?
No, sorry, I'm just going to teach it, but no, I'm not talking about it.
Jake was going to really, really, really awesome demo about the red fun, the jack one
and how they read it today, I.
This talk is not about that.
There'll be something about talking, but it also is to read to me, so let's see.
Cool.
OK, so broken docs can be a nightmare, whether you're a tech writer, a developer or a product manager.
They wrote your trust, they wasted a time, and they make your product look kind of immature.
But we're here today still about Dr. Detective, which can turn your docs from a source of pain to a delight through robust automated testing.
I'm Jake, I'm from Red Panda Data, which is a streaming data platform.
And we're starting to integrate Dr. Detective into our docs, so to get testing and output and automated testing.
Yes.
And I'm Ariel, I'm a technical writer at Samarky, where a data management integration and governance company.
And I'm starting to look at Dr. Detective to also test our docs and help keep them up to date.
Cool.
All right, so this is just the truth, something always breaks.
So imagine your company to support the shiny products, and you've been tasked to set it up and configure it.
This is a product that's like, it's promised to solve all your company's problems.
So you might start by feeling cautious the optimistic.
You start researching just to see what this product does, and you might stumble across some docs.
So things are looking pretty good. The docs even have a guide for the exact problem you're trying to solve. So you're confident.
This guide has step-by-step instructions, copy paste commands, and you copy paste the first command into a terminal hit enter and it gives you an error.
At this point, you probably lost all trust in the rest of the doc, and you're not going to continue reading.
And this is a common occurrence, because keeping docs in sync with a product that's constantly changing is hard.
Can you repeat that?
I'll talk a little louder.
So yeah.
I hope all the whole thing?
No, just a lot of sense.
So it's common for docs to break, because keeping docs in sync with a product that constantly rolls out new changes is just difficult.
Even a minor change, like changing a button label, or what can seem like a minor change.
Like changing a button label from begin to start, complete the doc, and that's going to erode trust in the rest of your docs.
So we're going to dive into why this happens.
And it's a real challenge.
So one of the big challenges, and this is probably happened to everyone, but actually question,
who here has somehow never dealt with broken documentation?
Ever.
All right, you're in good company.
There are the challenges.
I mean, the biggest challenge at first is that there are lots of updates.
Your product gets features.
Your product gets changes.
Your product gets deprecations.
The libraries your product depends on are deprecated, and you find out later.
And it is a shared responsibility.
So I'm, for example, a technical writer.
I could watch for every single change in the product.
That's not very realistic.
I'm a technical writer.
I don't have the role of project manager who could be responsible for telling me all of the new features that are coming.
But the project manager might not know about changes to existing features.
And that's the engineers.
But the engineers are not writing the documentation.
I am.
So at some point you have to figure out who is dealing with this mess.
How do you get visibility?
Somebody needs to know what's in the documentation.
You need to know what's in the docs to decide if you want to change them.
And there are a lot of moving parts.
You cannot do it perfectly.
And something will eventually fail.
Your users will find out.
And that will cascade into a horrible scenario where somebody realizes something is wrong.
I mean, something's wrong.
What tends to happen is you open a support ticket.
Which leads to longer resolution times.
And generally, if consistently reliable docs build the trust, then inaccurate ones will undermine it.
And that's going to think things like your retention metrics, your satisfaction metrics.
Because building the trust is hard, but losing it is super easy.
So what we want is we want users to read something that just works.
And there is a strategy to enable that and it's called docs as test.
And docs as test is not the name of the of Dr. Techive that we're demoing.
It is a practice.
It's the practice of automatically verifying every step outlined in your documentation.
Which sounds ambitious.
But just imagine for a moment, imagine you have every command, every UI flow,
and every configuration step automatically checked for accuracy.
That is a dream.
But quite honestly, it's kind of the goal and it's what you want to work towards.
So if something breaks, like a command no longer works or UI element changes, the tests catch it automatically.
This is a proactive approach.
It means your docs remain accurate.
It means they remain aligned with the products current state.
And your users and yourself are probably much happier.
Because the chances are you have to use your documentation as well.
And there are existing testing tools.
Docs, sorry, Dr. Techive is an intermediate tool.
It's not for complete novices.
You do still need some knowledge to use it.
But for example, you have tools like Cypress and Playwright.
And you could integrate them into your docs.
But if you are like me and you are not a JavaScript developer and you have to use search engines
or local LLMs to figure out code, this is a lot.
This is a lot.
The two snippets that just came on the screen do the exact same thing.
They test a URL.
They interact with the UI and they take screenshots.
It would be nicer if this were a little simpler to read.
Oh, and that's where the docs tested comes in.
So I'm going to get rid of the slides and just go into a demo.
I think if you want, I can just start in the end.
Yeah, sure.
So Dr. Techive is a testing framework built by many silver.
It is an implementation of the docs to test methodology,
and it aims to make tests much more approachable to technical writers
by writing with JSON objects instead of programming languages.
And in this case, the JSON, how it's presented, is aimed to be a lot more approachable.
So yeah, this is the basic structure of a test.
Yeah, it's just a JSON object and it's made up of step objects.
And each step has an action associated with it.
So here we've got the go-to action, which is a web page or a URL.
We've got a find action that will interact with that web page.
And each action also has properties.
So this find action has a click property that tells Dr. Techive
if you find this element, click it.
And so what you can do is you can aggregate all of these steps up.
So one after the other, they build a robust test,
and it runs through a real user workflow.
So you can define tests in a few ways.
You can have a separate JSON file like this,
but then you have to maintain the test separately to the doc.
So then what you can also do is have in line comments.
So within the document, you just define the steps in the same file,
which means if you update the doc, you can update the test at the same time.
There's a more complex option as well, and you can teach Dr. Techive
to dynamically generate the test steps.
So you basically feed it a bunch of regular expressions.
And you say, if you come across this pattern, this is the kind of step you need to build.
But that does take a little bit more set up, but then it also does mean that you don't need to maintain
the test separately to your documentation.
So, but I think this is the good, a good moment to show you the fun stuff
and show you what it actually does.
We've got a few demos for you, and we're going to start with a very simple demo.
Who likes cats?
Well, and if you don't like cats, I'm sorry you are going to get a cat fact anyhow.
The first demo is a very simple test, if I can briefly show it here as well.
It is a single API call to catfact.ninja slash fact.
All we are doing is trying to see if it gets a valid response.
Dr. Tech Dev is reading the input, reading catfacts, and done.
Now, the terminal is a little bit small here, but it outputs it test results in a JSON file here,
well, formatted, which shows you how many of the steps and tests actually worked.
For the detail, and oh look, this is our actual response.
Did you know that cats been nearly a third of their waking hours cleaning themselves?
And did you know that this fact is 64 characters long?
Well, we have this here, but the big thing is that we got a 200 response.
So, this test passed, which is exactly what we wanted.
We will move on to test number two.
Sorry, demo number two, which I actually labeled as one.
And this one actually uses some inline checks.
We've got some makeshift markdown comets here, with the actual JSON for the test.
We're going to check it URL.
Check another URL, check URLs, go to URL, and then do something a little bit better.
We're going to save a screenshot.
And if the internet connection here works well, we will launch a library Chrome session,
which took the script. That worked faster than on my computer.
And that was automatically taken by Dr. Tech Dev,
without needing our manual input when the test was running.
You can imagine running this multiple times, or running it in a continuous integration,
and continuous update system.
Test number three or two is going to be a little bit different.
We are detecting tests automatically.
Here, we don't actually have any statements, but there are some pre-configured checks here,
or markdown files, and which amount of things will check the links.
Let's run this here.
There we go. Once again, we are loading this in Chrome.
This is a Chrome that's been pulled by a JavaScript library, specifically for this kind of work.
Same image. No problem.
And the test output worked.
It used the Chrome application. It was on a Mac.
It checked the links.
You can read through all this if you want.
It found an element matching a selector. That's fantastic.
And it took a screenshot, and better than that.
It checked whether the screenshot varied enough from the last time the screenshot was taken,
which means you can actually use this to keep your internal screenshots up to date.
See if your application has changed enough.
Well, warn you if there's a problem.
But my favorite test here.
So, in keeping with the cat theme,
how about we actually do a little bit more of a recording?
We can also do some basic screen recording here,
in which we're going to open a website.
And we're going to search for kittens.
Hey, we found them.
And the end result is an actual animation.
You can also use Dr. Tech.
For screen recording.
If you have steps that your users need to follow.
For example, you could record the key presses.
You could record all the different steps,
and show it as an animated image afterwards.
And this is the start.
Because we can do more with this.
Cool.
So, I just want to show a real world example now,
which is a repander we've built this into one of our quick start
for a product called Repandiconso.
So, this is a UI, so it needs some screenshots.
We don't really like to have screenshots in dots,
because they're quite expensive to maintain.
If something changes, you have to go in.
Set out the virtual environment again.
Take the screenshot and it's hard.
So, I just want you to notice here that the version here is 24.3.1,
which is an old version.
And the test is defined in the ASCII doc markup.
So, you can see here these are Dr. Tech.
Test steps.
What we're doing is we're telling it to do exactly what we tell the users to do.
We're saying to run this Docker compose file,
wait a bit.
And this is important because this is a real test.
So, sometimes they're flaky.
You need to wait for Docker to start before Red Panda console is available.
So, we tell it to wait.
And then we tell it to do.
All the things we tell the users to do check the links
so that it will save the screenshots.
It's going to check whether or not there's a difference between the current version
and the screenshot we have here.
And if there's a difference between the new one that is about to generate.
So, I'm just going to run this.
And while that's running, I'm hoping it's going to work.
And I'm just going to explain how this will work.
So, we've got a package JSON here that's set up.
So, we've set it up to run specific test depending on the script you're running.
So, I've just run the test console doc script,
and this is going to just point Dr. Tech Dev at my quick start for Red Panda console.
We've also got this built into our CI pipeline.
So, we've got test docs.yamel.
And we've set this up so that we can trigger it from repository dispatch.
So, if there's a new tag in the Red Panda console source code,
we can run our tests on this quick start to make sure that nothing broke
in the latest release.
And then we can also just run it on every pull request.
So, if there's a pull request made,
it makes changes to the Red Panda console docs.
We'll run these tests,
and then you'll get the test output in the pull request.
Cool.
So, it's running, which is good.
Oh, no, it's got errors.
So, there's some errors here because of the internet connection.
So, what I'm going to do is just tether and try this again.
Okay, how long is the time out normally?
I've got it set to like 30 seconds at the moment,
and we will try this one more time,
and if I'm going to just increase the time out,
because it is just networking for me.
Because what it's doing on the back end is it's just,
it's pulling all the documentaries down,
and it's going to try and open
the local Red Panda console instance,
but it's not going to be able to do that if,
like, Docker doesn't work.
Well, while we're rating,
should we ask for any questions?
Yeah, good.
Actually, why not?
Normally, this would be at the end,
but while we're waiting for Docker,
and the network connection to do what it should.
Any questions?
Yes.
Have you ever used this new test changes
in your relationship to make a snapshot
of what you're interested in changing
or how you're talking about it?
Uh, us two of me.
I'm, so, right now, um,
I'm.
Oh, yes.
Have we actually used this to,
to test screenshots,
and to do a screenshot change in our documentation?
No, just test, just test,
just test if you look at the location of this,
but when you ever change,
when you're the present change,
so, putting a breath,
to visualize exactly what this,
don't even do some change change in the principle of the.
Oh, um,
that's going to be helpful.
So, I think,
um, what you're saying is like,
how do you know if the documentation change
compared to what the test says it should do?
And I think the idea is just,
you write the test to mirror
exactly what your documenting.
Uh, so, for that,
you need an environment that's set up
that's kind of going to be similar to what the user has,
and then you have to run through the steps exactly the same way the user does.
So,
talk detective supports things like CLI commands,
as you can check the output of them.
You can save them,
and then use the output within your docs.
So, then if the test runs
and the output changes automatically,
or doc gets updated with the output of the new version.
And it's similar to what we do here with the screenshot.
So, when we take a screenshot,
we save it to a specific location,
um,
and the test does the same thing.
It saves it to the same location.
So, it overrides.
And so, here,
the image will be updated in the doc,
and that's automatic.
So, all you need to do is just push your changes to GitHub,
or have your CI pipeline,
push the changes as a poor request,
or just write to the branch.
But that has completed,
and you can see here that the version's changed,
because it picked up,
there's a new version of a panda,
and so the screenshot changed with the new version.
And then inside the doc,
we just reference that screenshot.
We just don't want to show it here, which is annoying.
But we reference the same screenshot.
And then, like Ariel showed,
you get the run down,
and you can see if any of your steps failed.
And it's going to show me that that's that failed,
because the screenshot was so different,
and compared to the previous version of it.
So, that's what you can use in your CI pipeline,
and you can say,
if it does fail,
then do this,
create a GitHub issue,
or create a GitHub request.
And that's another thing that's great about doc,
to take a bit of has a GitHub action that you can use.
So, that's what we're using here.
It's called the doc detective GitHub action V1,
and that runs everything in a doc environment for you.
Let's go back to the slides.
As with everything,
bear in mind that your test actions
are in fact real actions,
and you are working on your documentation,
so make sure you are developing your tests well.
This is a part of the writing process.
It is going to create artifacts,
it is going to create logs,
make sure you clean them up,
practice good documentation hygiene,
and make sure to test against your data.
You can test against theoretical data as much as you want,
but ultimately,
you are doing this for your own data,
and you don't know what will happen if you use synthetic data,
implemented live,
and then you find out that your synthetic data
was not quite what you needed to do.
But these are all part of the documentation process,
and this is a way that you can also integrate best practices
and make sure that everyone is aligned.
Okay, cool.
So yeah, these are the resources.
Docs' test is a strategy,
like we said,
there is a blog post about it,
the money silver row.
He is also the maintainer of Dr. Techive,
so you can go there to see the docs,
and there is an active discord channel.
So if you've got questions,
or we just want to try it out,
we're both in there,
and so is money.
So yeah, go ahead and try that.
And that is it.
Thank you, everyone.
Thank you.
Is it only web tools,
and can you evaluate source code?
Yeah.
So yeah, so Dr. Techive supports CLI input.
So you can test CLI commands,
which also means that you can run scripts.
So part of the test that I just ran
will run the scripts,
the fetch the latest version of Red Panda,
and then store those in environment variable,
and that is the find here.
So it says,
Run show is the action in the step,
and then we tell it to run this JavaScript file,
and then we say to whatever is in the output of that script,
store it in this environment variable,
and this one.
And then what we have is in our Docker compose file,
we use those environment variables,
so they get set,
they get used when Docker spins up Red Panda,
and yeah.
So you can run scripts.
Can you use S,
to keep seeing more of that?
Yeah, I think so.
I think because it supports any kind of scripting,
I've never done it,
but I would assume so.
Yeah, so sorry,
the question was,
does it support ASCII cinema,
and I believe so.
And there was a question over there?
Yeah.
So kind of piggybacking out with that.
You mentioned you can make screenshots
of your current test run.
Is it also possible to catch the output of these commands steps,
and then have them out in server file,
to include in the documentation of background?
I haven't done that yet,
but I think you do that.
It is possible.
Yes.
I don't think we don't currently store the output,
but we check the output.
So we'll say,
as long as the output contains this string,
then it's good if not then it's bad.
The one thing I will say,
because you also asked about CLI commands,
that's something I'm testing right now.
One of the issues I ran into is that
one of our internal APIs is complicated enough
that I'm having problems with the way
Dr. Tettiff currently works.
The maintainer is working on that for a future release,
but in the meantime,
the workaround is to actually call command line tools
to do some checking and use that output.
And I have access to all of that.
Yeah.
So I don't know if you can see that.
But that's an example of output.
So after running this command,
the output should contain pandas of fabulous.
And that's how it works.
Fantastic. Thank you, everyone.
Thank you.