Our next talk, our next speaker, rather, is Ivan Pumman of, let's see, now, now, now, now,
but Pumman Maria, presenting, made a documentation from DSL to dynamic docs with ASCII
doc and Antora. Please welcome Ivan. Thank you. Thank you. Thank you. Thank you.
Yeah, we're entering into 4th hour of this room. Things are getting difficult. Yeah, thank you for coming.
So, yeah, I'm Ivan and currently I'm working as a team leader at technical startup called
synthesized in London, we're producing test data automation you can trust, but also I'm teaching computer science and Java
at number of universities. So, actually I also identify myself as a software engineer. I probably
produce more documentation, technical documentation and the slides than actual code in my
board. So, I would like to share the experience with the product that I'm currently working
at this synthesized company in particular, how we document it. So, what this product does? Like,
roughly speaking, it transforms and generates data in relational databases. So, it's core.
It has a number of things that we call transformers or generators. There are, like, several
kinds of them. So, to give you some example, it can be just random number generator with some, you
know, distribution statistical distribution. Or it can be a person generator which will
mock up somebody's gender age name and some, and so on so forth. Or we can give it several categories give them,
give it, you know, probabilities and it will generate, you know, categories and stuff. So, and more,
there are 35 of them and the whole product on top. So, what do we document? What is the surface,
like, user-facing surface of the product? First of all, things can be configured via
YAMO file or they can be configured via user interface, the editor. And of course, usually
this product evolves over time. It's several years old now. So, new transformers are added.
Like, features are changed, bugs are fixed. So, we need to keep a number of things in sync,
actually, in this product. So, the things that we must, that must stay in sync. First,
is Jason schema for the YAMO configuration. Of course, we want to validate this YAMO before even
we start, like, feeding it to our product. And we want context sensitive hints in editors.
So, we need this Jason schema. Always up to date. Also, the projectional editor UI,
which actually follows the same, like, yeah. I believe, like, people who are in this room,
you are big fans of everything is caught, but not every users. A big fans of coding.
So, people want this editors. And we must keep them in sync as well. So, how do we do this?
And finally, and there's the main focus of the talk is documentation, which actually
must follow the same structure. It must describe it. It must be always up to date. And in particular,
all the code snippets, this code snippets, they can be, like, copy, then paste it into
customers' workflows. And they must work without saying, oh, this argument is no longer valid
or something like this. This transformation has been renamed. So, in the very beginning of
the development of this product, a decision has been made. And I think we are still getting benefits
from this decision, like, many years after, is that we are using DSL approach as a single source of
truth. So, I'll throw part of the product is written in Kotlin, part of it is written in
TypeScript. The software developer starts from editing DSL file, the demo file, from which large
chunks of artifacts are being generated. So, I counted six in our particular case, like
DTOs, this JSON schema, the projectional editor, which is actually auto-generated. We don't
make our front-end engineers like to follow all the controls and switches. And the documentation is one of
this, you know, six outputs. So, what did we choose as a DSL as the domain specific language for this?
Surprisingly, in our case, it's open API spec. Well, as you can see, it defines all the
type names, it defines the structure, like in properties, what parameters do we have, it defines
descriptions, nullability, all the stuff, like lots of stuff, actually. So, if you ask why open
API spec, why such is strange, probably choice, well, because it's just because it's just
feed for the purpose. Because built-in documentation fields, it supports extensions, it supports
non-standard extension via X fields. So, if there is something which is not in a standard
open API, you just put a property anywhere, which starts with X dash. And you can put just any
additional semantic value you want. And what's important, we have a very high quality
widely used open API parser for the GVN for Java. And, well, I believe in our days, the success
of the product is mostly from using open source, which fits best your purpose. And writing
like some glue code. And if you are choosing your library carefully, the amount of glue code
is going to be minimum. So, in our case, we already had co-generators for Kotlin and for
TypeScript to produce our details. And the remaining step was to generate the documentation.
So, we decided to generate ASKIDOktor from the same source, from the open API spec. And then
rely on ASKIDOktor toolchain for everything that follows. So, actually, this yellow
tiny yellow part needed to be implemented. And by the way, 2020-26, you just task your agent
to write glue code. And it can be done really quickly. So, the full pipeline is like this.
So, as I said, there is a number of branches, a number of artifacts that we are producing. But
we have only a small amount of glue code, like from Swagger parser, like using
Kotlin, Kotlin is just also very reliable library, which is a builder of Kotlin code. So, we
just use this glue code and produce Kotlin details, JSON schemas that we need. And the document
is the documentation. One thing, like, when we think of auto-generated documentation, we
think of something like Python docs, strings or Java doc. And you know that they produce
HTML right away from the code. But interestingly, if you want to do something yourself,
what I recommend is to use some semantic markup, because it's much easier to produce than
HTML. And also, it will merge finally. It will blend finally with the rest of your documentation.
Because if the rest of your documentation is written in the same markup, you will get automated
documentation. And, like, written by humans, the documentation written by humans, in the same
style, with the same visual style. So, which is important. So, speaking about DSL, DSL is a single source
of truth. First of all, it's a design decision, a project design decision. It's not a decision
that is taken by technical writers. So, some other people, like, it must be taking it very
beginning. And it helps keep multiple parts of the product in sync, including, but not limited
to the documentation. It's very powerful approach. The choice of DSL, however, is context dependent.
So, in our case, it's open IP, open API spec. In your case, it can be anything which fits the purpose.
You have lots of options, like a fully custom DSL. If you feel like writing your own custom
parser of something. An existing spec language, such as XSD, open API, like whatever spec language you
feel fit. It can be YAML XML-based format, or it can be an internal DSL in a host language, such as
Groovy or Rubio Kotlin, which produces this, you know, syntax abilities to expose an internal DSL.
The best results usually come from choosing a technology with strong out-of-the-box tooling. So, you
only need a small amount of custom blueprint. So, let's get to ask a doctor part of the presentation.
And there's a question, like, why not ask a doctor? Why not something else? Well, here I'm a bit opinionated.
For DSL, I said, like choose whatever you like. For ask a doctor, I'm sorry, Daniel.
I'm opinionated. I'm from ask a doctor camp. I believe that it just outperforms all the other
markup languages. It has richer semantics. It has really powerful tables out of the box.
It has attributes and conditional content, which makes documentation dynamic. And it has countless
diagrams, code integrations. And also, it's truly cross-platform. Because if you're like
your project is in Ruby, in JavaScript or in JV, you're using your build tools and you
completely get ask a doctor tooling there. That's quite a rare, like, quite a rare example
of truly cross-platform tooling. And my favorite feature include about which I'm going to
speak separately. Complex tables. This presentation is in us,
the doctor, by the way. So there's a GitHub repo for it. You can have a look. So good luck.
If you want to do these things in Markdown and escape like chopping onions.
That's a basic thing, like, same with the many languages. Let's have a look at task
a doctor example. This call out, by the way, is supported by a
doctor. It's first class support, like no extensions, quite unique thing. And yeah, you
can do it in various languages. So, as you probably guessed, here, you see a
client like 2500 years old Delgaritum for fighting greatest common
divisor, presented in various languages.
Lua, prologue, Ruby, Java. And you may ask, like, what you make, like, feel,
believe that these are actually valid code snippets that they are actually producing
this idea. Well, I can assure you that all these snippets are tested before the whole
presentation being built using property like strong property based tests and the
besides integer overflows they are correct. How do I achieve this? I'm going to
tell you in a minute. But before that, I just can't help showcasing the
integrations that a doctor have, which actually allows you to keep
colorful, powerful, powerful pictures, illustrations within your documentation,
in plain text. Alfaindomic is graph is quite low level, but you can do very flexible.
You can do anything in it. If you want some formal stuff such as UML, you have
planned UML for this. If you are teaching, like me, languages, you need this
syntax diagrams and I'm really proud to show this slide because this tool in particular
was implemented by my third year Java students. And due to, like, amazing
community, an aski doctor, they included it as the first class support in an aski
doctor. So if you want to describe some syntax, you have JSON
tricks for this, and it's supported by aski doctor.
It's in universal itself, right? So you stack charts like bar charts, you need
them and your documentation, not technical documents. You don't need to copy and paste
pictures. It can be a hidden. And of course, it's not like that
frequent that you need a formula and your technical documentation. But if you do need
them, of course, there is only one format for the formula, like if you're taking
formula seriously, that's latex and aski doctor is capable for this out of the
box. So it's extremely powerful, but my favorite feature is still
included. Yeah, other tool chains they're missing it. So what it does, well,
a simple thing, right? So somewhere in my code base, I have this Java file and I
just include it and it's just being included here on this slide. But Java is
verbose and actually you don't see anything useful on this slide because of the
header like all this stuff. But as you can see, if I put special
comment lines here, I'm tagging a snippet and then later I can refer to this snippet.
So actually what I see here is not like a meaningless JavaScript snippet,
copy and paste it from somewhere. It's like a tiny window where from which you
see just a snippet of the whole big Java file. And then I'm put this on the
slide for my students and I say that this is the solution of a classical
problem of word counting. You have a text file and you count words and you
have a map from string to long blah blah blah. And the question is, how do we
make sure that this claim is accurate? We need a test for this. Okay, for
Java string API, I'm probably straight forward, but if it's some other tool
that we are documenting. We need a test. So what do we do? Remember that this
is actually not a snippet. This is actually a window for the big file. So this
file can be unit tested. So I'm writing a unit test for this. Oh, sorry,
Java is to verbose. Let me focus on the interesting part. And yeah,
this is a unit test which actually calls the method in question and
verifies that it's compiled that it's run. And here we are. If you keep the
commutation and tested example code in the same repository, if you use
include with tags to pull in only relevant chunks or only relevant fragments,
you build docs and run the example code tests in the same CI pipeline. And
this includes using same build tool. This is why it's important like for that
we have ASCII doctor, right? We can run ASCII doctor in NNPM. We can run ASCII doctor in
Ruby, build tool. We can run ASCII doctor in grade all the way. And it's either
green or red. If some of the tests are red, then the test breaks the
CI fails. You have no documentation. Final part is
Antora. Why Antora? Why DSL? In DSL, choose whatever you like.
This is my point. For semantic markup, ASCII doctor is the best. That's my opinion.
Antora, well, frankly speaking, you don't have any other choice if you are using ASCII
doctor. That's the truth. What's good about Antora? It has version
doc as a first class concept. So if you are doing release cycle for your library or
your product, you can do tags in your git repository or branches and Antora
will refure and it will publish them. Next, it has an opinionated layout for
models, assets, examples and stuff. You put some example here. You know where to put
image, where you put your SQL file, where to put your YAML, file example, whatever.
Antora has standard folders for this. It has specific format for cross-references.
So it has this notion of modules, examples, assets and stuff. Which helps a bit when you
want to move stuff around your documentation. So when you're not relying on particular
place on the file system, but your cross-reference is put in terms of modules, then you
can move your ASCII doc files from a module to a module and your cross-reference will
be working if you are careful enough. So that's really powerful feature for maintaining
of large documentation. Last but not the list, it's surprisingly fast. Like those of you
who know how long I can build cycles with great Laurenti and they can take ages.
Antora builds huge size in a matter of seconds, and this is really impressive like in this
days. However, and yeah, during previous presentations today, we had a lot of cases for
you know, not documents as code solutions. Like WIKI or things like what was it called,
last year, right? So this presentation brings like another kind of end of the spectrum.
Like everything is code. Everything is tested. Everything is on CI. Well, actually from my current
experience, I think, and this is still not solved problem, we must be, we must support something
in between. Because when you've got like large and Torah presentation, large and Torah
documentation, everything on CI, it becomes really difficult for people like solution architects
or marketing people to edit it, to move contents around, to add some descriptive chunks.
Also, it becomes to burden some to sync it with the list cycles. Like I want to update it now,
or no, we need to wait for the next list. So probably if your documentation is mostly descriptive,
is decoupled from a list cycle, probably you should think about something else like WIKI or other tools
that we discussed earlier. But if you want to keep it tightly in sync with the code and with the
features of your product, then this approach is definitely the best. So to conclusions,
keep the documentation close to the source of truth. Ideally, if you're using the same
repository on GitHub, with everything like with documentation and with code and tests,
generate documents as plain ad hoc and let us give doctor to do the rest. It's really easy
to co-generate ad hoc. Use the include magic in order to include testable examples and make
your all the examples of your code, be it SQL, Yamu, Java, testable and tested. And yeah,
the part of writing the documents and the part of engineering the whole solution is actually
the same process. So when they're blunt, you get the best results. So thank you for listening.
You can see the source code of this presentation here with all the examples and all the code
examples in this presentation are really tested. Thank you.
WIKI about WIKI, like, yeah, first of all, one hour ago there was a presentation
like which demonstrated that actually ask doctor like provides you with the ability to write
your own templates for output templates. So yes, it can produce HTML and can produce PDF,
like there are templates for open document formats. And I didn't heard about, like, you know,
WIKI pages, but I think it shouldn't be that difficult to write your own template because it's
like, namely from one semantic markup to another semantic markup. Yes, please.
How are you familiar with the concept of the programming?
Could something definitely, but yeah, I'm not familiar. I heard the, you know, the question was,
am I familiar with literate programming? I feel the name, but I don't know what it is.
Orgmode syntax? No, no, no, no, no. Let's discuss.
question of the day. Questions? Yes, please.
You have terraform and you want it to be beautifully documented.
100% it does work. This is the idea.
If you have something deep, but you need to find a
uniform, you know, documents.
But yeah, terraform has additional attributes and
yeah, I see like, no, no problems in implementing this.
Yeah, yeah.
Infrastructure risk.
Not like what?
If the question is, if the DSL is not open
API spec, but some infrastructure is code DSL such as
Ansible, terraform and stuff.
Yes, from what I know about this DSLs,
I would be capable of writing something which
will produce semantic markup for documenting this.
That's definitely possible.
Thank you very much for your help.
Thank you.