Great, so everyone, my name is Lisa and I'm the writer, currently working at JetBrains
where I create a commutation for IntelliG idea, so while my current role doesn't involve
any APIs at all, I've spent plenty of time writing the commutation for internal and external
APIs, and let me tell you, getting the feedback to improve all those dogs is one of the
biggest challenges in my career, like for all the time, and this struggle, even though
now it's in the past, let me to explore new ways to tackle this problem, so with our
advancing at liking speed, I started to wonder, could that be the solution?
I believe there is still plenty of hanging, low-hanging fruits there, and so today I'm
excited to share a proof of concept that uses AI-driven user simulation to test identify
gaps in API documentation, to test that, that way, and so that's not about replacing humans,
it's about giving us the tools to do more effectively what we already do best, so when
I've said that there are still a lot of low-hanging fruits, at least some of you
thought, come on, AI is already everywhere, but then it's needed or not, and years, in
this time of exploration, we have so many ideas about how to use AI in the commutation,
let's think of a few popular examples, you've seen or heard or talked about, it can
be improved search, intelligent code snippets, chartboards, write-off in your documentation,
summarization, highlighting, automated trend logs, oh good, all that sounds really great, so
many things that can make our life easier, and our readers' lives easier, but there's one
but to perform all these mentioned before, all these tasks, AI needs to have enough context,
enough information, so otherwise it's not going to give you needed answers or it's going
to boost it, so you want to get what you want, and who's providing that information,
a mere human being, so we'll return back to the goal dot old practice of gathering feedback
from stakeholders, from users, and planning well structured documentation that incorporates
that data, so that again, as I've said in the beginning, regardless of the company size
or the quantitation process maturity, select an actionable feedback on documentation,
it has consistently been a bottleneck, all the processes in my system experience, and this
issue is universal, latch organizations, startup, open source project, it's everywhere,
so that's why does that happen? There are several key reasons, let's say it's time constraints,
first of all, for stakeholders that just approve your pull requests with like, okay, let's
good, looks good to me, they assume that if they have any problems later, they can rely on more
experienced, do members, for example, so they don't have enough time to review your documentation,
in case of users, they just need a solution right away, if it's not in the good documentation,
they're going to find some work around, they're going to go to other sources, like stakeholder
flow, or anything else that can help them to solve their problem right now, so let's think like,
as a feedback is a kind of last resort, sadly, but it's an understandable choice, if you can get
your solution faster, okay, and expertise gaps, here is an interesting thing, it wants both
ways, so you can be two experience, you can be not experienced at all with experienced users,
so they just can feel some gaps in the documentation by their own knowledge,
so and they want to, even notice that something's lacking here, and so they think that all
these details are so basic and they're not needed to anyone, not only for them, with beginners,
it's all the way around, they assume that, okay, maybe it's not a bad documentation, it's not
in complete documentation, but they doubt themselves, and instead of recognizing gap in the
documentation, they just think, okay, they internalize that, this problem, the same, okay,
I should have known that, I should have learned it, and that's my hold, so and all these barriers
when you're in more explicit and open source projects, so because, okay, high quality documentation
is crucial for open source adoption and usability, and yep, contributors have varying
many hats, they are developer testers, writers all at the same time, so they're definitely
very experienced in that topic, and it's hard to change these perspectives, it's hard to change
these roles, it's hard to check your undercommentation, or it's hard to think of beginners,
because you are not a beginner for a long time already, community feedback, well, it's definitely
not that easy and especially on early stage projects, because many users has a take to give
your anything back to waste their time, if they are not a deeply invested in that project,
and well, to make them invested, you need to have good documentation, again, to adopt
your project, so they can adopt your project and become deeply invested, so yeah, the
documentation is crucial, without feedback, it's not possible to feel the gaps, it's not
possible to find them all by you, or at least it's very, very difficult, so the idea to address
this problem with AI driven user simulation, so this approach allows us to test the
documentation for gaps from multiple perspectives, so it can be a beginner, it can be more experienced
user, it can be some scenario, let's say I want to integrate this in this, and it's going to be
more specific questions, and of course, that reduces the cost, on the right of themselves,
take holders and sometimes that's the same person, so yeah, so I think the choice of this
prompting technique, the user simulation is quite obvious here, it's because as I said already
context is very wrong specific, you can use scenarios and bias is, well, I cannot say with
LLM since they try to, especially if they play the role, it's like completely reduced,
but eliminated, but it's reduced, let's say, workflow is like super easy, input your
open identification, you extract the needed information, you generate questions, cross-check
this questions against documentation, and you get the list of questions, so link is available
also on that talk page, if you, because I don't want to stay this slide for longer,
so domain magic under the hood, you prompt, so this prompt is really basic, what I want to start with,
of course it can be more specific for your situation, for your project, so just this simple one
actor is a developer who uses the APR documentation, one to integrate, our objectives is to
analyze with open APR specification, make the list of potential questions, and cross-check,
my interesting learning here is that for LLM's best standard open APR specification is still
version 2, so if you're not indicated, your specification is open API free drop something,
it's going to check you against version 2, and there's the list of questions, gaps,
okay, it's not like in the version 2, some limitations, so that was not here initially,
but I've got a couple of kind of philosophical questions, we'll return to it a bit later,
and I wanted to have cleaner output, more actionable questions, not something random,
and there are several ways to do that, to achieve that cleaner output, one of that is a few prompt
technique, but I found out it's a few short prompt in technique, but I decided not to implement
that way for this proof of concept, as a result, an improvement was not significant compared to
the effort required, so because I can just write one more sentence here, I get it already,
not 100% cleaner, but a lot cleaner than before, and I decided that it's like good result for
proof of concept, and the last thing is, okay, how do you want to get your output?
So yeah, I wanted it to have
this way, it can, what I actually did it like about this output, and there were a lot of repeated
questions for this different methods, for example, about missing mirror messages and error codes,
so maybe they can be grouped differently, but I like it to go method or the method,
so in the beginning, I'd just try to test it with unspecification, with all these intentional
gaps, this was a piece of documentation, I created a while ago for internal use, so we can
rule out situation that it was used in LLM learning, no, that just deleted some parts of
that specification, all these gaps were found, and many more to be honest, but for a
sake of a good example, I'll I've taken pet store API, which I'm sure all of you have seen at
least once, and some of you may be even memorized, they're already, because yeah, if you work with
APIs, it's a starting point, so the fun part, the first set of questions was full of philosophical
ones, it was possible to delete an entity that doesn't exist, yeah, so on the one hand, such
question highlights, the need to refine the output to get a cleaner, but on the other hand,
when I shared this result, with it like to develop as I know, that it led to useful discussion
about error handling and HK's clarity, so for example, do you need so specific error messages
with here and there, so it can be somewhat useful, but yeah, I think you don't want to have all
that questions like all the time, so and what about good questions, so how do you handle
duplicate pets, duplicate it, pets, so same name and attributes, can the ideas be reused
and the handling of bulkhead on your text existence is a huge question actually,
and actually in the past so API, we'll lack of so many error codes and there are messages
usually there are only one error code per endpoint, which is well, not enough for good API
documentation, let's say. So yeah, we can work with output, more, but maybe play in a bit
with philosophical questions also, sometimes, at least for one attempt, so obviously it can be
customized by the way, so first things, yeah, you can use different user personas, it's in
a different scenario, a LLM of your choice, and I would say it's not a big difference or
well, when I was using different LLMs and you don't definitely don't need the latest version
of huge LLMs, so for example for chatjipiti free dot 5 is good enough, you don't need
go further. So it's a way to know those capabilities teams can forward enhance the effectiveness
of user's relation to identify the communication gaps, and yeah, to sum up, so the tech world
is of all wind, but the quality of the communications is like still a cornerstone for successful
adoption, for successful usability, especially in the open source ecosystem, and AI is true potential
in helping human efforts, not to place that. By using ad-driven user simulation, we can
end-insify gaps, and then consistences in API documentation with remarkable efficiency, I would say,
and this approach not only reduces the manual burden, but also brings a fresh and biased perspective,
so yeah. So, and yeah, what I've shown you is just the beginning, a very raw
and it's already, it can demonstrate how we can use AI as a partner in documentation testing
process, but let it be just a starting point for your oral experiments and customizations.
So, this method, like any other can flourish only when it makes a community collaboration,
so yeah, my call is definitely to try it out, test your own API documentation, and share
your insights, and yeah, let's refine this idea together, after all, every step to a better
documentation is a step toward better software, so yeah, and that collaboration.
So yeah, thank you, and I'll be happy to hear your thoughts and questions, if you have any...
That's a question to you, so it's not a given question, it's a threisen question.
Did you know, I mean, the talk to clearly, that's sort of a rendition is very small,
but did you think about how to use these on our real cases?
So, because sometimes the orientation is very big, so you're going to keep more or less costs
and context problems, you need to search to a rack that sees them, do you ever go about how to
do like this? Yeah, repeating the question, so have I thought or tried to use this
on bigger documentation, whereas more context, so I haven't tried it, but since we extract
needed information, we can make this flow a bit more... we can have more levels and just
adding to the prompt only the needed context at the moment, so we need... we want to test
only this part of the documentation, and here's all the relevant, already extracted information,
yes, it should be possible. So yeah, it's a good thing, it's a good step forward.
I have a little bit of heavy tried using this on the next structure documentation,
but rather like now it's just gone. No, I haven't tried and I doubt that would work,
so even now with APIs there is structured documentation, as you said, it creates this philosophical
sometimes weird questions, and if it's like an element of possibilities of creating that
questions, well, I think it would be like not clean at all, not actionable at all, so it's
yeah, I believe a design for the structural documentation, yeah, here.
And yeah, I'll just do a little bit more about how you enable the company to go down the
prompt, so you have the role of the checkers and so on and play the documentation for.
So let me get back to that slide, oh yeah, so I believe a kind of
good. Because I'm a tech writer, I would have clear structure, and it's easier for me to,
if it's not all messed up together, it's just easier to try, okay, now I change only that part
and the check if I like results more. So all, since I've already selected,
user simulation is the general idea, of course they're going to be real, but, okay, we said,
who's doing what, but what's the objective? We're doing something to achieve something,
and yeah, how restrictions and limitations appear here, I've already discussed and output requirements
while it just makes it easier for us to work with output.
Any more questions or?
Okay, have you got any more roles already that you've tried out, or is it just a developer?
So yeah, I've tried just leveraging this, like say, level of the developer, so if it's
junior only learning how to do stuff, and if it's senior developer, and with junior it
there were a lot more basic questions about, so I want to say, like, come on, if you're
reading this documentation, why couldn't you google that before? That kind of basic questions,
but at the same time, it's again just maybe my bias, because I already know it for a long time,
and like yes, yes, this problem with expertise and both sides of expertise went low and high.
And so senior questions, senior developer questions, well,