Okay, hi everyone, so with the next talk, we have Steven, we're talking about talking
in talking to the community, I think we should be that hard.
Thank you, this plan, this talk was originally planned to have a co-speaker, but he couldn't
make it because he felt it.
So with myself, I'm Steven, I work on a project I call dogspec, and I'm going to talk
about that project.
Dogspec originates from the Dutch government and we have an issue there because we have quite
a bit of documents in government that are not accessible, that are not insropable, they're
hard to get comfort, and within Dutch law and European law, it's basically a requirement
that any information you make public because a government, it should also be public for people
who, for example, countries or have problems with eye-size or have a different situation
why they can't access information in the same format, and we took on the task at the
level here, that's part of the Dutch Ministry of Interior, to turn them into accessible
and reusable HTML.
So you might think the obvious solution is to just use a Spendock, that actually worked
for a while, until it didn't, Sandock is a very good conversion to with a lot of features,
it does a lot of formats, but it might be if you need specific features or you need
a specific way of output, then it's harder to use.
So we built around it, we created a piece of software, we call it first the P4 and the P4
P4 is a Pendock, P processor, and we also ended up doing post-processing, which was
actually more complex than doing the conversion ourselves, as you can see, we have piece of
inputs, P processor before it goes into Pendock, and when it went out of Pendock, we
used the same kind of software to post-process, it's to clean it up, Pendock would do the
core conversion, P processor would normalize stuff for Pendock to do it a bit easier, and
post-processing would fix up, put it in the format that we actually wanted, but you
lose control in this stack, because you basically use Pendock as a black box that you
pre-processing post-process on, it's complex, and the annoying part is every new requirement
in our application, it requires a change in Pendock filters, or in the input we put in,
or the output we got out and we need to process, so you keep on basically having brittle
changes, if I know the core issues, of course, Pendock is primarily a common line interface,
it does do a web server, but it's not feature-completes, it's for sure not design for
collaborative editing, and that's what I'm going to show you next, but first, but it's
we designed the AST-based VJBase that we can convert to, since in-between the input and
outputs, and yeah, we did our conversion ourselves, you can find a recent version of the
AST on this link, I'm now actually going to show something and what it does, so you have
better picture with it, so our project was basically a tool to make documents more accessible,
you could upload a document in our editor, and we would have validation on levels of like
headings or machine objects, you can fix them, this would disappear, and then we can
convert it back to order formats, and a user can, for example, just upload it to a content
management system or another system, what is very cool about this, I will zoom back a
little bit, is that validation message you see here is also basically an output for the
converter we built, we created an input reader for the word document file, and we also
created an output for the editor, so you can convert from the editor and the editor back
to order formats, this was mostly surrounded around accessibility issues, around documents and
documents, as a tool that would help governments make documents accessible, and as a tradition
in this room, I will also give a shout out to Blocknotes in the suite, we also build this tool
for Blocknotes basically where you can use it to import documents into Blocknotes in the suite,
it happens in the same manner, this so the dockage reader is always the same, it doesn't
matter if you export to HTML or another content management system, this was merged to production
last week, and it helps to migrate from Microsoft works in general of course, it works kind
of the same way as you saw in the previous demo, it just applies a document, it's being
converted, and then you can basically manipulate your content that was also in the original
document, so in order to convert we created an AST based on JSON, where you can divine
elements, it's very similar to other ASTs, also such as the Blocknotes one we heard earlier
about, it's also types, I use type check nowadays to type it and you can basically create
your own image or to other programming languages, means that you only have to maintain one
spec, and you can use it in order languages, I wrote the code base of the current converter
in a lecture, so that's also the current language I convert my touch to, and basically with
this AST you can easily map it to Blocknotes or tip-tap elements, if you're lost with this
spec was really an AST to describe any and every element, so that means if this happens
to your input document that there's a strange element inside of it, or layouts, we want
to be able to describe it, so we can at least try to convert, as the least amount of loss
as possible, we currently focus on.ex, there is possible to also convert PDFs, but that's
bit more that the search is on talk, as to do with machine learning, and we convert to
editors like Blocknotes and tip-tap and to formats like HTML and EPUP, and it is planned, you
can see in this chart I'm not sure if it's very visible, but you can see basically what we
want to implement this year and what we implement it, so currently for input, we support
talkics and tip-tap, but we also want to do HTML, markdown of course, and open documents,
we also want to import from Blocknotes, search again, basically export your Blocknotes documents
also to formats through our system, we want to make an export in.ex, markdown and everything,
this is a plan for the coming year, and we want to go to PDF for up with using types,
so types as in tool to basically render out your documents as PDFs, and the decision also
very, really want to go is rewriting the Alexa code to Rust, which also means you would be able
to run it in browsers with WebAssembly, we'd be able to run it as a command-line interface that's
also currently possible as an API that's also currently possible. A skew worker, as a library, of course,
and with FFI-dynamics to any language, this would practically mean that you can use this in any
projects, you can use this in any program language, and most importantly, you don't need a
server if you are an editor that needs Falcon version, you can just do it in a browser,
which is quite important for enter-integrated systems, because they can't expose
documents to the server, because the food basically breach and the encryption.
And that makes it quite interesting for projects like CREPAT and the next graph
where the server can see the document contents, and by ensuring the conversion happens
client-sized, we basically preserve the privacy of the user. It would mean that you can
convert without back-end infrastructure, which would do its latency, and would be quite real time,
and as a set-three times, and to end. I have some URLs here where you can see the projects,
you can see last week docks, you can see the project docks by itself, you can see anodok that's the
all the code base that we use at logias, part of the ministry, and I add that links, of course,
to Pandok. All right, this is my talk, you can see my contact details here in case that you
want to reach out, and I would like to take your questions.
I have a first question, so when we're talking about work conversion,
the question is how far can you go, because I worried it's a very large format with a lot of
features, so what are the limits in terms of what is supported, what is not supported,
most probably more important in the export, and link to that blog post has now recently
comments and suggestions, are you planning to support comments and suggestions in a word format?
That's the problem. Let's take on your first question first. Can you repeat your question
of me? Yeah, I also know today. Yeah, the first question was how far can you go on working
important? Yeah, how far can we go on working important? You can basically go as far as the
budget reaches, but it's not your technical question. It's quite hard format, because there are
quite a collection of versions of the formats that you can use to describe the exact same element,
and that's quite intense. There will always be some cases that you can't really cover,
and you basically have to work with a lot of test documents to import.
I think almost on a level of Pandok, almost. I think you do a little more with food notes
and end notes, which I don't cover yet, but it's yeah, plan to be covered. Your second question
was about comments in blockness. It would be very feasible. I also haven't seen their
implementation yet, so I'm not sure how hard it would be. I think their implementation
are, it's quite new also. Maybe also to your question. Pandok also like to cover some of
this formula, like we just like the work mode, and also of latex, are you planning to be
future parity in this side? Yeah, the question was, they cover penocorfered popular formats,
like automatic, latex, am I planning to be basically on par with features? It's hard,
because they support so many formats. At least take you a full year to actually be on par,
at least. I try to cover as many formats, but I focus on the more popular ones, because they are
more important. I'm not a 100% sure, I just do the really good. You said that you have millions of
old documents. If the idea to convert them all in, you know, more than format, or is it just
so that people can still access the old formats, because if it's the first one, maybe there should
be some automated batch where you're hanging out converting. Yeah, the question was about having
millions of documents. This was the case in the Dutch government, and your question was
to have it sorry. Are you intending to convert them all, which would mean having plenty of
them through this, and in terms of doing what you showed, you know, my girl? Yeah, so the question is,
are we planning to convert them all? The Dutch government has quite a bit of documents. It's also
hard to have a pipeline of it, because it's also separate governments. It would be quite feasible to
have a pipeline, but this product that I showed was really focused on the user basically doing the work,
the user being guided, and that's quite necessary, because the source can't always be accessible,
adding structures are not always in the correct order. So the item means that we need to
make guesses about what it should be, or we should just as a user. That's the hard part of the
pipeline is that it won't all be accessible. You would need to validate it, it would need to let
you choose, to let the user choose, as to make it accessible.
I have another question in your list of potential features in the future. You mentioned PDF
as output, I'm not sure if it was input or so. Yeah. Why for this, for example, in output PDF,
when there's quite a lot of solutions that already exist, like including just printing, for example,
or or, or so, lots of features that already exist to convert the HTML media for
problems. Yeah, but I think there's something good to do. Yeah, the question was, why
focus on output PDF, because they are quite a bit of solutions. I would say depends on money.
This is interesting for last week, so that's why it would be interesting for me to build.
That's a short answer.
Yeah, I'll explain.
I've looked at the unified Js. I've looked at the unified Js.
They're seeing objects, but I will after I'll just talk now.
Thank you.
It's great to hear this. We actually do something like this 20 years ago, so we decided to take a first
route and sort of handle the optimised word and made it for us to stay alive and to come there.
So I learned a bit about the last few format. We'll just talk in the docics to look at it.
But I confident that you can never become future complete as a connecting to the first question.
Are you not always in the life of someone who's trapped, so you speak about that always
to another breaking future and then so you can't quite get there, because I don't even seem to say,
this is not a problem. Yeah, the question, am I confident that I can be future complete?
Basically no. The well, yeah, the docics contains so many applications that's practically impossible.
I do think it will be feasible to convert most of the documents that users make
fully, because they won't use the entire set of orders available.
Being fully on par with which completion, on the external specgets,
the spec is so big that it's practically impossible.
That's what we just want to provide for the users' set of templates that you will speak
template for. Yeah, using templates makes it easier, because you actually control the inputs,
so it also makes it easier to convert, because you know, but you expect.
And that's the result, it includes some sort of check of provenance and approval of it.
So, what do you mean?
You make a transformation of content, does that result? The content, does it contain
metadata that says it's origin is here, and it was transformed by this and this and I
guess that it's condemned correctly. If it was metadata, it does include some metadata
for the moments. I think authors, I mean, that's a data generated by your transformation
that is done correctly, and it does not describe if it's done correctly, but that does mean
you need something like that. Sorry? You see demand for anything like that.
Yeah, it would be very interesting to have something that also checks if the conversion was done
correctly, but that means that you are not sure how this practically would work.
And sounds like you need a second application to check if the conversion would be correct.
But it is, I've heard it before in government circles, especially where they want
the conversion as correct as possible. It could be interesting, but I'm not sure how it,
but it would look like practically.
Sorry?
Yeah, because she was, do I provide round-fripped conversions?
Sorry?
It didn't happen.
What do you exactly mean with it?
I don't know if there's anything that changes, and then it's said back.
And really, other parts, which you don't support, still, it says, like,
listed tables or whatever.
It really depends on the output formats, on what is actually supporters.
Not sure if it answers your question.
But, like, there's actually some support, just keeping the data
because you don't understand.
Oh, do I try to support keeping the data that I don't understand?
This moment and not, but it would be interesting,
but it would also be hard to create a spec around that.
But it could be quite interesting, for example, if you letter improved,
document conversion.
Because then you can basically confer it again with the current knowledge.
But no, I don't do that at this moment.
Okay, do we have any questions?
No?
Okay, so thank you.
Sorry.
Okay.