Welcome everybody to my talk, bridging the gap from WordPress to RepLage. As always,
a few words about myself first. My name is Nicholas, I'm technically at Aluga.com, which
is a multi-lingual with your platform and dubbing tool suit. I'm personally
consider myself a restation and nix enthusiasts, so this topic might seem a bit out of place
from my usual conference topics. I do like computer graphics, interactive media, pen and
paper role playing games, actually what I missed here for this room. I do like natural languages,
so I'm fluent in English, my name is Muito Sparford Storage, Aluga Espanyol, but I know I
have a lot of people who don't know how to use it, but I'm not sure how to use it, but
how to use it. So you can find me on the web at Quartz.f and I'm also on the Federer's,
on Nicholas Quartz.com, where I'm also personally one of the elements that is running
this instance. And now that we have covered that part, also this is my first
foster, and as of a few months ago, my first time helping out with a death room. Thanks
for taking over DAV, so I can close this tip. Yeah, so what do I expect from this talk?
This is not a mature, open-source project that you can like grab and throw at your web
present instance and then have it become multi-lingual. It's mostly about sharing a fun
little story, a bit of a hacker spirit, how I solve the translations for our own
road press, power, marketing and product pages, and yeah, involves three free and
open-source projects, road press, elementary, which is based on road press and
rub late. We'll start with the motivation for this project, then we'll talk a
bit about what the initial idea was and what solution we arrived eventually, how
I implemented it, and what conclusions I derived from having finished this project
and roughly the past one and the half years, which was not like full-time job
but here and then adjusting it and hacking on it. So first of all, why road press, for
our product marketing pages at all, initially like our company existed for the past 12
years at this point, and for most of the time we were fine implementing our whole
marketing product pages directly as a collaboration between marketing and designers
and developers, but as we all of us probably know, the development time is expensive,
so needing this feedback loop back and forth between marketing, designers and developers
takes a lot of time that would like to spend on our core product itself. So we decided
two years ago that we would move this process to road press, because our marketing team
was familiar with already, our new designer was familiar with already, and yeah, they
can just adjust it at a text, at text, redact text, without involving the development
team. I'd like to note that this decision was made before the whole road press was a
WP engine thing one and a bit a year ago, so I'm not here to say, use road press for
everything, I'm just saying, it worked out for us, it's still working out for us, if
you know, better solution than just go with it, know how to feel it. Why rep late?
There are, of course, existing rep press plugins to do localization internationalization
to rep press CMS pages, namely WPML, PoliLang, and G-Translate, and we did try them out,
but they, the whole workflow, felt unfamiliar to our in-house translators, and our freelancers
who are much more used to rep late already, because we're using it for our product
translations, so the onboarding like for rep late was unnecessary, while we would have to
like give more to each of our translators, give more teaching, how do you properly use
this tool, how do you, how are you kept in the loop when you translations are required,
whereas it's rep late, they just open our rep late instance and already knew, okay, these
are the strings I have to translate, nice, or check, or edit, whatever. Also, rep late has
everything in one place, like it allows us to easily take existing translations, we had
already for new, similar strings, for translation memory, you can define a glossary, which
is really nice, and also, of course, it integrates with automated translation APIs so we
can, like, have an initial automated translation, and then go over to the translators in
our house, so you can see these are the translations that were automatically generated,
check them for correctness, precision, whatever.
Okay, so, of course, the final target is to get every translatable string on our CMS pages
into the rep present, into the rep late instance, how do you do that? Like, we have quite
a bit of text, like marketing people, it's like to talk, and that's a lot of stuff to
translate, and also, as opposed to UI translation, where most of the time you can just
say, I have one string that interpolates values, marketing people always reinvent the whole
text so you basically have to translate it completely in you.
So, rep late can translate a lot of formats, in our case, it's not the rep restator base,
you can just plug into rep late, so you need to find an intermediary format, and for most
time when you're having an automated pipeline, JSON is the ideal format for that, and
rep late can handle JSON just fine.
So, that means we need to get JSON message files out of our rep instance, so we can
feed them into rep late.
How do you do that?
Speaking, we set a key, we want one content block in the page to equal one string in
our JSON file, so one message in our rep late instance.
That's a bit much of generalization, so some block types can have multiple strings, like
lists can have string for each list item, or also if you have mixed content, if there's
some other HTML that's a bit more complex in between, then also you might have multiple
strings for a single widget type, but that's at this point in implementation detail.
Then from that later we can again export JSON and have to feed that back into our pipeline
some hour have to get that into the rep press page again, to like, serve the localized
version.
So, in-house we don't have much experience with PHP or rep press plugins.
It's not zero experience, but it's enough that we have set up here, we're not going
to develop this as a rep press plugin, but I personally have a lot of experience with static
page builders, and this is not much else than a static page builder, except that you don't
take markdown or like an SQLite database as an input, but a rep press HTML page.
I will briefly mention why we didn't go with the rep press rest API in the moment.
Now the problem is, if you want to have like one message per content type or content
block inside the page, you need to be able to uniquely identify the content blocks of the
page.
You need a message identifier that the JSON and then rep late can use to say, this is the same
message when it changed in the original language, and then potentially has to be adjusted
in the toggle language.
Unfortunately, the go-to solution for WordPress, which is the Gutenberg editor, does not
have stable unique IDs.
You can check out these two issues if you want to see a very lengthy discussion about that,
and also you will unfortunately arrive to the conclusion that unique stable IDs will never
come for Gutenberg blocks.
There was very sad for me.
Yeah, so what can you do instead?
You could implement custom block types and then say, okay, this is an attribute on my custom
block that is just for the message ID, and then have your marketing or designers, insert
the message identifier or like generate a random one, so it's stored inside the block
and remain stable.
That's problematic if you're copying the pages, which are marketing, people are doing
a lot, because they don't want to start from scratch, right?
They have like a template, and then it looks somewhat familiar for all product pages, and
then they just replace some blocks, edit some details, et cetera, and then you need different
message identifiers for the blocks where they change the entire text.
Also, you can't do the same for like, built-in blocks of Gutenberg, which means you
can't like, you have to reinvent the wheel if you want to go down this path.
So we said, okay, what else could we do?
We could just say what's the one thing that stays stable for message?
It's the hash of it.
We didn't stay at this, but for a moment, that seemed like the only solution we got, like
to take the shot to 256 hash of the message, and then use that as an identifying
rep late.
But that comes with the downside, whenever this text is edited, that's considered a complete
new message inside rep late.
Now translation memory makes this somewhat okay, because maybe rep late is able to stay,
still say, okay, someone fixed the type in the original English text, but this is still
the translation that should be assigned to this message.
But at this point, another thing came up that our designer was a lot more comfortable with
the other matter rep side builder.
Elementor is theoretically speaking, also practically speaking, free note and source, so far,
it's like on GitHub and you can get it under the GQLV3, they have a pro version, but
of course, GPL and commercial software do not necessarily cancel each other out, like you
can get a license, and then you're free to do with the code, whatever you want, they're
just not going to put a pro version, source code themselves on their rep side to download
without buying it.
Now Elementor, which they call their block, like equivalent, their good block equivalent,
do have stable IDs, and that made localization of these pages a lot easier, and also
you can attach custom data, like they have inside the Elementor rep side builder, they have
a section for data set attributes, and if you've ever coded plain HTML, you might know
the data dash attributes, where you can assign whatever name you want to put in whatever
you want, and then do with that what you want.
So a bit more about how do these identifiers for blocks look like, this is like one part of
our landing page, so we have like four text widgets on here, so you can see that we want
to translate, more namely we have a headline, and we have something called a text editor
widget below it, and if we look at the HTML source code of that, we can see they also
expose these things directly on the HTML you get from WordPress as data set attributes.
So we have this diff, like every Elementor widget is wrapped in a diff for the widget itself,
like now for semantic HTML that might hurt a bit of purists, but for our case it was completely
fine, so we can get the ID from this widget, we can see what the widget type is,
and now if we want to search for everything on the page that's written, the translateable,
we first pass the HTML, get all the elements that have this element I widget, and then filter
for which it has to run to translate. Of course, the localized part like the part we want to
replace and translate is not directly the element itself, so we still have to do a bit of
the send until we hit the text node translate, but it will save it more about that in a moment
in the implementation part. How did we implement this? Originally, we implemented this as an
on-the-fly server in Rust, because we said, well, maybe people aren't accessing specific pages,
and even then we can just catch them on the fly, but eventually we recognize that we're updating
the implementation to work around certain element issues much more than we're changing the pages
themselves. It was easier to say, we go with TypeScript so we can easily exchange the details
of this pipeline and then run this as I see eye pipeline when added, ever the pages were updated
and then deployed to our web space, have a CDN in front, etc. Still, this is relying a lot
on the Rust ecosystem, because we're using DNO, we're using the HTML parsers, DCSS parsers,
from the server project, and yeah, that made things a lot easier. Now, I already mentioned,
we have a fixed list of translatable widget types, I will show actually what that means.
If I go, this is a resource code, I'm planning to, as I will mention the end,
planning to upload more generalized version of it, I didn't get through it before first
of them, unfortunately, I'm sorry, but very up here we have like a list of translatable widgets.
So, headings that are any kind of headlines, level 1, 3, we want to translate that text blogs,
which is mostly paragraphs, but not only paragraphs, we want to translate that there are also
some buttons we have to translate. Those are a bit ups, because they have a very convoluted
content path, so that has some special handling, but overall, that worked out well enough.
Okay, so the pipeline itself starts with some pre-processing, like we have to rewrite your
else for where we're deploying it, download the assets from WordPress, maybe modify the CSS,
and also adjust the CSS a bit, as I will show in the next slide. We want to extract the messages
from the WordPress sites where you can generate JSON file for Replate, we want to translate it
inside Replate, so that's the part where a bit of automatic translation followed by human
quality checks comes in, and then we want to export it again from Replate to JSON and feed that
into our pipeline to generate the final localized version of it. For the pre-processing,
there's some things that are very specific to our setup, so they're not particularly interesting
for localization itself. For example, we don't want to just pull in the whole WordPress page and then
serve that to the end user because we're integrating that into our larger page, so that means in
particular, we have this page, that's what it looks like on our WordPress instance, like that doesn't
have much around it, we're stripping footers, headers, etc, because our website itself already has
that, then we're feeding that into Replate, this is, for example, the strings we have for this
rep page, so that's every text you see on this page, basically, is included in our
template instance to edit, quality check, etc, and then we end up with this on our actual website,
so that's like a React shell and in between we're embedding this static HTML that was output
for our page here for the German local. We're also removing all JavaScript from the
website, we're currently not allowing the marketing pages through embed scripts because they
would be able to access the cookies of our main product, it's still a bit of an issue for us where
we're considering to actually split marketing from the actual product of different domains,
anywhere on who knows like host names separation for cookies, probably has run into this before,
and what we're also doing is we're wrapping all the CSS in an additional ruler, so an additional
rule, so for example, if something changes the font rate for any heading line, we won't,
I don't want to have that effect the rest of our product, right? Now, how we're doing that
is through a package called Lightning CSS, that's based on also the Rust server CSS parser that I
mentioned before, that allows you to transform basically any rule you have in the CSS file cheat,
that's very neat if you do custom, like if you have custom things you want to do to your CSS.
And finally, we're hedging all the files we download from WordPress and then append the
content hash to the file names. I know there are WordPress plugins that can do that like directly
inside WordPress, but at this point we already had our custom pipeline from start to end and we're
like, okay, we do download the files, we can just hash them and append and there's no need to
involve additional PHP plugins in this process that potentially slow down the rest of it.
Okay, so how do we extract messages? I already mentioned that we're using servers,
HTML parser for that, that's needly wrapped in a package called DinoDom, but they're also
MPM packages and also other programming language packages that can access the same parser
and allow you to efficiently pass and search through an HTML DOM tree.
We're looking for the Elementor Witchit inside the DOM tree, I showed you the State her tie Witchit,
which we can just scan for. There are also like, if you know, beautiful soup for Python,
there are similar packages for Dino and MPM, where you can just insert and see as the
selector, get all the elements that match the selector and then work on them.
Then we're checking if that tribe is translatable, if yes, we proceed, then we curse into this
widget until we hit the translatable parts. At that point, I want to note that not only text notes
are translatable, like text notes are the parts that are like in the very end of your DOM tree
that the actual text content is contained in, you also have to translate things like all attributes
and titles and labels, so you need some special casing of course for these. And for the things
where Witchit are not sufficient or we can't detect the actual text content inside of the page.
We have a fallback where we just assign a dataset attribute with a custom message identifier,
so that's like the really last resort, if we see during testing a, this message didn't end up
in replayed, why is that we look into it, assign the custom message identifier and then we're done with it.
Now that we've extracted the JSON file, actually, we'll show what that looks like.
This is what we get from our pipeline for the page we had just now, so that's showing the page
slot, then the widget type and the unique ID that elementary has assigned internally,
and then we're just passing that two replayed, which we saw here, and can edit them in replayed.
So now we are done with the automatic translations, our team has looked over the messages,
done some corrections to say, this is fine, we can deploy this,
we will need to somehow feed these messages back into our WordPress pipeline.
That has to use the same algorithm to detect what content notes should be translated,
because you need to, of course, you need to replace the same note like the same text notes,
with the same ID, the same message that's assigned to the ID, or the same alternative view,
to set error, so that's all happening in one pass. And then we get the translations from
the JSON and insert them in the position of the note we want to replace. We're doing that
at the end, yes, but does it involve placeholder attributes? That's what I meant with special
handling, so for example, all attributes, label attributes, etc. That's like that needs some
special rules to say, this type like input types have placeholders or title attributes can be on
any element. That's title attributes are something you don't handle at the moment, but for example,
all attributes and images are something that we'd like just add, with a special message,
identify it into the JSON, yes. I will have to think about that at the end or speak
private, yes. Also, we realize that we need some inline HTML. For example, some languages don't
like italic, italic text, I think, cursive text. In the rabbit you don't do that usually,
but like the main English version, the reference version, has this cursive text. And then the
translator can say, okay, this has a BTEC, I removed that from the translation, but other
translations might want to heat that. Or also something we realized is like we originally tried to
split the strings at the boundaries of like the HTML elements, and then we had text note,
bold text text note, that of course assumes that you have the same order of these words
in the translation, which is not the case. So you might like start with the bold text that the
rest is the normal text. So instead we said, okay, formatting should be allowed inside the messages,
we take that as one thing, and then we sanitize the messages, we get back from wrap like to allow
only formatting, and also in some cases, anchor links, because you might want to link to a different
version of a page depending on the local U-translate U. Yeah, so that already lets us arrive at the
conclusion. My conclusion from this project is if it's HTML, you can translate it. It really
mostly doesn't matter where you got that HTML from. And if it generates and consumes a map of
messages, you can use wrap late. You just need to get from that step one to step two. Ideally,
you have text only notes, and you know how to stable and uniquely identify them. The next best thing
is you know how to reliably find and replace the text inside these notes, that's basically what we
had to do in the end. And if you don't have a stable ID, you could fall back to the ID,
an idea of hashing the original message, but it's not ideal, so maybe try to find something that
uniquely and stabilize the text first. Also, source code, I mentioned, I didn't manage to like get a
cleaned operation of it up and running on codeburg for today, so I already created the repository.
If you want to start a bookmoderate and then it should arrive sometime next week, and you can
like get a better idea of how this isn't fermented. Yeah, and with that we arrive at the end of the
talk, and I'm happy to answer some more questions. How do they work around the lack of
a stable ID and put in book? So how do other repressed translation plugins work around the
issue of not having a stable ID in Gutenberg? The Gutenberg blocks do like they are somehow stored
in the MySQL database, so they do internally have an ID, but there are no plans to expose them.
Now, as a WordPress plugin, that's not that much of an issue if you know where to look,
so what these translations plugins do is they hook directly into the WordPress database
to identify these blocks, but there wasn't an option for us because we're not too deeply
involved in repressed plugin development.
So the question is, or it's starting with a statement, like you can uniquely identify a web page by its
URL, or in this case it's a repressed page name or path, and then it's unlikely that like two
two times the same text at the same page would have a different translation. So that's not the
issue I wanted to get to. The problem is like you have like the original English version as
your reference, right? And then someone might to change the words, but the meaning doesn't change,
or you might like change punctuation, change a typo, etc. Now the string of that string,
the hash of that string changes, so it's an entirely new message inside of the JSON export.
Now as I mentioned in rep late, you have this thing for translation memory, so you can still say
up to a certain similarity that should like automatically assign translation,
that of course helps rocking around that, but we also don't want to put the threshold
too low for translation memory. Like if we have very similar sentences that actually mean a
bit different because of like one word changed, then we don't want to automatically assign the
translation of a similar structure. Now with the unique idea on directly the widget that has the
content, we can say this is the same content widget, even if like one word inside of the change. So
it's probably okay that translation is unchanged, but still our translators are notified
this like this string has a change, the original message, please double check it if the translation
is correct. If things don't content is changing, then that's not a lot of the question,
if the content is stable, do you still need this? Probably not, but it's easier to work with.
Like if you say I'm translating the HTML page in one go and then it never changes,
that's probably fine depending on what you do. If you never like if you're absolutely sure
you're not going to edit this in the next two years, then yes, sure, that works.
As I said, this repository is currently empty and I'm going to upload it next week.
Then thank you again, a lot for listening and having me today.