So the first pitch of the search room, we will talk about language support in search engine
and we will take the example of the military because I am marketing.
So first of all, what is the military?
The military is a search engine, an open source search engine.
Our goal is to provide a fast search engine and easy to use search engine.
So that's our tool motto.
The clear code here leads to the official website.
A bit more about me, so about me, I have five years for my research,
as an engineer in the search engine site,
so working on the old we store the data and old we search in this data.
And I work more specifically on the language support
with the goal of supporting as many languages as possible.
So what is the subject, so the agenda?
So we talk about the challenges of language support.
So what kind of problem we could encounter during our journey of supporting many languages?
Then our choices, old we did with address in many search,
those challenges.
And yes, the sort one, sorry, it's the conclusion,
but it should have changed that.
So, but first of all, I would like to make a smaller remainder of why all language support work in search engine.
And most precisely why it's important.
And here it's a smaller example of a tiny search engine.
Basically a search engine, what is it, is something where you put some documents
and it will extract words using a tokenizer,
which is the main part of our subject.
And then on the search side, you will query using other words.
You will extract word and try to retrieve this word in the documents
containing in the database.
And basically, what is the most important thing when we are talking about language support
is to extract, to be able to extract this word,
not manage them in order, for example,
to when you search, you know, with a capital H,
you are able to retrieve, you know, with the capital H,
but you know, without the capital H as well.
So this kind of process, you have the two main part of the tokenization
is one, the segmentation.
So where you have to take a text and split this text into several words
and extract word.
And then the second part is the normalization.
The normalization is where you unify,
and of the same kind of word together in order to be able to retrieve them
with a specific or another writing.
So for example, capital H or lowercase.
No, that we know all the basic search engine work.
We will speak about the challenges of language support.
The first challenge we have is the diversity.
There is a lot of languages.
Even if we have like 50% of the web that is written in English,
we have the other half of the web that is written in other languages.
And most of the other half is written in 15 other languages.
But if we count all the possible language we have in the world,
we can count around 6000 language in the world.
Obviously, they are not old on the internet,
but maybe in the future we will have a lot of language
and the ratio between English and other language
we decrease with the years.
And so when you start to have a community,
where you have some users that are not English speaker
and wants to use your search engine in English,
you receive some feedbacks.
And the feedbacks can be really diverse
and because each, obviously,
because that's different languages.
Different languages means different specific features.
Some things small things to support.
So for example, you have some language
that we have the segmentation is wrong
because just splitting on a new space works well in English
but maybe not in other languages.
And sometimes it's under normalization side
where it's more difficult than just lower casing character
to be able to unify the same kind of font.
And we not dig deeper into each case
but I suggest you to read them after.
It's really funny what you can encounter
in terms of diversity of problem.
So here, what is the issue is that
one, the problem are really diverse
but you must have the knowledge of every language,
every specificity to understand what you have to do
to support this language, which is basically not possible.
In my search, we are 25 working in my search.
Well, 25 people are not speaking 6,000 language,
obviously.
So let's dig deeper in a specific case.
So for example, the Japanese segmentation,
as you may know, Japanese is not space-separated.
So full sentence is written in full character
without any space inside.
And the funny example, which comes from presentation
from Minohou Zuka, which is the Japanese developer,
you may have a full presentation.
So if you want to know more about O to support Japanese,
the normalization, the segmentation, et cetera,
it's a really good presentation that is the character.
But one of the slide is speaking about the segmentation
and O difficult, it can be to segment words in Japanese.
And for example, for the same sentence there,
we have two kinds of possible segmentation.
That means different things depending on O,
you segment the sentence.
And so that's O difficult, it can be to segment
the text of a specific language or another.
And as you may know, we have the same kind of problem
on the normalization side.
The second issue we could have is the language detection.
The language detection, unfortunately,
is kind of an approximate detection.
And even if it's kind of works well
when you have a set of documents like thousands of documents,
it's easy to detect the language on over
these thousands of documents.
On the other side, on the search side,
because you have maybe one or two words,
it can be really difficult and really
approximate to detect the good languages.
And for example, if you are on maybe a Korean dataset,
it's really easy to detect that it's Korean,
because the Korean script is only used for Korean.
But if you are on a Latin dataset,
you don't, it's really hard to make the difference
between Italian, French, English, et cetera,
and you have to use approximate algorithm.
And as you can see in the example,
we have to wait the 7th, yes,
the 7th character to understand that it's not Chinese,
but Japanese.
So for a queries, it can be really difficult
to detect the good language.
And when we detect the language,
maybe we talk in a, we talk in a,
the query badly,
and we don't treat the document.
So it's really impact the relevancy.
And the last thing I want to talk about
is about the question itself.
Do you support this language?
Is it easy to answer this question?
Not really.
It's not really a yes, no question.
Because yes, you could say yes,
I do you support English, for example.
But yes, obviously I support English,
I speak on my space and I,
and I, and I lower case every word.
So what could go wrong?
And just in this example,
the user come back and say,
okay, but why, when I search,
can't, I cannot find any document containing cannot.
It's relevant in kind of,
and well,
it's a bit more complicated that yes,
or no, I support this language.
So about the military approach,
what I visit, I just want to,
yeah, okay, perfect.
The first thing we made is
relaying on an open source.
So the first thing we we made is
extracting the tokenizer,
the milli search tokenizer,
in a dedicated repo.
And,
and we worked on
the easiness of contribution
on this repository.
So we extract the code,
this way, the code base is way smaller
than the full, the world search engine.
And the second thing is,
we focus on what kind of action
the contributor has to do
to add a segmenter,
add a normalizer,
and in an instance,
the language report.
And there is two main thing,
for us.
Here is the segmenter.
So all do we segment the text.
So text into a set of words.
What the contributor has to do,
as to know,
is only oh,
should there a segment,
Japanese text,
or an English text,
into a list of forms.
That's all you have to know
when you contribute to charabya
to the tokenizer of milli search.
And that's,
and you don't need to know
and oh,
it will be integrated in milli search,
or in the tokenizer pipeline.
So just one function to implement,
text into set of words.
And then,
for the normalizer,
it's basically the same.
We made the same approach,
but by character,
because most of the normalization process
rely on characters,
more complex algorithms.
Some time, yes,
so we have some,
some alternative way of,
of coding an normalizer
to be able to be a bit clever.
But most of the time,
the user say,
okay,
this character should be normalized
like this in main language,
so that's all.
And so we provide the same kind of function.
We give the input character,
and we say,
okay,
oh,
should this character be normalized,
and that's all.
Then,
for people that are not developers,
because one of the issues,
as I said,
in the diversity,
is that we,
you can't know everything on every language,
and sometimes,
people that as the knowledge of the language
are not developers,
maybe that's linked with or just native speaker,
that know a bit,
but oh,
the language works.
And so,
the idea was to create,
open discussion on GitHub,
and in this discussion,
what we had,
we had some,
some users,
or some native speaker,
that were speaking about their language,
oh,
the language works,
etc.
And we had some contributor,
coming in this discussion,
didn't know anything about the language,
but that were available to develop
or code,
or implement a new segmentor,
or a new normalizer.
And in this discussion,
we were kind of,
had a code,
a codeation space,
where we discussed,
on the few improvements we should do
to enhance the language report.
Funny thing in this kind of discussion,
is that sometimes,
speak native speaker,
are speaking together,
in their native language.
For example,
in the Japanese language report,
I have a world discussion in Japanese.
I don't understand anything
at the end of this discussion,
sorry,
but at the end,
we managed to enhance the Japanese,
Japanese tokenizer.
So, it's a win for me.
Yes,
and another,
the fact about the Shahrabia,
is that some commits are in Korean,
so.
And then,
our last subject,
so our last subject is the language support.
Sorry, the language detection.
For us,
unfortunately,
the language detection is still an open issue.
The fact that it's a next,
approximate way of,
of guessing,
it's a guess,
on the language,
and because,
if we are wrong,
we chose the wrong tokenizer,
and we don't re-trive anything
with our,
our charging chain.
We tried several approach.
So, the first approach we tried,
is like,
okay,
because we know the documents,
what we could do is,
kind of,
guessing,
a list,
a subset of language that can be detected,
detected for this database,
because we know that,
the documents are only in Japanese,
and in Chinese,
for example,
and that's all.
And then,
at query time,
we just have to detect between these two language.
But, even with,
with this strategy,
unmonoling,
well,
database,
only English,
or only Chinese,
it works well.
But,
if it was,
a configuration,
a user configuration,
it would work well,
as well.
And on multilingual database,
so, for example,
if you mix Japanese and Chinese,
we add the same issue.
So, some time,
the,
at query time,
we mix detects Japanese into Chinese,
and,
and inverse,
in the inverse.
So, at the end,
we didn't treat the document.
So, the second approach,
we,
we add is kind of,
get rid of the language
detection,
and ask the user for which language is using,
and we only rely on the script,
because it's way easier to detect the script,
than the,
the,
the language itself.
Yes,
and,
about the last question,
which was,
do you support English,
do you support core and do you support everything like that?
As I said,
it's,
it's not,
an easy question to answer by yes or by,
or by no,
but what you can do,
is explaining,
oh well,
you are,
you support this,
specifically in which,
and that's the,
that's the goal,
we,
that's the thing we tried.
So,
it's not the best way to,
to provide the information,
but,
it's the first,
it's the first step.
What we,
we did is,
instead of,
you're saying yes or no,
we say,
okay,
we have a specialized,
a psych mentor,
that does,
that, that, that,
and we have some specialized,
normalizer,
that,
normalize,
your character,
or your text,
like that, like that.
Are you,
are you,
we,
are you,
like that.
Are you,
are you,
has well, and sometimes it's in the language distribution, and so some external
contributor can come improve, contribute to the tokenizer, and improve the tokenizer.
What time is it?
Yes, perfect.
So the conclusion.
So the first thing is that we continue everything.
There is too many languages to know, and we have to find strategy to
gather as much information as possible to support as many languages as possible.
And what we did is relaying an open source and open discussion to a gather information
and to implement and to co-implement the language support.
The second thing about, to talk about is the language protection is not accurate, and
unfortunately we don't have a good way of detecting the language.
So what we did is relaying more on the street instead of the language.
So I have more generalistic tokenizer thing, and if a user wants something really specific
to its language, he has to say, okay, I'm in German, and there is only German, so I'm on
the specific future of the German.
Yes, language support is not a yes, no question is the gradient scale, so you don't
say yes or no, but you can explain what you support, and what you don't support for this
specific language.
And that's it, if we have some question, if you guys have questions, if you don't have
a mic to pass around, you speak loud, and he's going to repeat the question for the
correct.
But this talk is very appropriate for you to respond to the data from, but I have a question
about the format of the number of things, are you also thinking about it in a short
time?
So I think about internationalizing the data in the way interpretable dates, numbers in
different languages and different countries, so what do you mean?
This is, do you interpret the format that is like the dates, the number, etc, that is not
only converting characters, but maybe reformathing or understand the meaning of what's
being.
That's it.
Yeah, yeah, okay, so unified the date, for example, yeah, so yes, it's a good question, so
the quick question is, is it the job of the tokenizer to reformat or to change the meaning
of something?
It's a good question.
I think personally, I would more add this as the consumer feature, so for example, if you
have a search engine using the tokenizer, I would say that it's more the search engine
job to convert a date from a loser, but that's the question that that can be escaped.
I have no read on source, sorry.
Yeah, we have a question from the room, is it going to read it out loud and try to answer?
Yep.
Question for the term, I've noticed that most monoling well data set are aren't really
monoling well as English, I'm German, not limited to two, words are everywhere, as
many search liquidity, do cleaning up the data set that the long wedge detector, like
quite long are lingua, generate, they are used, you're received from, sorry, I'm sure
for getting something to do, pretty loud, so people can get it down the video.
Yeah, so as many search as clean the data set based on the long wedge detection, I'm
not sure to understand the question, but what I say is, in Moscow, we suggest to split
the data set into the long wedge into different language, for example, German and English,
we split them, however, if we are talking about specific German English data set, the mix
of these two, considering that English is German, it's not really an issue because
for the German language, the most important thing is to split some of the German
word, because German is an agutinative long wedge, where you can put several words together
as a single word, but German people expect that you should be able to search each
inner word of agutinative word, however, in English we don't have that, so it's not incompatible
to where we have to connect, so it's kind of case by case, I don't know if I answer
well the discussion with it. So the other thing is, we have a matrix here, and there's
a second question, I'm not going to ask you to read it, but if you guys want to go into
the matrix room and answer the questions and engage the people that are asking the question
that would be great, okay? Okay. Now we're going to continue with question from the room.
Yes, so the question is, if we can't detect two or two languages, why don't we want to
run a search for each languages, so for example, if you don't know if it's Chinese or
English, you run a query for the Chinese, a query for English, it's completely possible,
we can do that in a really search, in fact, using federated search, but the fact is that you
will run a search for every language you have in your dataset, so this means that if you have
like 20 languages for each end user search, you will run a 20 different search, so that's a
cost to pay.
Hopefully better today, because the tokenization that you have to provide as a text to go in,
is something that a user can probably even want to do, and possibly something that is self-insourced,
feasible, as it could probably be for a company, we need to hire someone on the technical
search, so I have a try to learn it, or I have to immediately try to pay all of the things that
this is a lot of people.
So I try to, yes, sorry, so the question was, did I try to learn O tokenizer of the work in the
world?
No, did you try to machine learn the tokenizer?
So you don't get to take the document to parse it in, and you emit instead of the original
central instrument separator, or potentially multiple separate, which you might need for the
contracts like a term.
So, yes, yes, yes, so did I try to machine learn the document, so basically we take all the
document of the dataset, and we try to guess the O should we split the query letter, I'm
right, on this side.
So, to be honest, I didn't try any machine learn on dynamic tokenization like that, we are
kind of in a static way of tokenizing, so we know by in advance what O should we tokenize
this sentence or this sentence kind of, and we apply it for the document and for the search
as well, maybe it's more a question for a formulator, but the question is that in
milliseconds, for example, we are, we don't have all the dataset directly, but we build
the dataset with the user, adding more and more document, so all those it impacts this
dynamic, the dynamic tokenization, I don't know, and if we have to reindex the world
dataset, the world, the all the document that are already in the database, each time you
add a new or a few documents, it can be a problematic for us, but to me, I didn't dig into
that, I was more in the traditional way of tokenizing the tokenizing sentences.
Thank you, thank you, thank you for the listening world, I'm just intrigued by all the way
I set things up as you do the segmentation and then normalization, which is the other way
rounds to have the students done it for 35 years, I just want to bring this up, was there
a reason to choose that, that order of everything, so is it, you know, you're worth it
Yes, so the question is, in this tokenizing in Shabia, we are first running the segmentation,
so we segment, texting to word and then for its words, we run the normalization, and
way we do that and not normalization first and then segmentation, it's because of the
highlighting, in the research, is a, is a prefix searching gene, so put on the celly in the query,
you have only a part of the web, okay, and normalization is not only, is not only a character
by a two character, but maybe character to string, sometimes you convert one character to
full string, etc, and it's way easier for us to retrieve all to highlight the character,
I should highlight in the document, which is the non-normalization version, okay, when you apply
first the segmentation and then the tokenization, because you're just after to have some offset,
some relative offset based on the start of the word, and if I had to do the same, but by
applying the normalization first and then the segmentation, I would have to, to think, to think
in an absolute offset, which is way harder when it comes to highlighting a part of the word,
so yeah, I don't use information when you are normalizing first, and then tokenizing
doesn't it break the tokenizer, which I normalize in the first? So, those normalizing first
breaks the potting celly, the segmentar just after, so it depends on a segmentar, I know that the
Japanese segmentar would prefer to have a first phase of normalization to be sure that all the
characters in the same format, and it's way easier, because in Japanese you have several
different formats for the same character, you have the alpha character and the plain character,
and for example, the Japanese segmentar works better with the only full character, and if it
encounters alpha character, the segmentation is less accurate, but it's not really common.
That's a good question, so that's not possible, yeah, sorry, sorry, so I gave the example of the
English, the English can count versus canot or canot, and oh, we are doing under that character
by character in terms of normalization, you come to it, but to support that, you have to mix
between the segmentation, so first you segment can and not from canot, and then you normalize
sorry, and don't have a tick, a tick in to not in order to unify, and so for this specific case,
it's not a character in normalize, but to can normalize, which is a bit more complex to implement,
in the xarabia code base, what we did is we have some interface, kind of, where the character
normalize, it's a token normalize, with less method in it, so it's a subset of token normalize,
but you can implement a token normalize, if you want, it's just a bit more complex.