So, the next talk will be presented by Sid Rikul Mer, and it's going to be about end-to-end
enterprise search with Datafariq.
Yes, good.
So, I have the immense privilege of having 25 minutes, and my laptop worked right away.
So, when you have finished on time, so I'm Sid Rik and the President co-founder of
France Labs, who has a software maker of Datafariq, and so today, we're going to talk
about end-to-end enterprise search with Datafariq community edition, because we have two
editions.
So, Viagenda, we'll talk a bit about France Labs, organization, definition of enterprise
search, because I'm not sure if we're all in line with what it means.
Let me talk about Datafariq's architecture, the main functionality is, and some Gen AI,
I was going to say that everybody mentions AI, but passable did not mention AI, so it's an exception.
Maybe did not have time too, so you're ready?
Yeah.
Oh, yeah, wow.
Okay, thank you.
So, we were born 40, so who knew Datafariq before?
Yeah, I think you're expecting that.
That's good.
So, we are specialising not being known.
So, let's see, for 14 years of existence, we released Datafariq in 2015.
November, we're basing these on the French Riviera, with a swimming pool.
We speak in many conferences related to search.
That's why maybe you don't know who you are, enterprise search.
We are involved in open source businesses, so we are part of the French,
Cluster, French Association of Open Source Businesses, and European Association of Open Source Businesses,
respectively, CNL and Appell.
Members of the search network and customers in weird industry sectors.
So, the surprise, with sort of a make of Datafariq, so we provide expertise around it,
and also an Apache solar.
So, what isn't the price search?
So, I could be pasted via, what I'm quoting, actually, the definition for Martin White,
and he's book enterprise search, actually, that's been passed here.
He was the only serious book about enterprise search, so it's.
The enterprise search application enables employees to find all the information,
but the company processes without the need to know where information is stored.
And I did, oops, sorry, I didn't hit the microphone, but,
particularly because most customers, we want security.
By security, I mean, here, respecting the access control to documents.
So, what does the typical enterprise search solution works?
Well, it looks like a search solution, no surprise here.
So, we'll detail a bit more what it does in Datafariq, but, yeah, it's a search bar.
You search, you have a complete, you have facets to filter, and you have a results list.
Okay, no surprise.
Any addition, you have some monitoring, like statistics, analytics,
dashboards to see who search is for what?
And how does that usually work?
Well, there are several components from left to right.
You have a crawling face where you connect to the data sources,
the documents, whoever they are stored.
You retrieve them, you open them, you analyze what's in it,
you take the metadata, you take the content, you put it in an index,
search index, then you have the search components,
we take a care of in real time analyzing the user's query,
looking for the data in Vindex, and providing
the results usually in a UI, and also a component for statistics.
Okay, so now you are experts in enterprise search.
Yeah.
Okay, so what the data far is CE allows to do.
So CE is the community edition.
Because it's first them, I don't know.
You choose for a Rocks at me.
So it allows you to crawl multiple types of documents,
and I have some slides explaining a bit more what it does,
and this is thanks to Apache many full CF,
which is two minor ledge, we only existing open source framework,
doing this type of stuff.
We use Apache Chica, we'll see that afterwards.
Data Free CE allows you to do some search using VM25,
which is SQL2TFIDF, and so we will do vector search.
I will talk about it later, and while that we use Apache Solar,
you know Apache Solar, we're in good,
some nodding here, so that's good.
So you are able to do some analysis,
thanks to Dashboards, and you're able to manage your tool,
which is difficult to use interfaces for administrators,
from playing with the weights, doing some promo links.
I don't have time for that, but entity detection, etc.
You can use the admin UI of many full CF and Apache Solar.
So what does it not allow you to do?
So I put easily, because we don't form it anything.
Everything's open, so you can find the time to do it,
just that in order to survive, this is what I post up,
but we put as extra intent to price edition.
And security, and here I'm talking about access control,
paramission document-level permissions,
and to connect to a key directory,
they want to adapt this type of stuff.
Does not easily allow you to do big data search,
in the sense that if you need a cluster of data for a system
with distributed components,
this is not done automatically,
there is no answer, but recipient is kind of stuff.
And it's not easy to do a backup,
well, a nest, of course, if you are the Docker,
and it's persistent, you can just save it, but then.
And we don't prevent anything, okay.
So I hope you feel better now.
So what does it look like in terms of architecture?
So it's one, it's on our weeky.
And yeah, so that seems a gray box,
that's for the monolithic version of Datafari.
So in the gray box, it's exposing,
it's using interface to users on the left,
and we fetch the data, which on the right of the schema.
So you can see we have some Apache, some Tomcat,
some Zuki Perse, some Solos, some Casanoa, some Puzz Gray,
LogStash, for historical reasons,
Tika and I think, I've mentioned all of them.
And it's very, it's a big soup of stuff to manage
and to deploy.
So it lots of fun when doing the deployments scripts,
but we work it out.
So the main components for that,
we're using, so really the main ones
would be Apache mini 4 CF, because that's the very framework
for crawling the documents.
And it's Apache Solos, for the search part,
and it's Apache Tomcat, for owned Datafari web app.
And we provide an admin UI to install managing use.
If we dig a bit into many false CF,
anybody new, many false CF people?
Yeah, no.
No, okay.
Nobody cares about crawling, but it's an important step in,
oops, let's keep it.
So many false CF allows you to connect,
to, I don't know, for instance, I put,
yeah, file shares, I take the directory,
I'm an LDAP web, XWiki.
I mean, I think all of us,
many presentations today were mentioned,
this week is, I did my homework.
We have a connect of YWiki.
Yeah, you can have a pipeline of transformation
connectors to transform the dates,
have a choice of extracted.
Maybe you want to put some regular expressions
to find entities, maybe you want to
print out some particular document types,
or, yeah, I don't know,
if things can be done in this pipeline,
and then we push whatever has been analyzed
into the solar index.
So that's how many false CFs,
so you can see many false CFs
as an octopuses able to connect to many types.
Many types of data management systems.
It's also a month array.
I didn't draw them,
but they are coupé left, I think I hope.
I think the guy is dead when it was years ago,
so that's fine.
So he can open me formats,
to Apache Tika, so PDF, obviously,
liberal fees, Microsoft Office,
database tables, any other types.
So it goes quick, but that's fine.
So now you're expressing that patchymane false CF.
Yeah, of course, now about solar.
So well, solar is job is to take what a patchymane
false CF sends it,
and analyze it and create the solar documents
that it puts in Vindex,
and it's job is then at the search phase
to answer very fast to the user's query,
and to deliver relevant documents classified.
It's similar to open search,
if you just know open search with them.
And for the analytics components,
well, but now it has become pretty easy,
what we do, because in the past,
we were using L,
and then we were fed up with L,
and then we used Zeppelin,
and Zeppelin was grass roots.
So now we got rid of all that,
and we take everything that's come from
from the logs of Apache Manifold CF,
of Apache solar.
We still have log-stash,
we need to get rid of it,
and we put it in the dedicated search collection of solar,
and this is where how we feed our home-made
dash balls in pure HTML.
Also, we stopped frameworks,
like Zeppelin.
So if we detail a bit,
what you currently have with the data for UI,
this is the screenshot of the version 2 of React UI.
It's actually very similar to previous ones.
So this is, and it cost us a lot to do,
but we forgot to modify,
we look and feel so that you have the feeling it's brand new.
So it's brand new, a bit like the old one.
So you have a search bar on top.
When you start typing,
you have your dropdown that appears,
or you can also remove it,
but that's fine.
You have the suggestions for queries,
or the standard,
or the complete,
you can do,
we can have auto suggestions,
which is proposing documents right away,
to add some suggestions based on specific fields.
So if you have product numbers,
for instance,
if it could propose you product numbers,
as well, to complete.
So that's why we drop down.
Then you can see a bit on the left,
the tabs,
which allow you to filter based on a specific type of metadata that you want.
You can also opt for facets.
Any metadata can be a facet,
basically.
And you have different types of interactions with your facets.
Can we have a checkboxes?
Can we have the Windows,
for example time window, or price window,
whatever you want,
or if you missed that,
wait too much.
Thank you.
I hope you have lots of questions of the one.
Otherwise, I will ask you questions.
So this I covered.
You had the search tools.
I didn't put the screenshot about it,
but you can export your search results.
You can save your searches.
You can have email alerts.
So if you don't want every day to come to your corpus,
and ask that data vary,
did the new document appear.
Then the data vary will run these queries for you,
and send you an email saying,
if today there is something for you.
And you can do advanced search,
graphically to combine search for specific fields
with Boolean operators.
Okay, let's clear.
You understood using the face.
There's specific challenge normally,
but we never know.
And you can notice,
little box asks your content.
It's a teaser,
because we will go back to it a bit later.
But you can see it's fair.
Ask your content.
I'm seeing this because now it's time to talk about AI.
So historically we could connect
at the indexing phase to install
other field party systems to do a translation,
or imagine analysis.
In our case, most of the time
it was to do optical character recognition.
We've test your act for for image recognition.
And now it's kind of changing,
and it's more focused on using
Gen AI to do this type of stuff.
So it's actually easier for us.
And so we can connect
at indexing time and at search time.
And at search time,
it's more for the vector search and the rug.
And I will explain a bit more about the rug afterwards.
Visorinjectivity is actually so not co-founded,
but co-funded by European Union.
More precisely the NNGI search project,
called Neural Data Faber project.
View maze by Vologo.
Which is a great logo.
And the N2N prototype is planned
for what we end of this quarter.
So we think we need to hurry for that.
Visorinject is done in collaboration
with CIS, S-E-A-S-E,
company in the UK.
More precisely Alessandro Benedetti,
who is a committer in Apache Solar.
And actually yesterday evening
he did the second pull request on Solar
to add VMBedit's call of the embedding model in Solar.
So the PR has been done,
now it's waiting for reviews,
which means that from now on in Solar
independently from data ferry,
you just need to push the document,
actually like the parameter,
say which model you want,
and Solar will take care of doing the embedding.
Until yesterday it was not the case.
Until yesterday you had to do the embedding externally.
And then Solar was able to do the vector calculations
and vector management,
but not the embedding itself.
So it's quite fresh actually.
So Rag, what does it look like?
So we're going back to videos into phase.
And so the little box I should do on the lower rights,
if you click on it, it's a chatbot.
I'm sure it's not really a surprise for it,
but what does it allow you to do?
You can ask your content.
So the first mechanism is the typical Rag
recovery management generation.
Everybody's familiar with Hague.
Yeah, I have some handshake.
I will take it as a yes for everybody.
So we go as a Rag for all of the content.
But you can also ask questions for specific documents.
So in the search results,
you can notice that there is a little ask with AI.
But then if you pick on it,
the chatbot changes and allows you to ask questions
only for this particular document.
Because I mean, we have customers.
We have documents which are 6,000 pages long.
Which is not a good idea actually.
I think they should do several books.
But okay.
And at least with that,
you can ask a question only for that.
And we have ideals for quick actions.
So it's prepared actually prompt for instance
to do the summarization of a specific document.
And then you can decide.
You can create your own quick actions.
This is meant for for his purpose.
So what is Rags?
I knew you said you knew it,
but just behind the scenes.
So it's a standard.
So user, I did the drawings.
I hope you're amazed.
Okay.
Really proud of them,
especially for brain.
It's amazing.
So we use it as a query.
He's happy before doing the query.
But so data for it as a vector search.
Thank you.
So to the so-law, we request dimensions.
We have a dedicated collection.
Composive mini chunks.
Of the vectors coming from the documents that I've been indexed.
We classify them.
We take the top end depending on the context of the capabilities.
And we ask for GNAI.
Please answer to the user's query.
Using only these chunks as a context to the prompt.
You can either ask an onsite or a cloud cloud,
trade LLM.
As long as it's compatible with OpenAI API interface.
For instance, Mistral.
Yeah.
To promote some French innovation.
I know he's just a gentleman one.
I think it collapsed.
No gentleman LLM.
You don't know.
There is no gentleman in the room.
So that's fine.
No.
It's not good.
Perfect.
And then we have LLM generates a nice answer.
And data for it takes it generates a user interface.
The way it works, now that we have the Solovic top part.
So why do we do rag?
Well, why not proposing customers to train LLM.
So there are these are the rationales for the enterprise edition.
And this is the rational for the community edition.
So the enterprise edition is because if you train LLM with the copies of the commands,
LLM does not care about permission level systems.
It will generate the statistics of words.
And it will create an answer out of the complete copies.
So you will disclose confidential information to potentially to an employee.
So you can't, you can't do that.
And we flag because we have this first step of doing the vector search.
Based on the documents that Solov manages.
We do filter out the documents that the users are not allowed to see.
And this way, the answer generated is only based on what the user is allowed to see.
But for the customer of the community edition,
it's not easy to manage modifications in an LLM when you train it.
Imagine you have train it with documents.
And now the documents need to be modified.
You cannot just say to be an M4, get this document.
And here is a new version.
Okay, there are many things to do.
It was basically one being to retrain everything,
which is something you may not want to do.
I hope you're amazed by my roadmap compatible with the EU.
So that's a roadmap for 2025.
Q1.
It's a recruitment generation prototype.
As I said, the last PR was done yesterday.
So normally everything is almost ready on data very side.
So it should be complete in time.
Second quarter.
We'll finalize the transformation connectors.
Okay, the transformation connectors are for the many full CF pipelines.
So it will allow you to easily say,
I want to have classification.
I want to have entity extraction.
I want to have translation.
This type of actions when you do Vintexing phase.
Second quarter as well, we provide Victor Search instead of BM25.
So for the results list.
So you will be able to choose either the good old BM25 or Victor Search.
And then in Q3, you will have the hybrid search.
Are you familiar with hybrid search?
Yeah.
Well, it's good.
Okay, good.
But the hybrid search is you do it with two queries.
You do a BM25.
So the base query.
At the same time, you do a Victor Search query.
And you merge the results into just one results set.
And we will be using RRF.
Reciprocare rank.
Fusion.
So it's always hard to say.
Recipro.
Okay.
RRF.
As an algorithm.
And.
Six minutes for questions.
Because you're all eager to ask questions.
You can go on our website for the downloads.
I have already.
So it's Apache license.
Okay.
In Java.
We are not allergic to Java.
If you are a doctor.
Lovers.
You can use the container on the curb.
That's probably the quickest way for you to play with it.
Lots of documentation.
Okay.
We are now.
I think we have 300 pages of the group.
Some of them are maybe.
Deplicated.
But we are not aware of.
So if you end up with issues.
But best is probably to ask us.
You know, you do a GitHub discussion or GitHub issue.
And you say.
Is it?
It's a documentation.
Really.
Okay.
We don't know best.
But it's not easy.
So the code is normally on our GitHub.
But we have a mirror on GitHub.
But all the projects are on GitHub.
And yeah.
The main one is on GitHub.
And the LinkedIn account.
You want to follow our amazing activities.
Because we just recruited the marketing and.
So this day we have lots of LinkedIn contributions.
So if you want to follow.
That's what it was.
Six months.
It should follow.
The.
And I'm done.
Thank you very much.
The trick.
Do we have any question?
Five minutes.
Three.
Two.
Do you have a connector for next.
Or import the documents?
Uh, no.
In a sense that it's it's easy.
I mean, to have one without permission.
It's easy because you just need to.
You create an account.
Next.
I can't with full access to everything.
And you.
You index with mirror.
Or which is local.
Uh, but for doing it with the access control.
So we asked to.
A company specialized.
It would not be expensive to do.
But we don't have the money.
For that.
So.
Yeah.
It's doable.
For release.
10,000 euros.
But not enough.
But yeah.
What would be.
Thank you.
Other questions.
Um.
Is the.
The index only updated.
Uh, through crawling or is there.
Some.
So.
And the passive way or is there some.
Um, active way like if a document changes in one source.
The source can notify the search engine to re-intex it.
I mean, if it's here, uh, initially, it was thought to be both push and pull.
Uh, they have implemented just the interface for the push.
Okay.
So for now, it's put.
So we fetch the data regularly.
We've crawl.
Prior to crawl.
So you can configure.
Uh, but there is no.
It.
And you always crawl.
Everything.
I don't know.
This depends on the API.
So.
Uh, five shares.
For instance, yes.
It grows everything.
Although we have an agent.
Uh, for.
And, uh, as some be shares on windows, but monitors at the low level.
Uh, the OS level, modifications.
We saw those to do real delta in five shares.
But for instance, for excuse.
Uh, which is an amazing witty.
Uh, the API gives us the delta.
So it's really fast.
We don't need to record everything.
Um, do you have a strategy?
How to deal with images and.
And, uh, a lot of them.
Why?
I mean, let me refresh that.
I mean, first, there's seem to be several strategies.
How to index images.
So do we have any plans for that?
And the second thing is you could also consider images in the content.
Uh, for the chat.
So something you plan or you do.
Uh, yeah.
You mean as an input like we use a put to an image in the chat?
No. I mean, if the content for example references an image or if the documentary index
contents in image, you could provide the image to the LM.
I mean, we have.
Yeah.
So that's that's way further than the 2025 or the map.
It's how to handle the multi-modality.
Uh, so we we have looked very briefly at Colpalee, for instance,
for the followers of you for the multi-modality models.
Um, but yeah, the graph, the holy grail of indexing this day is how to be able to really
properly manage multi-modal documents.
Like a document which has a graph, an image, text, formula, etc.
There are ways for now.
But considering where we are, it will take some time.
But yeah, if you have a first pass when you do indexing, where you take everything as an image,
basically, when it's looking at a text, everything is an image.
And you have one LLM dedicated to detecting the types of modalities.
Okay. And he says, okay, here very similar.
And he calls an LLM dedicated to analyzing formulas.
It stores as a metadata, a summary of what it has analyzed.
So like this formula talks about this and that.
This image talks about the cat, etc.
And we store all that.
And then when the user asks a query, the text query.
So it's in the future.
What the user will ask a query.
We will look in this text description of the different modalities.
Which ones would satisfy what the user asks?
Okay.
And then it will call for proper LLM specialized.
So let's say I have a document where I have my financial statements as a table for the last two
years. And the question is, what is the financial revenue expected for the next two years?
Obviously.
The dream is that the LLM, the system will choose the LLM specialized in the graph analysis.
And then do the computation for the forecast because obviously this is where it stands.
So what's the goal?
The pieces are more or less ver, but I heard that Colpally generates so many vectors for one single page.
But it would compute in this row, it was calamity for now.
So it's ongoing work.
But not there yet.
Thank you.
Any other question?
We have time for one more?
No.
Okay.
And thank you very much.
Thank you very much.