Thank you.
Welcome.
Today we will be talking about the remarkable features of multi vector embedding balls.
We will try to answer if it is a revolution or just an evolution.
My name is Machin Santos and I'm a UV Corr engineer.
On a daily basis, I'm working on an open source for a vector database called UV8.
I'm Roberto.
I'm a research engineer and I'm part of the applied research team.
I have experience on approximate nearest neighbor search and compression.
We'll start with explaining what are the types of embedding models out there in the market.
Then we will show how those models are being used with vector databases.
We'll end with going into explaining how multi vector encoding algorithm is changing the way we are working with multi vectors.
And our presentation will end with short demo.
So embedding models are the models which have only one task and that task is to create a vector representation of the process input.
For example, when we are sending some sentence into it, we are getting back an embedding vector of a given dimensionality.
All of the modern embedding models are trained with materials, materials, reinforcement learning technique.
It means that during the learning process we are using multiple loss functions.
And that results in a model that is able to output embeddings of varying dimensionalities.
Before that method, we will have to train one model per each dimensionality.
When you are starting your journey with AI, perhaps you are building some rack pipelines.
You are most certainly using open models that are like from open AI, Google or coherent that are returning regular embeddings.
But there is also another family of embedding models.
They are called multi vector embedding models.
The difference is that for a process input, we are getting back an array of vectors.
An array of so-called token level embeddings.
And as you can see here, we are working with a multi vector, not with one vector.
And that also makes it a little bit harder for us to calculate a similarity metric for those embeddings.
Because now we have to take into account all token level embeddings for each of the query and document.
And we have to calculate a similarity metric for each of the pairs.
And then pick the highest results and sum them.
This is then our similarity score, our maximum score.
The formula, which we are using in order to find the similar vector, it is called maximum formula.
And maximum is basis for a late interaction.
Late interaction means that when we are working with multi vectors, we don't have to pull them into one vector.
And then calculate some similarity metric.
But we can just work with those arrays of vectors.
The coolest family of multi vector embedding models are the multi vector vision embedding models.
Those that understand text and images, like called particle quen, called nomic.
This image is taken from the called poly paper.
You can see that there is an example of the standard retrieval when you are working with complex document.
When you want to index that data, you have to first perform some live detection.
You have to find where the images, where the text then extract them separately.
Then of course, you have to pre-process the text, I mean, chunk it.
And then you can calculate some embeddings.
But with multi vector vision embedding models, you can treat a page as one image.
And then send that image of a page over to that model.
Then the model, what it does, it takes the image and divides it into small squares.
And for each square of that image, it calculates a token level embedding.
So then when you are sending, for example, a query, which also gets translated into a multi vector.
And when we calculate those similarity metrics for each of those token level embeddings pairs.
And we project that results in form of a heat map over the image.
You will see that the most relevant and highlighted parts of our query.
And our query was which hour of the day had the highest over electricity.
Generals in 2019, those parts are highlighted.
And this is the way we can visualize how those multi vector vision embedding models are able to see.
What is on the picture?
So when to use those models, they are based usually with visually rich documents with screenshots and presentations.
Now we are about to go over how those models are being used with vector databases.
Thank you, Martin.
So here we have two pictures of two cute dogs.
And what you can see is that we have the vector representation of our image.
And as Martin just said, we can have those vector representation regardless of the modality of our data.
So let's say you also have text or videos.
You can create those embeddings and similar data or semantically similar data will have similar vectors.
So here we have a pretty much learning model which is just an embedding function.
And as you can see here we have several dots which are vectors.
Here we are just in a 3-dimensional space.
And for example, meat and chicken that are quite similar are closing this vector space.
And here you have also apple and banana.
So you have this area with fruit.
So let's suppose you have your own data.
For example, a collection of wines, you will be computing the embedding of your data.
So either it's image or text, you will be computing the embedding.
And here you need a vector DB to sort those data.
Because whenever you have a query, you want to search in an efficient way the most similar vector.
So let's suppose your query is wine for seafood.
You will be computing the embedding of your query.
And then you will use the vector DB to search the top k elements.
So for example here, shardony wine.
There are several types of vector indexes.
For example, we with currently supports HNSW and flat index.
HNSW is a fully in memory index and graph based.
And flat index is just exhaustive search.
So you are computing all the distances.
On top of them you can also have quantization.
Quantization is a technique that allow you to compress the memory for those vector.
Because by default, you are using floating point in 32 bits.
And this can be expensive.
So you can use those quantization like PQ, BQ, SQ or AQ.
They have different trade-off.
So for example, if you are compressing more, you will be losing more information.
But you are saving more memory.
On the other hand, if you will be compress less, you will have still high quality results.
But today, we will focus on HNSW with multivector, specifically with movera.
And before jumping into it, here, there is an example of how HNSW works.
So HNSW stands for hierarchical navigable word graph.
And as you can see here, we have just three layers.
And in each layer, we have nodes.
And whenever we have to perform a search operation, we will start from the topmost layer.
And we will be computing the distances in that layer.
Once we find the closest vector, we will go on the next layer, and so on.
Until we reach layer 0, where we will be in a local region, and we will be looking for the most similar vectors.
So let's jump into movera.
Movera is a research paper that has been released from Google, research group almost three years ago.
And the idea here is to encode those multivector that must be described before into single vector.
That are called FDE, fixed dimensional encoding.
Why we want to do this?
Because if you want to compute similarity with multivector, this can be very expensive.
It's quadratic in the number of vectors you have.
And the idea of movera is that given to FDE, so choose single vector,
you will be when you are computing the dot product between those two vectors.
You will have an approximation of the maximum score.
And dot product can be efficiently computed, because it can be highly optimized using, for example, seemed instruction.
So the main steps of movera are three.
You have the space partitioning one, dimensionality reduction, and repeat of first and second step.
For each of them, you have a parameter, which is kcm.
It's telling you how much you want to divide your space.
And then the number of repetition, which by default, is 10.
So let's suppose you have the following multivector.
You have five vectors, and each of them has dimensionality 128.
If you are using, for example, hnsw without movera, you have to add all those vectors just for one document.
So this can be expensive, because hnsw is fully in memory, so this can bring the cos up.
So the first one is the space partitioning.
And here we are using a technique that is called cmesh, and it's based on locality sensitive ashing.
So in other words, you are based on kcm, which in this case is tree.
You are sampling tree random Gaussian vector.
And whenever you want to find corresponding cluster for a vector, for example, for a vector one,
you will be computing the dot product between vector one and those Gaussian vector, those three Gaussian vector.
Based on the sign of the dot product, so for example, for v1, you have all negative values.
You will encode it in 0 0 0, which is cluster 0.
For example, for vector 5, in this case, you may have that the first two dot product were negative.
The second, the third one was positive.
So the encoding will be 0 0 1, and you add it to cluster 1.
Here, you could also use cmesh, the main difference is that cmesh requires training.
And in this way, you can do partitioning without having any training data, and it's also data obvious.
So in this case, if you have kcm equals tree, you can end up with eight clusters.
So what you will be doing then is to just flatten them.
In case you have more than one vector in a cluster, you will be applying min pooling.
So for example, for cluster 0, you have v1 and v4, so we will be just compute the min for each entry.
Then you have cluster 1 with just one vector, and as you can see, we have some empty clusters.
The next step is the filling the empty cluster.
So for example, for those that are empty, we are substituting those zeros with the closest non empty cluster.
So for example, in this case for cluster 2, the closest non empty is cluster 1.
So we reported here, also the SIM for a cluster chip.
For 5, the closest one was v3.
As you can see, the length of this vector is number of clusters times dim, so 8 times 128.
So what we want to do in the second step, the dimension introduction, we want to shrink those vector.
So we will have the vector coming from the previous operation.
And here we will be using random matrices that are made of plus 1 and minus 1.
In this case, we have that dimensionality is dim times dproj, so 128 times 16.
So that when you will be computing the vector matrix operation, you will end up with a new cluster of smaller dimensionality in this case dproj.
And you will be just concatenating them.
So the new resulting vector will be number of clusters times 16, not anymore 128.
As you can see, we have done some approximation here and in the previous steps.
So in order to reduce the randomness coming from those approximation, we will repeat this process several times.
So which are the prompts of mover and is it worth to use it?
Actually, if you are dealing with a lot of documents, now we will see, for example, in the demo,
only with one document, but we will see how many vectors we will end up.
So let's suppose you are working with hundreds of thousands document.
You need some sort of filtering approach.
So you can use mover with hnsw in memory to create a candidate set.
And then you can record those values.
But the benefit of mover as well, the first one, the improving important times.
You just need to preprocess those vector and store them in hnsw.
Then you have reduced memory requirements, because if you have to build hnsw in this case,
you will have like five nodes and for each of them you have also to find the connection.
In this case, it's just one node and you find a corresponding connection.
Then one consequence is clearly the worst recall, but this can be fixed with risk coding.
So you will be using mover to build a candidate set.
And only for those vector, you will be reading from this original multivector and compute the maximum score.
But now it's time for them.
Yeah, thank you Roberta.
So for today you have prepared a demo.
If you want to run it locally, here is the link to the project.
We will be using V8 to store the multivector embeddings,
and call it to 0.5 call version model.
So let's first load the model.
As you can see here, we have two methods.
One is for vectorizing the image, and the second one we will use to vectorize the text.
All of those methods are outputting a multivector.
We will try to index this document.
You can see that this is like an NVIDIA investor presentation.
It's a very complex document.
We will try to find the image of text and images.
Also we will make some with some charts.
The idea here is to treat this one page as an image.
And one page will be our one chunk.
And then send that image over to the colloquial model to get the multivector embeddings.
And try to find something on those pages.
So the model is loaded.
Now this is a helper method which we will use to generate the embeddings.
Let's start with it.
Let's try to connect with it.
Let's wait if it's running.
It's running.
So first we need to create a collection.
We are creating a collection with one property page number.
And the most important part is here that we are defining a vector index.
A multivector vector index.
And we are enabling here moveira encoding.
It means that when we will be sending multivector's to evit,
those will get translated automatically to FDs using moveira algorithm.
So let's create a collection.
Let's generate the embeddings for our document.
And now you can see here that we have 32 pages.
And for each page, for each page we have generated a multivector.
That consists of 731 total embeddings.
So in total for 32 pages we have over 23,000 vectors.
But we have moveira.
Those 23,000 vectors will get translated only into 42.
So we can see how much of the reduction is on the resources.
So let's save those multivectors into evit.
And now here's another helper method.
I will use that method to generate the query using the call-con model.
So the query will be passed to it.
It will generate a multivector.
And then I will use this multivector in order to perform a vector search.
And I will just present the first result.
So let's look for a list of countries using AI.
You can see that I got a page which is like a roadmap.
And there are those countries that's something that I was looking for.
Let's look what is in VDS infrastructure roadmap.
Maybe there's an information about that of course it is.
So I just wanted to make a point that those multivector embeddings are good.
Also with pages that consist only of text.
And of course I really go to mixed content.
So let's make another request revenue and income charts.
You can see that we got a page that is describing the exactly what we are searching for.
I think that that would be very hard to achieve using regular embedding models
because it's really hard to extract that data.
So let's make two last queries.
First in Italian and the second one in Polish.
And we will be asking first what are the industries that will benefit the most from AI.
You can see that we got a page that describes those industries.
And they ask what are the plans for dividends.
And you can see that in 2025 there will be 300 over 300 million dollars dividends.
So you can see those models are also really good with multilingual data.
So.
Thanks Marcin for your demo.
So we have just seen how we can build an AI powered OCR pipeline and the steps are pretty simple.
The first one is the extraction of the document page using just like converting it to an image.
Then we use a multivector embedding vision model to create those vector.
And then we can store them in a vector db using the mover encoding.
And then you are ready for your semantic search.
So if you like our presentation, you can connect with us.
And also if you are interested in more details about Movera,
we have a blog post and also we have podcast episode with one of the authors of the paper.
Thank you for your attention.
Yeah, actually this is a good question.
So the question was which is our device for the parameters of Movera.
So by default, we are using KC equal 4, the projection equal 16 and the number of repetition time equal 10.
So in this case, the dimensionality of the vector will be 10 times 16 times 2 to the power of 4.
And as you can see, this is a longer vector with respect to, let's say, the regular one which are just 128.
So in this case, I would say it really depends on your data distribution.
So for example, if you have visually rich documents like where you have charts and tables,
you may want to have them to be accurate as much as possible.
So you can increase those parameter.
And actually something that you can also do on top of them.
There is something vector that you get, the FD is uncompressed.
But if you want, you can also use quantization like I will not suggest for BQ, which is shrinking a lot.
The data dimensionality, but most probably RQ or RQ fill wet fit well here,
where you are just going from 4 bytes for entry to 1 byte for entry.
Yes?
Can you question more against using standard embedding and vector search as we'll pass the level of data fraction of the second byte?
Also, this is a good question. So the question is if we have tried any benchmarking in doing the first stage ratio with single vector and in the second one with multi vector, right?
Okay.
So actually we have also tried something like this.
I didn't see any, let's say benefit in doing this.
Personally, I think the main challenge here will be to deal with those chunking technique.
Because personally, what I prefer more about this type of approach is the simplicity,
because if you are just going with a regular approach, you also have to do like these steps in chunking and layout detection.
And in this case, you can just focus on like one multi vector per page.
Okay.
Why are you listening to some sort of communication with this question?
It's trying to sort of approach with this multi-layer.
It's performance.
It's comparable example to the standard layer of building the HTML connector, which is basically having multiple vectors with the same label.
That's one performance way.
In the other way, of course, it's not actually using multi-modal.
Okay. So the question is, like, how this compares to HLSW with just multi vector, and also with single vector, right?
Well, one of them is the multi vector function, which is basically just having one label or a common label or a multi-layer.
So do you mean like a filter at search?
Like each label that needs used for as filter, or
this kind of, you know, that the scarecrowation is going to be referencing your location.
So plus actually allows you to have multiple vectors in the same label.
Okay. So in that case, you are also doing, let's say building a candidate set for those vector,
and then you will be computing the risk coding.
So for example, in this case, maybe you have that two or more nodes of the HLSW are part of the same document, right?
So they are just true vector.
Okay.
Yeah. So for this, yes, actually, we also have support for this, where if you don't pass, like,
de-movering coding, you can use this. And personally, I think this is slightly better quality,
but the main consequence of this is the memory pressure, because for example, in, in this case,
you have seen, like, you have 700 nodes for just one chunk.
So creating the HLSW since it's fully in memory can be expensive from,
if I don't recognize this thing, you can do something like that.
If you want to use memory to take the risk of vectors.
That's true, but you still need memory for the connection.
Because if you either, you are using BQ or using uncompressed, you still, for example, have the M parameter,
which is ruling you how many connections you have for each node.
So that can be still expensive, but yes, we have the support also for this.
You can use regular, for the multi-mortal one.
We have done also some experiments on this.
Personally, I didn't find any big benefits in using those approaches,
compared to just either HLSW with multivector or with, HLSW with multivector and Movera.
More questions?
Yes?
We have a question about the power and depth of questions,
pretty much about the difference between Movera single vector versus both vector.
Does that mean that it's potentially possible to learn a single vector representation
that has to be both vector if they're not using anymore?
Correct me if I'm wrong, the question is, can we use this idea of converting multivector in single vector
and we do it in the embedding model, so we just work with single vector.
Yes, this is something that can be done, and I think it's effective,
because when you are doing the training of this embedding model, you will improve the quality.
So that's something I think will definitely benefit.
I cannot do that, there are documents, screenshots, embedding models that are virtually doing that.
They are creating a one vector, out of the multivators,
and they are just sending you back that one vector,
and it's possible with those types of models.
How does it work with tables?
Because the total of the power and depth of the element should be,
I'm sure that all of this is going to get as much,
like say you have it in table, me, or that they patched it up,
and then you'll compress it in one vector, me, to lose a whole bit of space.
What can I do?
I actually didn't try that, but
if necessary, correctly, your question is, how do we handle those tables,
because we are working with patches, right?
Okay.
Yes.
I mean, what you can do is to have some sort of overlapping with patches.
We did not try, but I'm pretty sure this will help the benefits,
so you have some overlapping between each patch,
so you can, let's say, not lose as much as just doing four,
let's say, straight squares.
It was more, like, how bad is it going to be?
Is there a convertible to vectors and other things?
Oh, actually, something like this, specifically, I did not test it.
Okay, thank you very much.
Thank you very much.
So, we're moving to get out of this door,
more than that one, so that's what we're referring to,
which is what we're going to help.