I got the start signal everybody, hey, wonderful, let me go ahead and turn on my thing so it actually works.
This is me, and if you don't know who I am, you can look at my hat because my hat says the yoc.
That's me, I work at EDB, I'll just move on quick.
Now, for the recording, because this is going to go on the internet and we all know that
all of the AI is trained on the internet, I welcome our AI overlords.
That's going to go into the record, okay?
And so, I want to start with this premise, right?
A lot of people treat their AI like a user interface to a database, right?
I've talked to a lot of folks, and maybe you all out there have seen this as well, where you're like,
I've pumped data in, I built this application, it's not returning what I want.
What you want it to do is you want to say something like, tell me the products at this person bought.
And it doesn't tell me the right products, or it hallucinates a little bit.
And that happens quite a bit.
Who remembers, hey, oh, a few people.
So Microsoft had this great idea, it was quite a while ago, and it was like the first generation AI.
And what they decided to do was train-tay on Twitter.
What could possibly go wrong, right?
Nothing, nothing could go wrong with that, right?
But the training was an early bottleneck for folks because they learned that, you know, hey, you know,
as you get more data, things to start to evolve.
But how LLMs are built is they're not built to be deterministic.
They're built based on to answer questions based on the training data that they've got.
So if the training data is bad, your responses are going to be bad.
So let's talk about how databases play into this.
Okay, so how many people here have heard of rag?
Oh, my God, I don't even need to explain this then, okay, cool.
So traditional models are obviously really expensive to train.
Has anybody trained their own models like two people?
Does anybody, like, not trained their models because it costs like a bazillion dollars,
and it's going to, you know, like, cause all kinds of issues, like, you know,
you could power, like, you know, an entire city, just to train one model, you know?
So I mean, there's quite a bit.
So a lot of people have turned to rag apps.
Anybody actually building rag apps right now?
Hey, if you, okay.
So a lot of folks have turned to rag as a way to pull the data that they need out of their database,
and then augment the LLMs.
So since you know what it is, I'll blow through these really quick,
because we only have 20 minutes.
But how these are built when you take data and you use, you know, vector searches,
you're actually building this around the idea of finding similar concepts in the database.
Okay, so when you have a vector database, it has embeddings,
and the whole premise here is you want to find things that are similar.
So I have a movie review database that I use a data set sometimes.
I can tell you where it is if you want to hit me up afterwards, it's on GitHub.
But things like, hey, why do people in Texas dislike Star Wars?
If that's your prompt, when you go to the database and you do a semantic search to find similar,
you know, messages, it's going to look for similar terms, right?
So Detroit is not in Texas, so it doesn't really match Dallas isn't Texas,
so it's kind of close, you know, Star Wars and Star Trek and science fiction,
all are in the realm of that, you know, dislike and hate,
so all of those things will feed into the semantic search.
And the semantic search is really built off of embeddings.
Anybody here run their database or a data through and create it a whole bunch of embeddings?
Just to give you a view of you.
Okay, so embeddings are basically a mathematical representation of your data.
Okay, and the models that you use to train these on, you know,
have multiple dimensions, and the combination of the mathematical equation or the vectors
will determine like how similar the words are or how similar the intent is.
Now, there are a good bazillion embedding models.
Okay, and this is a really critical thing.
Every embedding model that's out there, and this is an old list, by the way,
these come out like every day, but every embedding model has nuances.
Okay, so when you take your data, your, you know, your PDF,
your data, your database, and you create embeddings,
it turns it into that mathematical model.
But some of these models actually have pretty significant limits on what they can accept in.
So, Bert here, for instance, right?
So if I go up to, up to here, and I do Bert, oops, crap.
If I go up to Bert, right, so Bert here, it has 512 tokens.
So any size that's bigger than 512 tokens won't go into the system.
What it does is it truncates it for you, hopefully, without any air message.
So if you have something that's this giant document that you want to use as part of your AI application,
it'll just truncate the data and only do the first 512 tokens.
So the rest of it just disappears.
No air.
Other ones will give you an air because they're helpful.
Why this matters is every one of these models is trained on different data.
So it will give you different results.
And they have different nuances.
So some will, you know, return, you know, an air, others won't.
Others, you know, are more graceful about, you know,
going over the limits.
So you just have to be mindful of that.
Now, this is why LLMs and vector databases are very different purposes, right.
So when you're talking about, you know, the things that are vector database,
you're talking about those embeddings that I showed you,
those are really things that are being pulled from long-term memory,
whereas the brain of the LLM, if you will, you know,
that's really designed to, you know, be that generative AI component
and you're augmenting it with what's in that memory.
Now, I mentioned that 512 kilobite or 12, 512 token limit on some of these.
Okay.
There's this process called chunking.
How many people have done chunking before?
Okay, a few people.
How many people have done data warehouse like ETL pipelines?
Okay.
So think of it like this.
Chunking is basically an ETL pipeline to break up the text of, you know,
the documents that you want as part of your AI stack,
into smaller chunks that fit under the token limit.
That's pretty much it.
Now, the thing is, there's actually a science to how these actually work.
So you can chunk in different ways, right?
So if you only got 512 tokens, which, you know, isn't very big,
you could chunk on, let's say, the sentences,
or you could chunk on paragraphs, or you could chunk on something else.
Now, when you look at like this text here,
and if we were going to chunk this up and send it into our pipeline,
imagine if you were to do the sentences, right?
So if I'm talking about Star Wars reviews here, the pacing, however, was an issue.
If that's all it knows, it doesn't know it's from Star Wars,
it doesn't know it's from a movie, it doesn't know anything about that.
So when you look for a similar semantic search,
and if you're just looking for that sentence,
you're not going to get what you want.
Now, if you come down and you do it by paragraph,
you'll get a little bit more, but you'll notice that even in the paragraphs,
not all of the information you want is in there.
So you're going to see that that becomes a consistent problem.
Okay, before we, let's see.
And this is why like the most important person in the organization
is the data engineer for AI applications.
So you really have to understand the data itself,
and what's going into the data, how you're processing it,
how you're setting it up, how it's going to help you achieve what you want.
Okay, so it's really about that understanding.
I'm getting a lot of feedback here, human, right?
So, okay, so before I go there,
what I want to do is I want to show you a quick demo demo.
And so let me go ahead and throw this up.
So, I don't this application.
All right?
And you can see on here what I've done is I've chunked five different ways.
So I have a summary, which is going to take the blocks of text
and summarize it into something that fits within my chunks.
So I send it through an LM to pre-process.
I have it look at semantics, so it tries to say,
does the next sentence fit with this sentence?
I have fixed character, which is probably what most of you would do,
or have done in the past, which is just like after 300 characters cut it off.
Okay?
And then I have sentence based, and then I have a paragraph based.
Now, notice something kind of odd, right?
Why is the paragraph chunks more than the sentence chunks?
Any ideas?
No?
So, in this case, we're doing medical records,
and doctors who do medical records, they'll do like a line for something,
then they'll do two, you know, like enters,
and then write the next line, then two enters.
So it has a ton of extra, quote unquote, paragraphs
that wouldn't necessarily exist in, you know, just based on the sentence.
And since the sentence is looking at, you know, the, you know, punctuation,
it doesn't hit.
So, what I can do here is I'm going to come over here,
and we're going to take a look at, for instance,
hey, if I've got a headache, what should I do to treat my headache?
All right, so this is going to run several of these models
at the same time.
This is all the exact same data.
The only thing that I've changed here is how I break up the data.
Okay, so you'll notice that the responses as they come back vary.
All right, so the first one here is going to show that as a headache,
you know, what is it?
I'll try non-drug-mad measures, such as deep breathing.
Okay, so if you've got a headache,
really I want to tell me that I need to take two aspirin,
but they're not going to do that.
Oh, look at this one.
If you have a recent bone marrow transplant, how did that come up?
I asked for, you know, like headaches,
and it's flagging like, you know,
something with a bone marrow transplant that's kind of crazy.
And when you look at the individual data that's being returned from these,
I flagged anything that shouldn't fit that,
and so you can see that it's actually matching this as similar
because it's saying something like, you know,
in the post-transplant setting, the headaches, you know, include.
So this person had a transplant, and one of the things afterwards was a headache.
So they're like, oh, well that's a headache,
because it's similar enough to flag it.
So you need to make sure that as you chunk things through,
you get the full context.
If you read that full note, for instance,
you would see that the full note talks about,
he had a stem cell transplant, right?
So this has nothing to do with my headache,
but it flagged in anyways.
It's horrible thing to see.
Now it's similar when you go over to the, you know,
embeddings.
So each one of these different embeddings models,
okay, is going to have different items.
Oh, I forgot to create these embeddings, and one, that's okay.
But these are the models that are generating those mathematical equations that I showed you.
Okay, very similar, okay, where you'll see things that come out
and say very different things for each one.
So this is what is going to ruin most of your applications
if you don't get this right.
Because I've heard from a lot of folks that I've talked to.
They're like, man, Jenny, I is a problem because of accuracy.
You know, I want to get, you know, like,
I talked to insurance company, and they're like,
I want to get, like, my claims for this person.
I sent it through to Jenny, and it doesn't give me the right claims.
Right?
All right.
Let's go back to slides.
We've got seven minutes that's power through.
All right.
So I'm going to blow through this one because this one's a little redundant.
So let's take a look at how this works outside of just those embeddings in those chunkings.
Okay?
So let's go back to that.
Why do people hate Star Wars in Texas?
If I use just the semantic search that's part of, you know,
whether it's Postgres or Elastic or whatever,
what you're going to end up with is you're going to end up with something that looks kind of right when you look on it on the outside looking in.
But when you look at the data that's returned,
and that it's using to make this, take a look at the data.
Remember, I'm asking why do people in Texas hate the original Star Wars?
Well, the first thing you'll notice, there are no original Star Wars reviews.
Right?
There's none.
And when you look at the states that they're from, there's none from Texas.
So this completely crapped the bed, right?
Like this didn't give me anything that I asked for.
And this is why people get frustrated.
Okay?
Now there's three ways, three approaches to querying to fix this.
Okay?
Vector alone is what most people start with and they don't realize they have to do more.
They forget that they're using a database.
So they forget that they can actually add a wear clause to some of these.
Right?
So adding a wear clause where you can do predicate filtering is incredibly important.
But there's also a full text search.
So you heard earlier there's a couple people mentioned BM25.
There's other full text indexes available for different databases.
But there's a difference between full text and the predicate.
Right?
A full text search.
Hey, if it says, you know, people in Texas hate Star Wars, it will look for every one of those keywords.
And if they don't appear in the review, then that's going to be a problem for them.
So this is where you actually have to add the traditional filter.
Right?
So adding in like where the databases that are where the movie is is.
Well, that's going to get you the original Star Wars.
But it's still not reviewing Texas.
So then you add Texas.
Okay?
And then you add, hey, where the ratings are horrible.
And you keep on going through that process until you get the results that you want.
And then you get it.
But not only is this like more accurate, it's also faster.
Okay?
Because what you're doing is you're filtering down the number of records you have to go through
in order to do that semantic search.
So you don't have to search through a whole bunch of crap in order to find what you're looking for.
So it really cuts it down.
Now, some people, I'm going to say this, not everybody.
But some people, when they see like that type of stuff, the first inclination is,
well, the data's in JSON anyways.
So I'm just going to throw JSON into a field and create my embeddings on that.
Don't do that.
Okay?
Embeddings are not JSON aware in most cases.
What that means is all of your keys, all of your attributes, will be treated as key words as part of the semantic search.
Okay?
So all of a sudden that's going to skew your results like mad.
Okay?
So don't do that.
Now, there's a lot of advanced topics.
And I generally do this as like a couple of hour talk actually.
So I won't get too deep into these.
But there are things you can do.
So a lot of you might not have data that already has like,
predicates that can filter where clauses that you can filter on.
So for those, you might want to actually extract metadata from text.
So one of the great things about LLMs is, if you get to give it a list and say,
can you classify the data and fit it into one of these ten buckets?
It can pick those ten buckets.
And then you can store those as predicate potential filters.
Right?
So that's one thing that's really good.
The other thing is you can actually go through a chunking process
where you chunk multiple different ways.
So you can have one that is chunked on,
let's say sentences, one that's chunked on paragraphs,
and then combine the results.
But I'm going to tell you that is going to take additional time and resources.
So it's always a trade-off.
How much accuracy versus performance do you want?
You can also summarize chunks.
So if you have a really big document and there's a lot of garbage in it,
just send it through an LLM ahead of time, prepare it.
Store the summary.
I've done that.
I've got good results as well.
Because then you can just say, what is this thing?
And then you can go in and do predicate filter and
or full text search later on.
So there's lots of other fun topics here.
I won't go through them.
But you know, you can hit me up if you want.
But these are all things that I have seen come up, right?
Whether it's guard rails on combined responses,
so you can say, like, make sure you filter out anything that's,
you know, PII or, you know, gunky, you know,
sending chunks versus full docs to the LLM to summarize,
deduplication, right?
Pre-post filtering, all kinds of different topics here.
So wrapping this up, AI is the interface to your data
and it can derive a lot of new insights for it.
But you have to be very careful, right?
So remember, vector search is similar.
It's not equal.
You know, you know, predicates beat key,
weird filtering or for vectors or, you know,
they beat, you know, semantic search or even full text.
You know, you're in Betty models, have limits.
You know, and there's always going to be trade-offs.
So happy to answer questions, we're just that time.
So, yeah, that's it.
Very fast.
Thank you.
So, come on down if you want to answer questions,
because I think they're going to kick us out.
Very kick us out, aren't they?
Soon.
Yeah, we can go until.
Yeah.
Can you repeat the question?
Sorry, was that?
Yeah.
Oh, yes.
Okay.
So when you do the embedding.
Yeah.
So when you do embeddings and you just draw JSON into the text field,
you know, then it's going to create the embeddings on that.
Just like it's a doc, okay.
So you can do prefiltering and pull out the attributes.
You can also run it through an LM ahead of time.
But a lot of times, it's best just to extract the data you need
and just create the embeddings on that as opposed to doing the full JSON.
There are a couple models that claim to be JSON aware,
but I don't think they are.
Because remember that chunking, right?
So if you have a 512 token limit and if your JSON's like massive,
it's going to break that JSON up so you're going to lose all the format anyways.
Right? And so then you're going to lose all the continuity,
and you're not going to know what the hell it was.
No, yeah.
Sorry, I didn't catch all that.
Let me come up there and here and then I'll repeat the question,
I guess.
All right, what does that mean?
If you have multiple files, like not one file, so we embed them.
But we have like 10, 5, oh, 20, for example.
They are somehow connected.
How can we embed it?
I mean, should we just kind of get in anything,
but it will not work, I suppose.
No, so what you might want to do is you could do this is more of like a graphing
where you could have like some sort of connection at the database to connect them.
You could also summarize them all together to get a smaller version,
but then do like a double chunk where you could chunk based on paragraphs and then combine them.
But ultimately, typically what you do is,
since you're breaking this up, you just stored in the same vector table.
And so if it's stored in there, it can search for all of it.
It's just going to search if it chunks individually.
So just my return with all the docs or parts of the doc.
So that's how you do that.
Okay, and is it possible that you mentioned the graph as I'm just like that?
How can we connect the parts of the graph as I'm just like that?
Is there a graph of database?
Yes, so there are several graph databases that can do that as well.
But you can do it in any relational database as well.
So you could just say like in your table structure, you could say like,
hey, this node is related to this node,
and they're all part of the same group,
and they'll always, you know, search them together.
Thank you.
Yeah.
All right.
Thank you.