Hello everyone, thanks for having me here.
Today we're going to talk about retrieval augmented generation or rack where it works,
where it starts breaking and whether knowledge graphs can help us with that.
A couple of words about myself, my name is Miki Tekimarski, I'm a technical
co-founder of a startup where we do AI agents for autonomous negotiations in procurement.
So I've had a chance to explore different LLM-based architectures, including rack and
these extensions.
Let's imagine a common scenario.
You have a large knowledge base of unstructured documents, for instance, company policies.
You would like to ask questions and get answers based on these documents.
You, as a solution, you configured a rack pipeline on top of that.
First, everything looks great.
You ask simple questions, get valid, valid responses, but then questions become more complex.
Questions that require connecting multiple documents, questions about contradictions, or
about how something evolved over time.
And suddenly, rack can not address that.
This is what leads us to graph rack.
But before talking about limitations, let's first align on what rack means at all.
So I'll call the default implementation venue rack in order to distinguish it from its
extensions.
So the idea is simple.
You have a large unstructured knowledge base.
You want a large language model to reason on top of that, but you can not pass all the
knowledge base just because all the lamps context window is limited.
So you decide to split that text into chunks and then on each question, you retrieve
top-key most similar chunks and pass them only to the large language model to serve as
a context to answer your question.
So for many cases, this approach works, but that's why it became so popular.
But the problems start when questions stop being local.
For example, questions that require combining information from multiple documents.
Questions that depend on relationships between entities, global queries, like what are
the main themes in this data?
Obviously, it requires high-level view on the whole data set, not separate chunks.
Just dealing with trying to spot contradictions, for instance.
You might ask which documents disagree or contradict to each other, or questions related
to temporal reasoning.
Again, chunks of data, you do not capture what time they are relevant at or maybe invalid
at.
This is where knowledge graphs come in.
GraphRack is an umbrella term for all these solutions that try to augment vanilla
rack with knowledge graphs.
So the idea is simple, maybe if we add structure, entities, relationships, maybe cluster,
these entities generate summaries of those clusters.
We can recover the reason in that vanilla rack lacks.
Actually, GraphRack follows the pipeline similar to vanilla rack, but with much more
going on under the hood.
We can roughly split it into three stages, building a knowledge graphs from your unstructured
data, typically text, retrieving relevant subgraphs to be used as a context for answering
your question, and the final question generation using that context.
From that outside, it still looks like ask a question, get an answer.
However, internally, it's much more complex and more expensive, obviously.
By the way, there is an open catalogue of GraphRack patterns.
You may be interested to take a look at it.
Let's consider some of the main open source GraphRack solutions available today.
It's not an exhaustive list.
I have picked some active lamenting solutions, which focus on different query types, Microsoft
GraphRack and lemma index, both cover the whole rack pipeline from the building of the
knowledge graph up to the generation of the answer, while graphity covers only knowledge
graph building and context retrieval, leaving the responsibility of setting up LLM reasoning
on top of that context and generating an answer to you.
I picked two text corpora for testing.
Christmas Carol as a static knowledge base and some toy data set with synthetic events to
serve as a evolving knowledge base.
Let's start with Microsoft GraphRack.
The main promise here is an ability to answer global queries like what are the main topics
in this data.
It works by extracting arbitrary entities and relationships and clustering them into hierarchical
communities, generating a summary of each community.
It creates multiple abstraction layers, like even summaries of summaries, which allows you
which allows the system to get a high level view of the whole data set or separate clusters.
And thus, answer this global queries.
However, it comes with some significant constraints.
First of all, you cannot enforce an ontology here.
In other words, entity and relationship types you would like to use.
It's arbitrary in GraphRack.
In incremental updates are not supported.
So as new data arrives, you would need to reindex and it's very expensive in GraphRack.
We'll see that later.
That also, it's worth mentioning that tracing exact source documents that contributed
to the answer is also difficult here, because the answer is generated out of the summaries
that comprise a lot of different documents or even summaries of summaries, as I said.
In addition to that, it lacks some operational features.
For instance, you cannot configure persistence layer indexes stored as pocket files locally.
So I indexed bold Christmas Carol on both Microsoft GraphRack and Vanilla Rack.
And even on that relatively small data set, GraphRack consumed a lot of inference tokens
for indexing, while obviously Vanilla Rack requires no inference on the embedding the chunks.
And took some significant time and more importantly queries.
So they consume much more tokens as well and take a lot of time as well.
So the local query was who participated in the family dinner and the global query was what's
the central lesson of the story.
And again, if you new data arrives, you need to index and that's, again, indexing costs,
you spend the same tokens.
They use some cash and inside, but some steps, but community summaries should be recomputed
and that spends the most of the tokens and the most of the time.
Also an important part query types and GraphRack are explicit and you're supposed to choose
them manually.
Of course, you can try different query types for your question and compare the results.
Local and global are two original query types.
Drift, they added that later.
I haven't experimented much with that.
It's some expanded version of the local query.
And basic is the Vanilla Rack just for your convenience to compare the results.
I mentioned the provenance track and so tracing the documents, specific documents that contribute
to your to the answer.
I just wanted to show how GraphRack returns that there's a lie returns it embedded in text
answer.
You can see it's indexes of the entities in packet files.
Obviously it requires parsing here, but they're Python API returns it as structured.
Let's move on to the next solution, which is Lama Index.
It's scaffolding library for Rack pipelines.
It includes interfaces and different implementations for data ingestion, indexing, retrieval
and persistence layers, layer and among other index implementations they have
property graph index, which is another graph Rack implementation.
It's highly customizable supports different query approaches and allows you to enforce specific
entity and query types.
Therefore it's not designed specifically for some particular query type, but it's supposed
to be used as a configurable solution.
And it's mainly a Python library.
They have a TypeScript version as well, but it's limited, for instance, property graph index
is absent there.
I did the same comparison with Lama Index, it consumed much less resources than GraphRack,
but still much more than vanilla Rack.
But as I said, it's configurable, you may come up with different results, so the advantage
here is more control, you configure what to extract, how to build relationships, etc.
Another interesting solution is GraphIti.
It focuses on dynamic and temporally aware knowledge graphs.
What does temporally aware mean here?
Peter deals with events, they call it episodes, having an occurring time for each event.
It allows them to use that occurring time to answer temporal questions, and as new data
comes in, instead of overwriting or simply overwriting old facts, it rather invalidates them
with time stamp to preserve that historical accuracy.
That allows you, potentially, to answer your questions, such as what was true at some
point of time in the past, or what changed between that point of time and now.
Testing GraphIti on a static data set like a Christmas Carol would be meaningless to properly
evaluate temporal reasoning, we would need some data set of events that evolve over time,
where facts change, get invalidated, and unfortunately, I didn't have enough time to prepare
a fully representative data set of that kind.
Instead, I just for proof of concept, I created a small data set of events, describing a fictional
employee's career path.
However, even with this simple data set, the default graffiti setup did not produce correct results.
For example, when I asked who is the employee's current manager,
graffiti return to relationship, it considered current, even though, according to the event history,
the employee had already been promoted and reports to a different manager.
However, it's a default setup.
I did not use any ontology, did not enforce specific entity relationship types.
It used some small large language models, so I assume the quality issues stem from that,
and therefore there is room for experiment and improvement.
The main risk here also is performance.
Every new event triggers an LLM-driven knowledge graph of data.
Even on small events from my toy dataset, an average ingestion for one event,
was took 15 seconds, which is a lot, and given the use case, they describe as the main one,
some real time events, it looks like it's feasible only for
slowly changing data.
But if you have some high volume event streams, then it's probably not feasible.
So you would need too much inference resources on that, and latency is crazy.
And query time, it's about exclusion in your reasoning, so it's just about retrieving
the relevant nodes or entities or subgraph.
You can configure that, graffiti allows some search configuration, including real ranking methods.
And that's just what they claim I received on that toy dataset, similar results,
but it may be not representative due to the size of the dataset.
And at this point, and all these questions arises, how do we actually compare these systems?
So cost and latency are easy to compare, but what about
answer quality with different query types on different datasets?
And in terms of how there is still lack of adopted benchmarks,
some benchmarks have recently been released, but haven't gained an adoption yet,
like graph-rack bench, where they do, where they compare different methods with quantitative
metrics on different query types, exactly for, I listed them, or Microsoft benchmark QED,
where they do pair-wise comparison, focusing mostly on local versus global queries.
But the point is that in practice, you would need to set up your own evaluations on your own data,
and queries you care about to understand the actual performance.
There are also other open-source graph-rack solutions we haven't considered today that apply
different techniques. You may be interested to look at it as well.
So what can be conclude? Vanilla Rack is simple cheap and effective in some cases,
until reasoning becomes structural or global. Graph-rack approaches genuinely extend what's possible,
but introduce significant costs, complexity, and operational risk.
There is no universally best solution, and without standardized benchmarks,
adopting graph-rack blindly is dangerous. And the only reliable path today is evaluation on
your own data with your own questions.
Thank you for your attention. If you are experimenting with Rack, it's extensions,
please send me a message. I'll be happy to answer your questions, so participate in the discussion.
Here you go for the question.
What's Rack's solution? The question is, what Rack's solutions do I apply at my company in my project?
So currently, we do not use Rack for our current solution, but as a start-up, we iterated with
different ideas, and one of our products was an anti-corruption copilot, which was intended to spot
corruption risks in governmental bills. And there we applied Lama Index to design a graph with clauses
and connections between these clauses to enable, for instance, you have references to another
clause, and if you deal with chunks only, there is some reference like in clause 5.1,
it's stated that something, and you do not know what is in that clause. And a knowledge graph
allowed us to model that relationships between clauses and sections of the document.
But unfortunately, that project was decommissioned due to finding, like a funding, and now we work
on autonomous negotiations, there is no Rack at least now.
The question is, embedding, say, used, which data type and which dimensionality?
Dimensionality. Okay, it was, it's not open source, it's open AI text embedding small.
And dimensionality is the default one, if I'm not mistaken, 148 dimensions. That one.
And comparison, like you saw the numbers were rather rough, but I did not mean that to make it
representative to some millisecond accuracy, I wanted just to bring some sense of what to expect
at all from these solutions. I tried to use similar large language models, GPT5 Nano.
It's not a commercial element, the quality is another. But yeah, just to make it representative
of course, so that the same encoders used and the same inference speed approximately.
So the question is, how to build the graph effectively? Because I experimented at the
how to build the graph effectively. Because I experimented at the, for example, more chunk
are better than bigger one. But what solution have you used? Microsoft Graph Rack, or you
applied some different? On static knowledge base. Unfortunately, I haven't performed
too much experiments on that. So I wanted to see them rough performance. But I haven't applied
Microsoft Graph Rack in production. I just haven't used a use case for that. But what we
lean towards, I don't know if it's correct, but we rather use long context Windows
to add large models to capture more rather than chunking it to on small chunks. In just
in order to capture more context, when you start chunking, when you do more, come up with smaller chunks,
you kind of lose that context. So the bigger the chunks, the more the accuracy of the extraction.
But again, it's more expensive since you have larger, you should use larger model, which is
pretty much more. The question part of the user, like, I don't know, these terms between embeds or similar.
Graph Rack is a, let's say, it's not configurable. So you pass the unstructured knowledge base,
unstructured document text, text, especially. And it does everything under the hood. So yeah,
could you repeat the question? Sorry. Sorry, because I also implemented a custom
graph without any framework. And I, to cluster the, to make a cluster, the topics I use as a
semantic, maybe you use something similar, maybe not. Not really, but graph Rack, they apply
a latent, latent cluster in the king. So maybe you can cluster not based on semantics, but rather
on distance on the graph. I mean connectivity between nodes. Graph Rack, if I'm not mistaken,
it uses the same approach, so maybe it would be more effective. Yeah, exactly. But,
you get some recent top of clusters. Thank you very much. Thank you.