Good night, talk, William.
Great.
Hey, everyone.
Great.
Thanks for joining.
Good to see everyone.
I'll be too much yelling at the bar last night.
It's my voice is a little weak, but we'll try to give it a go.
So this talk is building a gented graphical APIs with LLM tool use and knowledge graphs.
You can find the slides, either that QR code or that short link there.
My name's Will.
I work for a company called HyperMode.
We'll talk a little bit about what that is in a second.
But before we dive in, there's a lot of terms in the title of this talk, right?
It ends up being a little buzz wordy.
I didn't really intend that when I wrote it.
But let's maybe level set a little bit and talk some about the pieces here.
So the first one is GraphQL.
How many folks are using GraphQL or have used GraphQL?
Okay.
Maybe that's like half maybe.
So GraphQL is an API query language.
And so we have this strict type system.
We define the types in our API, how they're connected.
This is the graph piece.
And then at query time, the client describes exactly the data they want to bring back.
How to traverse through their application data.
So GraphQL makes this observation that your application data is a graph.
And you get back response that exactly matches the selection set, the pieces of the data graph that you said you wanted to be returned.
I worked on open source GraphQL tooling.
When I worked at a company called Neo4j, when GraphQL first was open source in like 2016.
And that was like a database integration where we were trying to generate a schema from a database.
Generate database queries from a GraphQL request.
And as part of that work, I wrote this book published on by Manning, full-site GraphQL applications.
This talk today is a little different take on how we're using GraphQL at Hyper mode and some of our tooling.
Anyway, so that's the GraphQL piece.
The next piece of this is Knowledge Graphs.
Knowledge Graphs, I think, are having this like, resurgent interest now in the context of AI.
It turns out that if you use a Knowledge Graph to keep track of your data and pass that context to these LOM AI models,
then really improve the results.
So we'll talk a bit about that.
But what is a Knowledge Graph?
There's lots of different definitions of Knowledge Graph out there.
To me, a Knowledge Graph is just an instance of a property graph.
So a property graph is this data model where we have nodes, nodes are the entities, the things.
They have a label like the type, a way to group them.
That's why we sometimes call this the labeled property graph model.
And then we store arbitrary key value pair properties on nodes and relationships.
So here's a very small Knowledge Graph.
So I'm up there somewhere a person named Will employed by Hyper mode.
I'm giving a talk that has a topic, LOMs.
There's another person up there, David Allen.
He have a talk yesterday.
It also had the topic, LOMs.
We both used to work for Neo4j.
So I think one of the important pieces of Knowledge Graph is that there's some canonical representation of the thing.
So for example, we have the topic LOM that's connected to a couple of talks at this conference.
And when we have another talk that's about LOMs, we don't create a duplicate LOM node.
We connect relationships to that one node.
We have a canonical representation of the topic LOM.
I think this is kind of the most important piece of this concept of a Knowledge Graph.
When Google released their Knowledge Graphs, and I want to say like 2012 is part of the Google API.
They talked about this concept of things, not strings.
And I think that's a really good way to reason about Knowledge Graphs.
Okay, so the next buzzword in my title is LOMs.
How many folks are using LOMs to some degree?
Yeah, a lot of folks.
So these are models like Cloud or some of the OpenAI models,
or the Open Source models like DeepSeek, which came out fairly recently,
or LOMA, which is made available by meta.
And the basic way to interact with these, we give the model some prompts and LOMs predicts text.
So they're just predicting, given your prompt and everything I know,
like what are the most likely next strings of text?
And that's what we get back.
But we can also use models in more sophisticated ways.
And there's this concept of agentic workflows for working with LOMs.
Again, there's lots of different definitions of like, what is an agent?
But fundamentally, I think an agent or an agentic workflow is just when your LOM is making a decision
about the control flow of your application.
This is typically done with what's called tool use or function calling,
where you're defining tools or functions that are available for the LOM to decide to call.
And you describe the type of data that you can get back from calling that function or that tools.
And then the data is passed back along with your original prompt,
back to the LOM again, and you say, hey, this prompts with the new data,
are we done yet?
Or do you want to call another tool?
And you sort of iterate on this until you get to the output.
Sort of like the basic building block I think for this concept of LOM agents.
So at HyperMode, this is a company I work for now.
We build open source tooling to enable developers to build what we call model native apps.
We'll talk about what that is in a minute.
Our main open source projects are Modus, which is a serverless API framework.
This is mostly we're going to talk about today.
We also build and maintain D graph, which is a graph database.
Anyone using D graph?
Of course, a couple of folks.
Badger, which is distributed key value store in Restredo,
which is a cache library.
These are all written in Go by the way.
Go is a big piece of what we build with.
And then all these projects are open source, Apache 2, build, deploy whatever you want.
But the HyperMode hosting platform like aims to be the best cloud hosting experience for these things.
That's how we're supporting those projects.
And so today we're going to talk about Modus, which is our serverless API framework that HyperMode builds and maintains.
We released this. I want to say in October.
So like fairly, fairly new in the world.
So I mentioned this term model native app, like what is that?
Well, we think back to the three tier architecture.
This is how I learned to build apps.
This infrastructure still exists today.
We have a client that talks to an application server that talks to a database.
If we want to scale this up, we need to deploy another instance of our application server.
Put a load balancer in front and the sort of thing.
And like all those pieces still exist today.
And apps still get built exactly this way.
But I think it's the abstractions that developers think about that have changed.
So instead of thinking about application servers,
typically we're just thinking about functions and deploying this sort of serverless environment.
And AI models, and we now want to incorporate us like a first class citizen into the applications we're building.
And we're using different types of databases and APIs that sort of built from from scale for the beginning.
So that's really what we mean by model native app, where the model is sort of a first class citizen.
And we're using more modern frameworks in ways to build and deploy our apps.
Developer experience is really important in this world.
So let's take a look at the developer experience for modus.
So we said this is a serverless framework.
The abstraction that we think of for business logic is a function.
So we write a function.
Now modus uses web assembly to target as many languages we can for development currently.
We support assembly script.
Anyone using assembly script?
A simply script is a type script like language for web assembly.
We also support go.
How many go folks are there?
Okay, cool.
A lot of folks.
And we're adding more language as soon as the Python's coming soon.
Other languages coming soon want to add as many as we can.
So you write your function.
And in the modus SDK, you have connections to data and APIs,
abstractions for working with models.
Your functions compiled to web assembly.
We inspect the types that you've defined and the signature of your functions to generate a GraphQL API.
And then when you query that in point from the client.
Modus because we're using web assembly.
Sort of using this execution plan that maps resolvers to functions that we've defined.
Has this sandboxed invocation specific to that invocation of your function.
So web assembly gives us secure sandbox memory space for each invocation,
which is quite interesting.
If we look at the components of modus for the development,
a lot of this is driven by a CLI that exposes the way for skeleton and new app for working with the SDK
and for compiling and running locally in web assembly,
then for production.
We're typically just looking at the run time.
And again, we can deploy this to hyper mode,
but the zombine source.
We can run this anywhere that we can run web assembly modules.
If we dive into the modus run time architecture,
some interesting pieces here are perhaps pointing out that there are host services,
so there are functions that run outside of the web assembly sandbox.
So these are our connections to models, our database connections, things like this.
We mentioned this concept of the web assembly sandbox,
which gives us these nice memory safe secure sandboxes for each invocation.
Our run time is built on top of the was zero, go web assembly run time,
if anyone's using that.
In the GraphQL piece, we leverage a lot of wonder graphs, go GraphQL tooling
to generate the schema again based on the types that you've defined
and the signature of your function.
Cool, so let's take a look at what using this looks like.
Is that big enough?
No.
Cool.
So to get started with modus, we installed the modus CLI from MPM.
I've already done this, so that shouldn't really do anything.
And then we're going to say modus new.
This is going to create a new modus project.
Like I said, currently support go and assembly script.
It's very similar to type script.
So we will choose that, and this is going to skeleton out a new modus project for us.
And pull down the assembly script SDK and just starting off with a very basic
hello world project.
There are a couple of interesting, well, first we're going to see what files we have.
So a couple of interesting things to look at is modus.json.
So this is our app manifest.
This is where we configure the endpoints we want to generate.
And then we would also add connections to any data APIs or databases here,
or models that we would want to invoke in our app.
And we can use local models with hosted by HyperMode in this development environment.
So HyperMode can host models off of hugging phase.
We also host models like DeepSeek and Lama, or we can connect to a third party model provider like
Tropic, company I, who just declare that in this modus.json.
And the other thing file is our index.ts, which just has a very basic function.
Hello world, it takes an optional string and returns a string.
So let's say modus dev.
And is going to compile our project.
Generator GraphQL schema and run that locally.
So we'll switch back to, this is the modus API Explorer.
If you think of this as kind of like graphical if you've used GraphQL before.
And we can see here we have one query field.
Say hello that maps to that function that we defined.
If we run it, we get hello world.
We can pass in our optional argument.
So that's like a very basic hello world example.
That's not super interesting though.
Let's see what are some more interesting things we can do.
Well, going back to this concept of the model native app.
I think there are like some building blocks here that we really want to expose.
In modus for building these model natives at native apps.
And this is where models are first class citizen in the application that we're building.
We're not just building a chatbot.
Like we actually want to leverage our LM to make decisions about control flow in our application.
And so there's kind of three like building blocks or three steps here.
The first is tuning data and LMs together.
So this is where like our internal enterprise data is fetched and passed in the prompt in the context of our LM.
To make that output more relevant.
This idea of function calling or tool use, which we talked about at the beginning, where we're defining functions for the LLM to sort of choose the control flow.
Where to fetch data from what APIs to call to resolve our prompts.
And this idea of knowledge graph rag.
Rag is retrieval augmented generation.
Where we're fetching some data and passing that in the context to our LLM.
That's very similar to that first chaining data and LMs together.
That's basically what they call naive rag.
With knowledge graph rag, we can leverage the power of the knowledge graph,
so the context of relationships in our data to again provide more relevant context for our LLM.
So let's look at this first example, chaining data and LMs together.
So we'll take a look at a basic demo app that I've built.
Let's assume that we're running an online blogging website where you type in,
like the text of your blog and then you hit a button and publish it on the internet.
And something that might be useful.
Some AI back feature that would be really nice to add to that would be,
one, could we generate the SEO meta tags automatically based on the content?
This is non-trivial because there's a certain way you want to construct the meta description tag to optimize for SEO and these sorts of things.
And then also, could we generate suggested titles based not only on the content of the blog,
but also in the style of the blog author.
So that's going to be based on data that we have in the database that backs this online blog application.
And so this is an approach called naive rag or retrieval augmented generation,
where we're fetching some data from our internal database that's relevant for this specific query.
The specific prompt that we're going to send to the LLM and passing that along to improve the results.
So let's stop this guy.
I think this is the right one.
So this is the code called this moduspress.
And there's a couple of functions here.
One is generate SEO.
So you can see the prompt we're passing to the LLM,
your SEO expert creating each file meta tag for this content.
Here's an example of how we want to return it.
And then we paste in here the post content.
And this content comes from a postgres database.
Somewhere down here we have a query just a fetch author from a postgres database that we have running locally.
That has a very small table here that just has one author and one bio.
So let's run this modus dev.
This is going to compile that.
Oh, and we can also take a look at the modus JSON for this project.
And we can see here that the model that we're using is meta's LM model hosted on hyper modes.
This is an open source model provided by hugging face.
I then also connecting to just a local postgres instance.
Okay, so back to our API explorer.
So now we have a couple of new endpoints.
We have generate SEO and post content.
So let's start with SEO.
So I wrote a blog post earlier.
Actually, Claude wrote a blog post for me about eating chocolate in Belgium.
So we'll copy that.
And so again imagine, this is the back end piece, the API piece.
But imagine the front end of our app is here's some UI for writing a blog post.
So writing this blog post about eating chocolate in Belgium.
We want to automatically generate these HTML meta tags that are optimized for SEO.
And again, there's a certain way that you write these where it's like verb first or something.
I don't know how SEO works, but here's an example.
Discover the art of Belgian chocolate making blah, blah, blah.
And what's interesting is these LMs are non-deterministic.
So every time we run this, we get a slightly different response.
Which is kind of fun.
Okay, so we're not like, that's not a rag example.
We're not like passing any relevant information there.
So let's look at the example for generating a title.
And remember this one, we want to be in the style of the author.
So that's me.
Travel category.
Let's say travel.
And then here's the blog post again.
And so what this is doing now is it's going to that post-gres database.
Looking up the author, reading their bio and the prompt says,
You know, in the style of the author based on this bio,
If we had, you know, my full post history, we'd pass those along as well.
And here we're getting selected suggested titles based on our blog content.
And again, non-deterministic, so each time we get a slightly different response.
Which is kind of fun.
Cool.
So that's sort of the basic naive rag approach.
The next one here is function calling or tool use.
And so this is how we want to implement like this basic,
agentic workflow where the LLM is making some decision about the control flow of our app.
And so we define some functions.
You can look at the code for this.
Let's stop this one.
And here we go, zoom in a bit.
So in this example, we have a very basic warehouse that has some products.
So we have things like shirts, shoes, trousers, hats.
And we have two functions.
We have get product info.
So you can pass a specific product that will say,
here's how many shirts I have, here much they cost.
And then we have get product types that will say we have shirts, trousers, shoes, hats.
And we basically define these as tools that are available to the LLM.
And in our LLM called based on the query that we have.
And maybe just an interest of time, we'll think I have this one in the slides.
Yeah, so we can pass a natural language query.
This one says what's available to buy.
And you can see here the response is the available products to buy.
Our shoe hat trousers are in shirt.
And we're also logging like the functions that the LLM chose to invoke.
And in this case, it just invoked get product list to return what are the products that you have.
But if we ask something we're complicated, this kind of runs off.
How many complete outfits can I buy for $1,600, but I don't like hats.
And so now you can see the different calls the LLM makes.
Well, first it does get product info.
shoe hat trousers shirts.
And then now it's getting the price for each one of those.
And it knows enough not to include a hat in my complete outfit.
So I said I didn't like hats.
So it knows a complete outfit is shoe trousers shirt.
And I can buy two of those because they cost $800 all together.
Very fancy clothing shop, I guess.
Not somewhere I would be shopping.
Okay, cool.
So that's the basic idea of function calling and tool use.
This next piece, knowledge graph rag.
So this is where we want to leverage a knowledge graph to improve the retriever piece,
to improve the data that we're fetching to pass to the context of the LLM.
So in the first example,
chaining data and LLM together, the naive rag approach,
we're just doing a select star from author table where the name is my name.
But we can do much more complicated, much more sophisticated retrievers
to find relevant information.
So for example, we could use embeddings.
If we're dealing with lots of unstructured data,
we can chunk up a bunch of text, calculate embeddings, do vector search.
We can also traverse the knowledge graph to find more relevant pieces of data.
And this is going to depend on what the domain is.
But typically, if we're dealing with unstructured data and constructing a knowledge graph from that,
we'll have our domain graph, which, in this case,
we're going to build a movie recommendation app.
So our domain graph, in this case, we have movies,
good users that have rated a movie, so this is like a mashup of the movie lens,
data sets, and IMDB movie lens was used for,
I think, like, the Netflix prize for finding movie recommendations.
But anyway, we have a domain graph, but then sometimes we also build,
like, a lexical graph, where we've chunked up unstructured data,
and that we're calculating embeddings based on that,
and we can traverse from our lexical, into our domain graph, and sort of through there.
There's this graphrag.com site.
It has a lot of information on the different patterns,
different retrievers that are used for graphrag.
But if a couple minutes left, let's take a look at this example.
So we're going to use embeddings, embeddings is basically just taking text,
and converting that to a vector of numbers,
and then we can do what's called vector search,
vector similarity search, to find embedding values that are close together in vector space.
So most databases support vector search now.
Let's see if we have this one running, maybe.
So we take a look at the code for this one.
So we're connecting to an ear for J instance.
We've already calculated these embeddings using the minilem model,
which is, again, hosted by head of the code.
So we're connecting to an ear for J instance.
We've already calculated these embeddings using the minilem model,
which is, again, hosted by head of the code, provided by hugging face.
And then at query time, we're basically just doing vector search to find similar movies.
So let's take a look at the app here.
Is this running?
So we search for a movie by title, size search for Cosmos,
and we're looking that up in the database,
and then doing vector search to find similar movies based on the embeddings.
And the embedding of the, in this case, the plot.
So that's the thing that we embedded in the database.
So the code for this, if you're interested, is available in this modus recipes.
GitHub repo.
This one is specific for the movies kit.
All the examples I showed today are in this modus recipes repo.
This just shows different ways to use modus for building cool apps like that.
And that is all I have, getting grab the slides here,
and I think we are out of time as well.
So thanks everyone, and we'll see you around.
Thank you.