So, for the next talk, let me introduce Victor Luboslowski with a talk about an end-point elementary
blueprint for security teams.
All right.
Thank you.
Welcome to my talk on end-point telemetry.
Like, I mentioned my name is Victor Luboslowski.
I've been in tech for over 25 years and I'm currently a principal software engineer at
Fleet Device Management.
I'm also the tech lead for our security and compliance product team.
Fleet would make end-point telemetry for corporate security teams and device management
for corporate IT teams.
So, I need to start with a confession, until recently I wasn't a security person.
I'm a software engineer, I like building things and for a long time security felt intimidating.
It's felt complex, hard to test and constantly in the way of getting real work done.
And every time I try to learn it, someone tried to sell me something.
I just want life to be simple.
I want to write code, I want to show features, I want to understand the systems I'm responsible
for without dragging around the pile of security baggage I can't fully grasp.
So when I became the tech lead for a security product team, I realized something revealing.
If security feels like this to me, it's probably feels like this to a lot of engineers.
So the question for me wasn't how to add more security.
It was how to make security something engineers can actually inspect and understand.
One of the core problems in endpoint security today is a mismatch between responsibility and ownership.
Engineering and security teams are responsible for incidents.
They're the ones on call, they're the ones doing post-mortems, but they don't own the telemetry.
They don't control what's collected, they don't control the schema, they don't control retention or replay of how detections are expressed.
They're accountable for outcomes, but dependent often dependent on black boxes for evidence.
Now an open source, we're used to owning our systems and to end if something breaks, we can inspect it.
If we don't like the defaults, we can change them.
If we want to experiment, we can, but in security, we've somehow accepted the opposite.
So when we talk about endpoint telemetry, we're talking about what machines already know about themselves.
Machines such as laptops, servers, point of sale terminals of the store and many others.
Endpoints generate some of the richest signals we have, processes, files, network connections, start a behavior, et cetera.
And yet teams often only see a filtered summary of the activity.
The raw data exists, the endpoints know it, but the engineer's responsible for security don't control it that data.
What's collected, how it's represented, or how long it's kept.
This, that mismatch changes when endpoints are treated as systems we can interrogate, not opaque components controlled by a vendor.
And once you do that, you need the way to manage those questions at scale safely, consistently and transparently.
So let's take a look at what endpoint telemetry looks like when ownership is put back in the hands of the security team that's responsible for it.
If you're responsible for security telemetry, you're really responsible for a system.
And just like any system, it's easier to reason about when responsibilities are clearly separated.
The goal of this blueprint is simple, make each concern independent enough so you can change it without breaking everything else.
Each layer should be swappable without collapsing the system.
So everything in the system starts at the endpoint.
They're not a layer on their own, they're the execution service for everything that follows.
From their endpoint telemetry breaks into four concerns.
The first layer is controlled, this is about intent.
What questions are we asking endpoints and how do we change those questions safely and consistently?
The second layer is ingestion streaming and storage.
Once they deletes the endpoint, this layer is responsible for moving and retaining the data reliably.
The third layer is detection, this is where we decide what matters, turning the raw events into signals.
And the final layer is correlation, correlation, intelligence and response.
This is where humans and automation investigate visualize and act.
Each layer has a responsibility and the goal is that you can change one layer without breaking the rest.
The real value here isn't the layers themselves, it's the boundaries between them.
Each layer has a clear contract, it defines what it consumes, what it produces, and just as importantly what it does not care about.
For example, the ingestion layer doesn't need to know which detections exist, it's job is to move the data reliably, it shouldn't change when detection logic changes.
And the detection layer doesn't care how data was transported, it shouldn't need to know whether the events came from a buffer, a stream or a file.
When those boundaries are respected, each layer can evolve independently.
That's what gives you the flexibility without chaos.
This is a very open source way of building systems.
Clear interfaces, explicit responsibilities, no hidden coupling.
This is how we restore ownership.
When responsibilities are clear and boundaries are explicit, team can own their telemetry and to end without needing to own every implementation detail.
Now let's walk through these layers, one by one, starting at the endpoint agents.
For the endpoint agent, we're using OSquared, OSquared is an open source project that's been around for over a decade and is part of the Linux Foundation.
It runs a lightweight agent on laptops and servers across macOS, Linux and Windows platforms.
At the high level OSquared lets you interrogate endpoints using structured questions.
OSquared makes endpoint data accessible without forcing you to write custom programs.
You don't need different commands, formats or OS specific knowledge to answer basic questions.
System state is expressed through a single SQL interface using concepts like tables and schemas and engineers already know about.
If you can write a basic query, you can start asking meaningful questions about endpoints, that lowers the barrier without dumbing anything down.
What makes this model work is that everything shows up in the same shape.
Data from files, operating system API, system services, and event sources is normalized, because the data is structured and consistent, you can combine it.
That's what turns endpoint telemetry from scattered signals into something you can actually explain to other people.
So this is a real OSquared query.
At a high level, this is asking a simple question.
Show me SSH processes that are listening on a non-standard port.
The key point isn't here, isn't the syntax, it's what it represents.
Process information and network information already normalized so they can be combined directly.
OSquared has around 300 tables in the core project alone.
The cover things like processes, users start to behave your files, network connections, packages, hardware, and system configuration.
And the model is extensible, people create their own tables and expose application specific or custom system data.
That's what makes OSquared a foundation you can build on, not a fixed set of signals.
So in this blueprint, OSquared is an agent that sends its data to the control layer.
It focuses on observation.
OSquared doesn't dictate how data is moved, stored, or analyzed, it just makes high fidelity endpoint data available.
We choose OSquared because it aligns with open source values.
It's transparent, it's inspectable, it schemas are open.
If something doesn't behave the way you expect, you can see why and change how you use it.
That gives teams real ownership over their telemetry.
Now there are other ways to collect data from endpoints without a unified agent model.
Some teams rely on scripts and schedule commands.
Others use lock shippers like fluent bit or file beat.
And at the lowest level platforms expose system specific APIs.
Now all of these approaches can work.
They just shift the complexity onto the team.
OSquared sits above all of this.
It integrates these data sources and exposes a consistent, queryable view of system state across platforms using one interface.
OSquared answers one important question.
How do we observe endpoints in a way engineers can understand and trust?
The next question is just as important.
How do we manage that safely and consistently at scale?
That's where a control layer comes in.
The control layer is about intent.
Once endpoints are observable, the next problem is coordination.
What questions should endpoints answer when should they answer them and how do we change that safely across thousands of machines?
The control layer turns endpoints into high fidelity queryable sensors without treating each endpoint as a snowflake.
The control layer is responsible for three things.
First, central configuration.
Query schedules and settings are defined once and applied consistently.
Second, live interaction.
During an incident, you can ask unique questions and get answers immediately.
And third, consistency.
The same questions, the same schemas across all operating systems.
This is what turns endpoints to memory from an ad hoc collection into a system you can rely on.
So in this blueprint, fleet plays the role of the control plane.
Fleet is primarily MIT license open source.
The repo also maintains a small set of advanced features in a clearly separated directory under a different license.
But everything we cover here is free and MIT license.
Fleet was built to manage OSquery at scale.
It works with standard vanilla OSquery.
Fleet also provides its own agent that wraps OSquery.
That agent handles lifecycle management and its capabilities like additional tables, software installs and running scripts.
On endpoints.
Fleet manages configuration, schedule squaries, and enables safe life queries at scale.
All while people can control separate from ingestion, detection and response.
This is a screenshot of the top level dashboard of the fleet UI.
Without a control layer like this, endpoint telemetry tends to drift.
Different machines run different queries, configurations diverge.
Incident response becomes guesswork.
With the control layer intent is explicit.
You can see what's being asked, the rent points, why's being asked,
change it deliberately, and see the activity audit of who changed what.
And all of this can be managed to get up with configuration stored in version control.
That's critical when security teams are accountable for outcomes.
There are other ways to manage endpoint agents.
Things use platforms like Wahzu, which combine control collection and detection into a single system.
That can work well for small fleets, but it tightly couples concerns.
In this blueprint, we're intentionally keeping control focused on coordination,
so the rest of the system stays swappable.
Evented OSquery tables give you near real-time signals, but only if you don't drop them.
That means the control layer has to work hand-in-hand this reliable ingestion and buffer.
Control defines intent, the next layer makes sure nothing gets lost.
So once endpoints answer questions, that data has to leave the endpoint safely.
In this blueprint endpoints send telemetry to fleet first, our control layer from there, data is forwarded onward.
The job of the ingestion layer is to take that data that's already been collected and move it reliably to downstream systems.
This is where buffering, back pressure and durability are primarily handled.
Not on endpoints, not in detection logic.
This blueprint ingestion is handled by vector.
The control-flare layer fleets, forwards telemetry using web hooks of our writing logs to files.
Vector then picks that data up and takes responsibility for moving it downstream.
Vector is designed to sit at this boundary.
It handles back pressure explicitly, so slow consumers don't cause data loss.
It supports structured transforms, which lets you normalize or enrich data before analysis.
And it's configuration model is friendly to security teams.
Auditable, versionable and predictable underchange.
There are other open-source options for this layer.
Some teams use the previously mentioned fluent bit, which is lightweight and widely deployed.
Others use lockstash, which is powerful and heavier to operate.
These tools can work well, what matters isn't the specific choice.
It's keeping ingestion focused on reliability and structure, not detection and response.
So once ingestion is in place, we've solved the first hard problem, getting endpoint data off and point reliably.
At that point, streaming becomes an option.
Streaming lets you decouple producers from consumers replay historical data and rerun analysis as detection logic evolves.
Not every system needs streaming, so this is an optional step.
But when you do, it should come after ingestion, not instead of it.
This is where streaming fits into the blueprint.
So in this blueprint, streaming is handled by Apache Kafka.
Kafka sits after ingestion.
It provides durable streams that multiple consumers can read independently.
Detections and analytics and experimentation can evolve without changing how data is collected.
Like I mentioned, not every telemetry pipeline needs streaming, streaming makes sense when detections change frequently.
When you want to replay historical data, when multiple consumers need the same events,
or experimentation is expected, not exceptional.
If none of these apply, you may not need this layer yet, and that's fine.
There's one anti pattern I want to mention here, and then it's skipping the ingestion layer.
This is where things often go wrong.
Sending endpoint data directly into Kafka is leaky.
You push back pressure onto the control layer, which propagates it to the endpoint.
You lose control over buffering, schema changes ripple outwards and break consumers.
Kafka is a great streaming system, but it should not be your ingestion layer.
In this blueprint, ingestion absorbs the mess and streaming preserves flexibility.
Once data is reliably ingested and streamed, it has to land somewhere.
The storage layer is where telemetry becomes durable, horrible, and useful over time.
This is where you support investigations after the fact historical analysis and new questions apply to old data.
Storage isn't just about keeping data, it's about being able to ask hard questions later and get answered quickly.
In this blueprint storage layer is built on clickhouse, clickhouse is designed for high volume analytical workloads.
It's fast for the kind of question security teams usually ask, filtering aggregation and slicing large amounts of event data.
Because it's column oriented, you can query large data sets efficiently without scanning everything.
Most importantly, clickhouse gives you direct access to your data.
No proprietary schemas, no hidden query engines.
There are other open source ways to store telemetry.
Some teams use search-based systems like open search or elastic search.
These are great for text search, but can get expensive or slow at scale.
Other store data in object storage and query it using table formats like Apache iceberg.
That works well from batch analytics, but is slower for interactive investigation.
These approaches can all work, they just optimize for different access patterns.
The key is choosing storage that matches how you actually investigate incidents.
So this point with collected the data moved it reliably and stored it in a way that's fast and accessible.
The remaining question is no longer about infrastructure, it's about meaning.
How do we turn all this data into signals that actually that matter?
How do we express detections in a way that's understandable, testable and reusable?
That's where detection comes in.
For detection, this blueprint uses sigma.
Sigma is a rule format, not a detection engine.
It lets you describe suspicious behavior in a portable, readable way independent of storage transport
of vendor.
That portability is important, it means your detection logic isn't locked to one backhand
and it can evolve as the rest of the system changes.
So in this blueprint, sigma is used to offer detection logic.
Those rules are then translated into native SQL and executed directly against clickhouse.
But there's currently a gap in the ecosystem.
There's no official backhand doing that translation to clickhouse for you.
I'll throw there is a SQLite backhand that can be used as a starting point.
So in practice, teams usually build their own tooling here as they grow and mature.
Sigma defines what we're looking for, SQL defines how we execute it.
This is a concrete example of a translation.
On the left is the Sigma rule expressing suspicious behavior.
And on the right is the equivalent clickhouse query that executes it as an alert.
This is where intent becomes execution explicitly and visibly.
You can see exactly how detection is defined and then how it runs.
And because detections run as scheduled SQL queries, there is also deterministic.
The logic is inspectable.
There's no scoring or heuristics or vendor magic.
You can test detections.
You can version them.
You can rerun them against historical data.
That's what makes detection something engineers can reason about.
There are other ways to express detections.
You can write raw SQL directly.
That's powerful, but often less portable and harder to standardize.
You can use stream processing systems like Apache Think or KSQLDB.
Those can work well for real-time use cases, but are more complex to operate.
You can rely on always query only detections, but you lose hindsight and replay.
Or you can use seam style systems like open search with built-in analytics.
Those can work, but often hide logic behind abstractions.
Each option has its trade-offs.
The goal here isn't to pick the best detection system.
It's to make detection logic portable, understandable, testable, and owned by the team that's responsible for outcomes.
Sigma gives you a common language. Clickhouse gives you scale and replay.
Together, they let detections evolve without rewriting the rest of the pipeline.
Detection alone isn't intelligence.
A detection tells you something happened.
Intelligence tells you whether it matters.
Intelligence requires context correlation and humans in the loop.
This layer is where raw signals become something a team can actually act on.
In this blueprint, the correlation and visualization are handled by Grafana.
Grafana works well here because it's SQL native.
It can build dashboards directly on top of Clickhouse.
It fits naturally into GitHub's workflows.
Dashboards alerts and queries live in version control.
And it's already familiar to most infrastructure teams, which lowers the barrier to adoption.
This is a real Grafana dashboard.
It shows processes running on a single device.
But this data can be sliced and aggregated as needed across the whole fleet of devices.
The important part here is not only the dashboard itself.
It's that detections, investigations, and context all live on top of the same data.
You can pivot from detection to historical data to fleet-wide patterns without changing tools.
That's what correlation looks like in practice.
Once something is detected understood, it needs to go somewhere.
Grafana has a loading, a loading built in.
Alerts can be routed to Slack, ticketing systems, or security orchestration, automation, and response tools.
But response isn't just automation.
Sometimes the right response is to pause, investigate, or ask better questions.
This layer connects detection to real-world action,
without hiding the logic or locking you into a workflow.
Grafana isn't the only option for this layer.
Some teams use like tools like Kibana when they're already invested in the elastic ecosystem.
Others use SQL-focused analytics tools like Apache Superset or Metabase for ad hoc investigation.
These can all work.
The thing to focus on is that visualization, correlation, and alerting, sit on top of your data,
not inside a closed detection engine.
As long as this layer stays focused on helping humans understand and act, the system remains flexible.
Now that we walk through each layer, let's put it all together.
This is one concrete way to assemble the blueprint using real open source tools.
Start on the left and follow the data.
OSCARI runs on the endpoint and expose this system state.
Fleet sits above it and handles intent configuration and live interaction.
From there to elementary flows through ingestion and buffering into streaming and storage.
Detection logic is defined separately,
translate it into SQL and execute it directly against the data,
and then correlation, intelligence, and response sit at the point where data and detections come together.
This isn't the only stock that works.
What matters is that every boundary you've seen is real, enforceable, and swappable.
This diagram exists to show that that separation holds all the way down.
Now, whenever we talk about open endpoint telemetry and detection pipelines,
the same question comes up.
Is this meant to replace EDR?
The short answer is no.
And it's important to be explicit about that.
This blueprint does not replace EDR.
EDR tools are valuable, and many teams rely on them every day.
But attackers play a constant cat and mouse game with vendors.
Signatures change, behaviors shift, evasions get discovered and patched.
Even top tier products like CrowdStrike can be evaded.
Not because they're bad, but because the game never stops.
That's just the reality of endpoint security.
This is why visibility diversity matters.
Open endpoint telemetry isn't about beating EDR.
It's about verifying and complimenting what your other tools claim to see.
When you own the telemetry, you can cross check detections, investigate independently,
and ask questions your other tools didn't anticipate.
Open endpoint telemetry gives you a second set of eyes.
One, you control.
That's what defense and depth looks like in practice.
Now, this talk isn't about a specific stack.
It's about a way of thinking.
A way to break endpoint security into clear concerns.
A way to draw boundaries you can reason about.
A way to build systems that evolve without collapsing.
If you take one thing away, let it be this.
You don't need a perfect tool.
You need a system.
You can understand.
You should leave the three things.
A mental model, you can adapt to your environment.
A stack you can run locally inspect and change,
and the confidence to own your telemetry and to end.
Security improves when telemetry is transparent, replayable,
and owned by the team that defends the system.
Now, before I wrap up, I want to pause on this.
These are some of the teams using fleet introduction today.
They're very different organizations with very different constraints.
We've learned a lot from working these customers
and many of the lessons I talked about come from operating at this kind of scale
on the real conditions.
That's it for this talk.
If this blueprint aligns with how you think about endpoint telemetry,
I'd love to hear how you're approaching it in your own systems.
The details vary by the trade-offs are usually similar.
Here are a few links about fleet.
We are hiring.
Here's where you can find me and some of my other content.
This QR code is for the session feedback
for Fossent Dance specifically,
if you guys have time to fill it out.
Thank you, everyone.
All right.
I'll give you a slide if you guys want to talk.
We have time for one little question.
No pressure.
No question.
Okay, thank you again.
Thank you.