Hello.
Morning.
Welcome.
Stop moving tables.
My name is Brian Bohrum.
I'm a Prometheus maintainer.
I've worked on the code for about seven years.
My day job is at Grafana Labs.
Hi.
I'm young.
I guess I can call myself a team member now too.
The PR is still open, so go vote for it.
Sorry.
Do I need to repeat?
All right.
My day work I work for Red Hat on the OpenShift Monitoring
and the rest of the time I try to contribute to Prometheus.
All right.
So who's here? Who's hard of Prometheus?
Okay, that's good.
Who uses Prometheus?
Everybody nearly?
Who uses Prometheus three?
I don't like maybe five percent.
Is it okay?
Is it income down to report your bugs?
Is anyone like totally new to Prometheus in the room?
Nobody's opening up to that.
So we don't need to go through the architecture.
So that's good.
That's going to save some time.
I did want to put this.
The QR code takes you to the documentary.
Produced, which is really nicely like produced and edited
and so on a really high quality video.
And it has the history of Prometheus.
So I recommend that if you've never seen it.
Yeah, I mean, the point of the architecture.
Prometheus is a one process solution.
And there's a lot of other things out there.
And someone was just saying a few minutes ago,
you could add Kafka and stuff like that.
So the big thing about Prometheus is you just run it.
One thing it runs.
It does your monitoring.
And you're alerting.
So that's the kind of the key part of the architecture to stress.
Right.
And so not everybody has to go out and watch that documentary.
We have a little bit of history here.
So Prometheus started that soundcloud.
It's over 12 years ago now.
As you can see, after the initial commit,
there was around the 2.0 release.
And when I say around, like I'm talking years, right?
It joined the CNCF, but also graduated the CNCF as the second project ever in each categories.
And after that, there was a lot of focus on stability.
There's non-purpose support versions now.
And of course, now we have with 3.0 since November last year.
Yeah.
I think that's it for a quick one.
Right. You release.
Obviously, there's a lot of stuff in there.
You know, you have the cornerstones there.
Obviously, we won't ever be able to talk about all of those in a single talk.
Like, some of these are a single talk.
In fact, there are talks out there already.
This just takes you to a YouTube query that has the current committed three talks.
This will update over time, of course, which is nice.
So today, we'll pick out a few of these and we actually look at some of what we consider breaking changes.
Of course, with a major release, we can finally release some breaking changes, which is nice after seven years.
If you're interested in how to migrate, we have a very nice migration guide there.
That should help you out with everything, and if not, let us know.
And I guess with that, we can start on the actual talk.
Right.
So we're going to go through a few of the changes.
So there was a change to reggaex handling, which is certainly one of the most memorable ones for me.
So, yeah, what does that do?
Everyone uses Prometheus, so you should all know what this does.
So I'll give you multiple choice.
So it actually doesn't do anything.
It's choice A. Who thinks that?
There's a trick question.
Who thinks it goes through every possible value carefully checking each one?
I'll say no, nobody thinks there's no option C.
So in prior to 3.0, it was option B.
So it's completely obvious that it shouldn't do any work, because dot star matches everything.
But, surprise, the goal code that was used to implement reggaex, the dot does not match a new line.
So every time you run one of these queries, it would scan every possible value to see if it had any new lines in it.
And you would not see those.
But, nobody has, like Hansa, who's got new lines in their labels.
Right. So this was just a stupid mistake.
But, for me, Thias is, as culturally, is passionate about breaking changes, about compatibility.
So we did not fix this bug until a major version.
So that's, for me, that's the most memorable thing about 3.0.
Once you decide to do a major version, we didn't do a major version just to fix this bug.
But once you have decided, and we kind of decided it's about time, like seven years.
That seems like long enough for a major version.
Once you've decided you can break things, well, the floodgate's open.
So we certainly fixed this one, and I was very happy about that.
And those kind of queries, if you have a lot of values, they go faster.
So, it's good.
Okay. So there were a few breaking changes in 3.0.
Luckily, they all fit on one slide.
I could read it.
We're not going to go through them.
But, yeah, just to give you some idea.
So that's the release notes.
That's a list of everything that was changed.
Some of them are sort of fixing stupid things.
And other ones were things that you could already make it work better with a feature flag.
But you had to know that.
You had to turn on the feature flag.
That was zero.
We basically turned on all the feature flags that you really should have been using.
But we did not turn on for compatibility reasons.
So yeah, that's the breaking change story.
Let's move on to elementary.
We had a talk earlier.
Some of you might have been in about elementary.
It's the big thing.
The big standard for instrumentation.
As of 3.0,
Prometheus by default will accept metrics in the open telemetry line protocol,
OTOP format.
So one of the big things about that is that the metrics are being pushed to Prometheus.
It's always been a pool model.
It sits there, pulling metrics in from wherever they come from.
And OTOP is a push model.
Because I don't know.
I'm not going to be rude about OTOP telemetry.
Only got three minutes.
Yeah.
That's going to take an hour out of the talk.
So anyway, it is a push model.
So Prometheus now accepts metrics being pushed to it in OTOP telemetry.
So that's nice.
One more for me.
Yeah.
So we decided to support all characters.
That's why we refer to this as UTF8.
The one character that matters to telemetry is the full stop, the period.
The level names in open telemetry always have a dot in them.
And Prometheus has always banned the use of the dot.
And we thought, well, if we're going to allow one more character, why not let's just allow them all.
So you can now write Chinese and your labels, you can write accented characters.
Isn't the level names where they were previously restricted?
But we do make you quote anything that is outside of the previous allowed set.
So just put it in quotes like that example.
That's in the migration guide and there's a whole special page on what do we mean by UTF8 support.
So again, we throw it in the floodgates.
You can use any character.
You can name your metrics after emoji.
I did submit a talk to coupon called status equals poop.
But it was not accepted.
It's a shame.
Oh, I'm doing one more.
Why am I doing so much talking?
I don't know.
I think I do.
You do this one?
Yeah, okay.
So we're looking at another breaking change that was interesting.
So we changed range selectors or how they behave.
In Prometheus 3, you have a left open interval.
Sorry, left close.
Can't confuse.
So if you have a sample that coincides with the left border of your interval, you get that sample.
That's can be surprising in a lot of ways.
So we wanted to change it, you know, make it left open.
So that sample will not be included.
And, you know, we thought it's a breaking change, but what can happen, right?
The feature actually causes impact mostly academic.
Oh, that's nice.
So you have to get your time stamps really accurately allowing you to notice a difference.
In the migration guide, we're sort of a little bit more insightful.
We explain this a little more, you know, we do this.
This might impact for some subcrew use cases.
Some front ends that do certain things.
But really mostly tests will be impacted, right?
Easy peasy.
Turns out, oh, maybe we should show a query like this.
So this is an example of such a subquery.
Right?
If you don't know subquery, this is my pointer.
This down here makes it a subquery, so it takes to you request total.
Looks back for one minute and gives you a range vector with one minute steps.
So you can see already, I forced this a lining here, right?
We have a one minute interval, but we only have one minute steps.
So the second sample would fall right on this one minute, a quarter.
And it turns out that these queries are actually more common than we thought they would be.
So some people have problems now.
But I want to use this query now to actually look at the new UI.
Because the new UI, besides looking much nicer, brings a really cool new feature.
And that's the tree view and explain.
So we have this query here, as I said, right?
I wasn't lying, this doesn't return anything.
And Prometheus 2, this would actually give you a graph.
But we can fix it, and Prometheus 3.
We add one millisecond to the range query, and cut out.
Nice, right?
But if we turn on the tree view now, we can actually look at what's happening.
So the tree view gives you all the components of this query.
And you can actually select the individual components and get the results for that.
So I selected the subquery here, and you can see, oh, we only get a single value at a time.
This is the old query, right? That now breaks.
If we add that millisecond, it's two samples and the rate suddenly works.
So that's really neat.
I think that's the coolest feature of the new UI.
I'm not a UI, I guess you can tell.
There's more cool stuff, right?
This actual explain tab also gives you a really nice explanations when your query doesn't work.
So here, you know, that's a different query, but basically we try to match something that doesn't match.
We don't get any results.
In the old world, you know, you had to dissect your query and figure out, oh, where does this go wrong?
The six-lane feature now tells you, oh, there's a label set that don't match.
Fantastic.
We should give a shout out to Julius Moot, who single handedly implemented all this.
And his company Prom Labs.
So if you want to, if you want to buy any previous training or consultancy,
reach out to him.
Well, he'll show up in one more slide or call it out.
Right.
The last one I wanted to show is this dreaded duplicate match group error, right?
Who's here, have that ever?
Not too many people, surprising.
If you ever run into this, this can be a bit tricky to debug, but now with explain.
It'll tell you, because it has access to your data.
It'll tell you actually the match group and where it has multiple samples,
and you can go out and fix it.
So how neat is that?
That's for the new UI, or back over to you.
What?
Went wrong.
This is going to be exciting, because I can't remember what's on the next slide.
What did go wrong?
Nothing.
Nothing, right?
Well, no.
So measuring what went wrong by what did we have to put out an emergency patch release for?
Well, not match, actually.
There were only three bug fixes in 3.0.
About one.
Nobody, nobody deploys 3.0.0.0, right?
Always wait for the patch release.
So the first one was actually introducing the thing that people, the most people had problems with,
which is the left open for sub queries.
But we did want to make it consistent, and we didn't know it was going to.
I personally had never seen one of those one-minute colon one-minute queries,
until the bug reports came in.
Why would you do that?
Yeah, so there was a memory leak, and we had forgotten to support
UTFA on one endpoint.
But that was it.
That was 3.0.0.1.
Right, so no back to yourself.
Yeah.
So with Prometheus 3, there was other stuff going on.
Basically, somebody highlighted that,
oh, somehow contributions are dropping.
It's not as active.
Let's change that.
So we want to make the Prometheus community more welcoming to members or more contributors.
And as part of that, there is actually a new governance document, I guess, we call it.
In the works, that's not ready yet.
But look out for that, and now's the time to join Prometheus.
At Prometheus, I showed a slide how sort of the contributions developed running up to 3.0.
And now we can actually look at 3.0.
So this is the contributions by contributor for any given release.
We can see the release numbers there at the bottom.
And I mean, looks pretty good, right?
3.0 was definitely a success in gaining new contributors and more contributions.
This next graph there, this next not so bumpy one is Christmas New Year's.
Everybody was off.
And yeah, I mean, one contributor here sticks out a bit, right?
But even without this person, this would still be quite a significant increase.
And this is actually the UI addition here.
So that was merged as one big lump, so that's joyous as work on the new UI.
I said he would show up again.
The same thing we can see, it's not just code, right?
Not just a new contributions, but reviews are really important.
So this is from the death stats, CNCF death stats.
You can look at review contributions.
Okay?
Okay, I got to do this one handed and in the wrong direction.
Give me a sec.
I take the mic.
Let's go over here.
Okay.
Back to it.
Oh, and the script is tiny if you want to make that bigger.
Very important to a project, right?
If you don't want to contribute code, but you know, say, you know, an integration that we have from
this course is for services coverage, so you can review stuff.
We like outside input and pointing things that could be done better.
So yeah, this also worked for 3.0.
This, the bump here is really sort of this year at the end.
Again, we have a bit of a drop over the change of the years, but hopefully we can get that back up in the coming year.
So yeah, that was really nice.
And I think it's worth a round of applause for all the contributors who made 3.0 possible.
So, call to action.
I mean, you can join us.
This is picturing a PR.
We just agreed to add 22 more people.
Actually, we're doing two things.
We're changing the governance.
So that there's maintainers and a very much smaller steering committee.
And so it used to be a bit of a problem adding people because it made voting harder.
Because you have to get more and more people to agree.
But because we're changing to have a steering committee, we're very happy to add.
That's just the people a little bit easier to read.
So these are, these are people who a lot of them were instrumental in getting 3.0 out the door.
And a number are just working away on some bit, maybe not premises itself.
Maybe one of the exporters, something in the ecosystem.
Anyway, these people have been working hard.
And we are adding them to the Prometheus team.
We could have another round of applause for that, couldn't we?
Congratulations.
All right, we'll let it open for questions now.
Anyone has any?
Yes, I think we repeat the question.
Oh, you could repeat the question.
You did a shout.
This is more of a joke, but when is more does zero?
Oh, when is four does zero?
Well, yeah, we have talked about that.
We don't want to be another seven years.
We thought, yeah, we sort of discussed, should we say that it's planned for one year?
And then it probably take two.
Yeah, but not another seven years.
Yeah, quicker than that.
More questions?
If you can, please stay seated for five minutes because if everyone leaves,
we won't understand the questions anymore.
Oh, wow, we can just not talk too closely into this one.
So, are there any changes planned for, for the open metric specification,
connecting to the Prometheus 3?
Are there any changes to the open metric specification related to Prometheus 3?
I think that's a no.
So there is work underway to produce an open metrics too.
And maybe I should explain open metrics.
So open metrics was an attempt to produce a standard, which was not tied.
People like being avoiding vendor locking, right?
That's a big thing.
Nobody wants to be locked into a particular vendor.
And in this world, Prometheus gets treated as a vendor, which is kind of nuts,
because there's no company behind it.
There's no billionaires.
You're not, right?
No, yeah.
No, you just got added to the team.
So, yeah, check in a week.
So, yeah, so the open metrics was an older, older than
open telemetry attempt to make a standard.
And it wasn't as wide ranging.
It was just metrics, didn't do traces and logs.
It didn't catch on like open telemetry did.
So effectively, all the excitement is with open telemetry.
But on metrics still exists as a standard.
And there is now a group underway aiming to produce an update of that,
which I think will be a 2.0.
Hi.
Often times, I have difficulty reasoning about like writing unit tests is a,
I find it a bit challenging from for Prometheus alerts.
I was just wondering how we can use the new explain UI,
or if you could throw the sample data to the explain UI that we throw to the unit test
to help make it easier to reason about the unit test.
So, let me take one.
I'm not sure I get the question.
I get it.
So the question is about unit testing rules like the learning rules.
And you notice that there's cold new information in the UI,
and could that information be used to help unit test your rules.
Is that the question?
I think the unit test, like throwing the sample data to the unit test,
make it easier to reason about unit test.
Oh, you want to bring sample data from the UI into your testing rules?
Oh, the other way around.
Like if I have sample data that I throw at the unit test,
you want to use the UI to explain why your unit test is failing.
Yeah.
Ah, got it.
Right.
I mean, if you have a Prometheus on top of your unit test data,
you can probably do that.
But it doesn't.
I mean, there's a, I think the,
I think the context is prompt tool,
which has like, canned data for rules.
And I think I would kind of like that as a good idea.
So submit a feature request.
Please.
Or PR.
Or just implemented.
Yeah.
We got one more up here, then we can come down here, I think.
Is GRPC planned for OTRP?
Since you said you supported.
Nobody knows.
No.
I don't think at this point, but certainly not an expert,
but maybe the expert is running on.
I think that's another one.
We just need someone to do it.
I don't think there's any philosophical objection
to supporting GRPC for OTRP.
Is there any development on the agent mode of Prometheus?
So if I want to have like multiple Prometheus instances
that remote white to the same central Prometheus instance,
there was this experimental agent mode.
But it never really worked that way that I expected it to be.
It's still through in memory like nothing.
And I ended up using the open-term issue collector instead.
So we did have a slide about performance.
Is that there?
Yeah.
So your question was about agent mode.
And we declared it stable, didn't we?
It's no longer experimental.
But it is not relatively any more performant than it was before.
However, Prometheus III, so these charts kind of show that the CPU usage
and memory usage of Prometheus III is better than like a couple of years earlier
and way better than 2.0.
So yeah, we did a lot of work on performance.
It has got better basically with every release.
But the agent mode is not like a spilt like white speed demon.
We've got two more minutes.
Are there more questions?
No, one over here.
What are you talking?
Let me say please do try it.
We just released 3.1.
3.2.
For this last question.
Hi.
Any plans to allow exporting OTLP directly?
So if I have a data store, for example,
I don't accept OTLP that I don't need to translate it from remote by two.
There aren't any definitive plans.
Do we take the remote right two slides too?
Yeah.
So we have remote right two.
That sort of wasn't really a plan feature for three, but coincided with it.
And that does allow to even think about this now.
So it's way more structured that can do the, you know,
you know better than I do where the issues were.
But that would allow implementing this now.
Yeah.
I mean, we looked at, so the question was about, are you going to implement remote,
sorry, OTLP writing from Prometheus?
And I certainly find it hard to tell what's the difference in value between that
and the open telemetry collector.
So if you can achieve what you need with the open telemetry collector,
which paced in a whole bunch of Prometheus code,
you know, go do that.
If you cannot achieve what you need to, please file an issue.
And once again, yeah, join us.
Please contribute.
Make Prometheus a more wonderful thing in future.
Thank you very much.
Thank you.