okay can you hear me can you hear me so just to soundtrack just to hear me
perfect so thank you so let's track on my name is Lorenzo I'm a humble
flink contributor why my friend here honk is actually a proper flink
committer okay Apache flink committer and we bought war for AWS but we are
not talking about AWS today we are talking about Apache flink and
prometters as open source okay so no AWS service is here just I mean
asking cooperation just to understand you all this the first question I
think I know the answer because we are we are an observability track how many
of you are familiar with promitius okay kind of expected I would say but how
many of you are familiar with flink okay much fewer fine we will explain very
briefly what is flink and what we are talking about and today we are talking
about Apache flink and prometius together for observability use case so
you're definitely in the right track spoiler at you are not talking about
observing flink that's a different story is a soul problem we are not talking
about that so I'm actually handing over to Hong to explain what we are
talking about thanks Lorenzo so I think I think let's set the scene a little bit
first right so I think main thing is like observability right what do you
want to observe the first thing so most of the time we think about stuff like
servers right so we have a server bunch of some compute instances that you
want to monitor like earlier talk about tracing we're talking about logging
right you you have a server you want to monitor it and standard right you have
some form of dashboarding some form of alerting right for me this and
refine our stack standard right so most of the time this works you know why
why do you need what's there there's no problem to solve right but nowadays we
have lost stuff stuff right there's lots of other devices that you
probably want to observe and you probably want to think about how you're
going to build a platform to look at these things so just a couple of examples
here right you have a bunch of devices that at home right IoT devices right
you might you have phones your phone applications that might have you know
not work connectivity issues right sometimes the data is on sometimes data is
off you might have different kind of like vehicles around right you might have
let's say delivery parcels you might want to track right so observability I
don't know satellites for example I don't know like stuff that loose
connection periodically right so stuff basically you have things within the
meta connectivity you have you have dispersed devices and most of the time
you have many many many many devices you don't just have like I guess you have
a lot of services well but most of the times you have probably a lot more
devices right a lot different types of device you ideas and how are you going to
observe this can we use the same stack that's the question right we have a
soft solution you know the familiar with this interface we know how to use
prompt you all we have lots of you know skills within this stack we're
familiar with it can we reuse this that's the question right and hopefully we
can but we're going to explore it a little bit so that's the landscape
about let's get an easy one first right observing service right so we know
most of the time like there are different ways of doing it but like
parameters of those two different ways like let's talk about this one way
first right so you can put in a promise exporter put in some form of end
point you have a scraper right from it is scripts and then you store in a time
series DB and then you set up your visualizing and a learning right using
prompt you all the query and then you set up your graphs simple pull model right
so from it is pulls from the from the data source that you want but can you
apply the same to IoT devices right just do the same thing possible probably
not right how you're going to set up a server how you're going to set up an
endpoint how you're going to be able to kind of collect this data on a periodic
manner intermittent like you have intermittent that will connect to be what you're
going to do so what we do is like we use the push model from it is right so
that's from it is gateway so what you do is then you will basically have
those components remote right using a particular API protocol that we have
remote right protocol and you write it in the promise is right so this kind of
you probably like I guess what's this custom component I guess depends on your
system so it's generic in a sense that because there are lots of different types
of devices right so how do you monitor these things most of the time you want to
set up some integration from your device into into something else let me
stop you it doesn't matter the point is that you're pushing from those devices
into into from it is but what are the characteristics of this kind of problem
space right so one of the characteristics is that most of the time your devices
have small hardware you don't want to kind of set up like a huge huge memory or
huge CPU on your for example your home device on your like I guess if you
use up a lot of memory and space on your phone that's not that's not a good
application right so so the point is you probably want to have you want to keep
it as light as possible right you want to send probably things as like I guess
it's frequently as possible in case some of the connection gets dropped right you
never know that you drop some packets so there's something types of problems
there are specific to this problem space we'll explain more about it later and
why maybe the current stack might might not deal very well with it so yeah
so let's try let's see so let's explore promque offers just a quick summary
for in case people so you can do all things in promque all right so you have all
your data and promedias right then you you can filter and you aggregate the
the series in real time is really really powerful right the semantics you can
do with it you can do rate you can do derivations right derivative sorry and you
can do like I guess window wing as well there's lots of powerful things you can
do with promque yes but then there are a couple of limitations as well so like
a couple of things around joints so enrichment is is something that's pretty
out of the set up so for example if you wanted to let's say send a record with
limited amount of information with little like let's say just a new idea of
your device and you don't want to enrich it with extra information that you
want to do join over in promque in promque it's like that's not something
that you probably can solve that right if you wanted to you could but then you
know it gets complex very easily the other one is maybe some form of custom
derived metrics let's say you have some form of state machine right that
you have in your device and you you want to use that as as a join with another
metrics so it's not that it's it's bad it works right it's just that in the
specific use cases it doesn't work and when you have super high cardinality
data right you have lots of new ideas what you're going to do with it right so most
like if you have let's say hundred hundred thousand of devices or fifty
thousand devices stuff like that you start to hit the limit where you want to run
like over a windowing over a thousand hundred thousand cardinality you fit you
get into some problems so swap performance on high cardinality data so
let's I guess go back to the IoT devices that we talked about just simplify
it diagram right on the left you have some observed assets just to simplify
it genericify it right so something you want to observe basically right the
problem space we're looking at is high cardinality we're looking at probably
potentially millions of devices high frequencies so some of these devices
send lots and lots of data constantly right maybe different standardization
as well some of them send very frequently some of them send sometimes one
second how it standardizes how you make sure that you have the same
target our cross it right and then a couple of things as well like like
maybe they they send repeated data right you get duplicates in case you
drop some of them right so there's different types of problems and then just
to summarize sorry here so you must use to push model we talked about that not
mass but push model fits better high cardinality is usually a common problem
high frequency or different types of frequency as well and usually you want to
keep it like send something super short string right so usually you don't have
the context info so for example like on a device you probably don't know
the creative the make like you again you can add it but then you will increase
that's a trade-off right so you can also reduce and maybe the
contextual info might change the time right you want to keep it life so stuff
like that so if you just use the same stack process on query right then you
run into the problems that we talked about on high cardinality you have
an expensive operation right and you're doing that constantly every time you
refresh it that's what I guess you can cash and stuff like that but the point is
like in the end you still have this expensive query if you want to live data
that you have to run constantly every time when you refresh it that's what
another thing we talked about enrichment as well you probably can enrich it as
easily you probably can you can plug it in into graphite this law of
different graphite will powerful but the point is that there are all your
complexity on this side right so what can we do to simplify it right and help you
to for example you know build up a more extensible or more can we can
be shifted out of it so that's where flink can help you right so what you can
do is that flink can start in as a pre-processing engine because flink has
very powerful time semantics it can handle lots of like like it's built in
handling of time semantics like hopefully it's gonna explain it a little bit later
but basically like you can take it in a time series data it can process it for
you you can simplify it for you and there's lots of I guess similar to Chrome
QL if you think about that but you can do it in a large distributed scale right
and run it across like distributed clusters they can process a lot of data and
you can do it in a reliable way you can if your if your cluster goes down you
can replicate the data and produce the same results right so it's a kind of thing
that can scale very well and it's kind of thing that can produce you consistent
results which is important when you want to look at like counting for example a
simple count you want to count the right number you cannot count twice right
so stuff like that you you can produce consistent results pre-process it
generate your enriched aggregated metrics put in the promidious any of that's
what can just query those so it's kind of like allowing you to shift I guess
to complex these somewhere else is it possible you know of course there are three
offs which you'll talk about but you know this is one potential solution that can
help you so I know what to learn so yep thank you so let's talk about
flink just very briefly and specifically for this type of use case not in
general flink but for who is not familiar with flink what is flink this is
actually from the flink website from the apartment flink website this is the
definition let's go through it because these actually contains all the relevant
key points of flink so a flink is a framework and a distributed processing
engine for state full computation over about an amount of data streams so
what does it mean first of all is a framework is a framework that allow you to
code so you write code Java mainly let's say you're a Japanese language I'm
saying Java for simplicity but you can use any jv language or Python jv and
first okay so you can write code and this is where the power is because you can do
super simple aggregation you know count over tongue window that's super easy you
have very high level primitive for that but you can go down as much as you
need so you can really implement the business logic you're writing codes in Java
or Python it's also distributed processing energy because it allows you to run
it at scale in the distributed way and it scales a lot okay and it's designed to
be fast to manage to handle huge throughput huge is mainly designed for state
for computation what is a state for computation every time you need the
application to keep in mind something account is stateful because you need to
keep the count keeping the count is not difficult you have a variable
memory what is difficult is doing safely in the application stop and restart so
if you just keep a variable when you stop and restart everything is lost so
this is the part that is complicated then fling provides guarantees on this
check pointing say pointing are not going down in that but this is out to the
box with fling so it provides what is going exactly once internal state
guarantees so the count will never be messed up if the application crashes and
restart this is guaranteed I'm using the count as a simple example but can be
something much more complicated for example mini state machines about the state
of one device you can have a state machine for each device and this is state
okay and fling will handle this for you and the last part bounded and I'm
bounded that's what's what is this it's probably not so important for our
use case because we are focusing on streaming but actually something is
there's thing in fling is generalize batch as a sub case of streaming so with
the same API you can actually implement the use case running to on a batch of
data you have a five set you run it as you would do with spark then use which
to stream and you run the same as streaming running continuously the API is
the same and the way fling does it internally because it really generalizes
the bounded data set the batch data set as it's just a stream with bounce
again this last part is not so important for us today because we are we are
talking about spring processing so it's an engine that keeps running in other
important thing about fling is it has multiple connectors connectors the
component fling that allow you to connect to an external system so think
can read from many many system this is totally not exhaustive okay just
if you sample them and write to many many system streams messaging but
also bounded data set you can read files you can read for a database same
thing for writing and again this is not exhaustive notice that permit is not
there or use not to be there and I will get there in a minute so let's go back
to our use case so what we are talking about what the home introduce it we
want to do something like this so we have our observed assets we ingest them we
don't care how you ingest many many tools for doing that different
technology we don't care so you ingest them and you have all the events all the
raw events from your devices and we want to pre-process it with fling and
then write directly to providers that we consider like a time series
did we and then address this what you are actually familiar with actually you
need an additional component in the middle because you cannot push into fling
fling is an application you can push into the application you need to read
fling need to read need to pull the data from some source and because you need
to flow data in you need to stream data in the most appropriate transport let's
say for this kind of use case is a stream storage like Kafka for example Kafka
is probably the best fit but also like pools so you can use messaging you can use
services like in-eases streaming again messaging will probably be fine but
usually with fling with state food stream processing a log semantic like Kafka is
better because they allow to keep guarantees and go back to some point and
replay if needed but anyhow you need a component in the middle so your
ingestion layer layer has to adjust this data whatever they are and send them to
Kafka or any other stream storage and then you will have fling consuming a
real time with quite low latency from from Kafka do the processing and then
write directly to providers so and what can you do with fling effectively well
you can solve the problem that phone was introducing because it's very easily
with the fling API with the programming interface to reduce cardinality because
you can aggregate horizontally and you can actually aggregate horizontal not just
by the dimension you already have but because you can do enrichment because you
can easily look up an external database for example to look up for as
on we're saying the model the maker of a device the location what is the
owner of the device so all data that you probably have in database with fling
is very easy to do it efficiently not you know in the dumb way just quering
every time a database looking up and adding additional dimension to your data
this is a mention are actually necessary for doing a meaningful observation
and because your automation becomes easy to use this additional dimension to
reduce cardinality reduce the number of actual metrics because maybe for some
use cases you don't want to you know follow each car each device alone you
actually want to aggregate them and if you want you can reduce them okay you can
reduce cardinality granularity means frequency in the sense because it's not
uncommon that where the distance of devices the frequency they send row events
is very high it's actually the sending event not just for observability the
sending event for other jobs and they may send multiple times for seconds and
multiple times of multiple samples per second is too much for observing for
additional alerting is a lot of course you can throw it to parameters but it's
a waste of resources and of course you can do filtering if you want filtering
out of layers whatever and you can very easily calculate the right metrics and
this is the right metrics it's not just you know some to other metrics of
course you can do it and you can probably do it easily also in prompt you
well but as we are saying you can for example implement some simple state
machines so based on row events they're not necessarily metrics they're
coming from device you can decide that the that device was moving from this
state to this other state and then back because flinky state full it will keep
the state also if you stop and restart or your application crashes so you keep
track on this and then you emit the state periodically as a metric okay of course
you need to turn it into a number we are talking about metrics but the way
here of course but this is something you can do is very powerful and you could
put any logic there it's programming logic so the top three features are very
useful to scale because they simplify scale you know you reduce the number of
samples you're sending to parameters so because we are talking about tens of
thousand a hundred or thousand device it's a lot if they're sending each
is sending one metric per second and probably each device is sending a
bit of 10 metrics or something that you very easily reach huge number of
cardinality and frequency and during the top one you actually reduce the
throughput to to permitted you can do some you can we just kind of don't
need no granularity the other up to you really depends the bottom one
enrichment and the right metric actually simplify analysis or make some analysis
possible as I was saying for example if you don't have the maker if you don't have
the location of the device and maybe the device don't know the location you
know it because of an external process you have it in a database but the device
don't know it device is only sending you the ID how can you analyze something
that is happening in a room if you don't know if you just know the idea of the
device unless you do some form of enrichment so actually the bottom ones
unlock some of the observability cases unlock some cases that you cannot do
as well otherwise sorry just checking don't we are in time so let let's look at
the integration between flink and promoters so what we are talking about just
you need a bit in the our flink application this is a very simplified
representation of a typical flink application so in the flink application you
always have one or more sources they are connecting to the external system
Kafka in this case or whatever you have many connections doing that you do one or
more transformations whatever what we described and then you end up sending the
data to one or more things we are showing one here but you can actually I have more
we talk about that later and our idea is you have your observative assets you
ingest it we omitted in this picture Kafka in the middle then you have your
application reading from Kafka for solving and sinking in this case directly
we know intermediary to primitives using the push remote right
primitive interface okay so if you are familiar with flink you and I'm sure you
have heard about the flink primitives metric reporters okay so you were maybe
wondering what is this guy talking about it is a small problem it's already
there we're talking about something completely different okay the
primitive flink related primary report is the
time for observing flink is the time for observing the application it works perfect
works well it's a sole problem but you cannot use you cannot hack it to handle
your data it will never scale because your flink application even if he's running
at a huge scale even if he's super complicated that we never had tens of
thousand or metrics and the exporter is not the time for that it doesn't provide
any guarantees it is not designed to handle data is designed to observe flink so
we're talking about something different we are not talking about this what we are
talking about is that sink we need a sink so until recently there was no
primitive sink connector in flink you can implement your own if you want
remote remote right protocol is well documented the only thing you have to do
everything by yourself because there is no primitive client you have to use HTTP you
have to serialize you have to batch if you're trying to send one sample at a
time a primitive will just stop receiving data very very soon you will not be able
to say you need to do proper batching you need to be sure that if you
parallelize the right you keep the sample to the same time series in the same
power right or otherwise permitted we start speeding them back because they are
out of order and other things like that so it was possible what was very
cranky while conversely with the connector just a thing component you just put
it there and you send the data so this actually was released recently in last
November is actually a contribution we contributed the two of us and a couple
other colleagues contributed this part of the Apache flink project okay it's not
it's nothing property is it's part of the Apache flink project it does support
with the recent version they support a version flink 111 20 the API is the same
for thing 2 we have to test it to be honest it should work fine because the API
is the same but flint 2 at the moment is kind of in flight soon it will be
ready but I'm sure we will test it I'm not sure it will work and this just
showing the documentation very briefly we tested it because the goal was
working at scale so we tested the up to like the seller says up to but these are
real numbers one million events per second with my one million second one
million metrics a cardinal to one million this is not the limit we stopped there we
stopped there but because we said okay this is probably nothing and it was
working and the flink class that we are using wasn't huge at all okay
wasn't huge at all we just stopped there and frankly is I stopped there because
I had to request an increase of the quote of managed from it and I said okay
one million metric one million event a second it's probably good enough let's
stop it but I'm sure is capable to go further so they actually some trade
of course nothing and nothing is a free launch so the first one is this
solution requires coding there is no no code solution and you need to write
code at the moment is only Java we are not yet supporting the Python flink
Python interface we will probably get it soon but usually if you want to have
big volumes and a low latency is better Java again I'm saying Java JVM many
people are using Kotlin for example or Scarla Scarlet is becoming less
special but for different reasons but you can use it of course with flink the
other point is well you need to run your flink job so you need to set up a
flink cluster you can run Kubernetes you can manage it is not crazy brilliant but
you have to do something okay or you can use a managed managed service wing
but anyhow you can run whatever you want so but you have to consider this of
course and you have to consider whether your observability team has
this skill or maybe you need to bring in someone with the skill so actually
handing over to Hong for the conclusion I think we are in time thanks
right so I think just to kind of summarize what we've covered right so the
problem space you look at right is like devices lots and lots of devices right
so where at scale we have lots of cardinality of high frequency maybe like
you might need to clean up your data a little bit like you might have like
devices which only can send small amount of contacts right that kind of
problem space and we still want to use the kind of stack that we're familiar with
right we don't want to build a custom monitoring solution we don't want to
build a custom dashboarding and letting solution right you want to use what we
already have you can use flink to plug it in basically so you don't have to
learn the end users don't have to learn something new you can use that and flink
we're a bit able to allow you to scale your things you're observing right
so that you can still use the same solution without with low latency and
you can yeah so it solves it for you basically so I think another kind of
kind of easing a little bit of how for the internet flink can do as Lorenzo
mentioned flink connects a lot of things the job we showed is very simple linear
right one source one transform one sync that's not all we can do right so
the point of bling is that you can like one of the things is stream and
batch unification which can talk about later if you're interested but the
idea is that you can you can produce data at high speed and you can produce
data like in the batch and you can use the same job to do it the idea that you
can read our body that's it's a lot of things that it solves but in this
case it gives once you plug it in you get lots of things for free which is the
primary trying to say so you don't you can call obviously you can call your
server and can manage it but using flink plugs into a lot of things so the
connector we wrote that's free you don't have to implement the remote right
up for the call and everything you don't have to manage the time
semantics you don't have to manage the checkpoint thing everything comes for
free and in fact what you can do here is sync flink and write into multiple
sources it can write the same data with consistency into multiple sources so
for example you know a lot of people they they want to write it into this
dashboard installation for maybe a particle use a group but maybe they want to
store it like for example an a data link right somewhere in long term right
way you don't have to like you can query over like five years right without
without having issues and what you can do is flink and allow you to write it in
both cases using the same job using one distributed cluster one job and
distributed cluster to write the same semantics and the same logic into the
off both of these things you update the logic of your job both of the logic
changes at the same time of course there's coupling and stuff that's straight
off they can talk about by the point is simplifies all your logic into one so
you don't have to worry about like like diverging data analytics platforms
which is a mess sometimes when people stop building incrementally so I think
it's and flink scales so you don't have to worry about that you can migrate so
I think I think another thing to mention as well is handle multiple types of
formats so if you want to do right to data link with a different format that you
write to promitias for example you can write in the column format up to you you
can do it flink and handle it for you and you don't have to worry about it so
hopefully that's convinced you a little bit about what one potential solution you
can try yeah just a couple resources we will we have shared the slides so the
first one is the documentation it's actually part of the flink of
documentation so you don't have to search is the a connector to the
documentation the second one is a GitHub repo with an example of the use case
we described of course it's synthetic we have a data generator and the
digital generator is simulating they could connect the vehicles sending the
different metrics and and well you can set up everything and you can run with
tens of thousands of vehicles is actually another flink application generating
the data and you can see how it works and also we have a separate application
just writing the rule data is the are in promitias so you can play with the
dashboard to see how it is getting something useful from the aggregate and
rich metric and try to do the same with the raw metrics.
Romantic are simple but kind of realistic okay so you can play with it it's it's
a little repo and that is it please get in touch if you have any questions if
you have any questions here we are here and the question we have one question please
for me say that I'll ask you absolutely have to leave so you can all hear the
question and answers okay so I have to go over there
Hi the enrichment use case is really nice the enrichment use case is really nice
but for the cardibility reduction and the granularity reduction promitias already
supports let's say 2 million metrics for cities now you have added a Kafka there
you added a flink cluster there so I'm now adding more nodes more machines
so can you do a cost can you share some cost analysis like okay I'm reducing
two million how much more nodes I have to add in order to get the reduction
absolute I think it really depends on the use case to be honest so if you just
do cardibility reduction frequently reduction it might be an overkill the
thing is sometimes you cannot do credit on it reduction because you don't have
the additional dimensions so I think the key features enrichment to add
relevant dimension and then we'll allow you to reduce the others the other
thing is it might be well you need to scale promitias but I reckon if it is
done if it just that you can probably do with promitias the reality is what we
have seen is most of the time you also need to have additional information
dimension that you don't have because the device is just sending the ID and
the metrics and the measurement or an event or event this will happen
let's say so you're through it we didn't do any proper you're sure sorry
a bit of an acquisition but yeah just add just under the add to that I think
another thing that this you're right you're adding complexity right at the
front but why is it the right place that I complex deep because for example if
you're writing something like Kafka and you want to change the type of
aggregation you want to run you want to run on the exact same data over the
last I don't know Kafka can stall like that's multiple tiers you can stop
so you want to run over the last one year and you want to change the way you
analyze it fling can do that for you you can read from Kafka one year ago
repress it right into someone else if you're using it you can just plug and play
right so the idea is that you can do it multiple ways but doing it here like
people I guess people are trying to sell this idea a bit more because because of
that you can you can change the way you analyze and the list is changes right
you get new data on I don't know like one year down the road and you're like
hey you know let me let me read we cut it in a different way right this gives
you flexibility yeah has gone and do you find that you sort of push the
problem back like further than with my no cardinality hello okay do you
find that you know obviously cardinality explosions are very common and
premieres but do you find that you have it creates a sort of an overhead
of maintaining those kind of rules and jobs like further down down the
stack closer you know do you have is there do you have a sort of a rule
explosion then you know maintaining this huge rule session is that I mean I
find that with recording rules and premieres which is the way a lot of people
tackle this kind of thing that you just end up maintaining this massive rule
session so do you find that with the fling jobs or is that a thing yeah so
if I understand the question correctly so one problem we face when we do
all this processing and premieres which can do you can do like a rule set that
you can kind of have the right metrics and then but then after why it starts
exploding right you have lots and lots of rules and you don't remove the rule
because maybe something later on is gonna use it so you get a huge list so
fling I guess you can end up in that situation I guess the difference is that
maybe in fling like I guess you're accepting the fact that you're gonna
code something I guess you can write you can write in SQL you can write in
Python you can write in Java but the thing is then you have the flexibility of
not a config file you know the flexibility of testing that code you have
flexibility of integrating it with dummy data and assuming that you're
you're and then forcing the interface the data interface that whenever you have
this type of data you're gonna get a discount results so it allows you to have
some pump your I guess it does add complexity sorry not gonna hide it it is
complexity but it adds complexity in a right place right where you can you can
then enforce against API then we talk about data contracts we talk about
different things think it's not the only solution you can use like something
like spark which maybe you're more familiar with but we can talk about the
differences between spark and flingulate on as well this batch of the streaming
stuff I know so if you need to add rules you need to add them somewhere so
either either put them in point QL or in your final parameters in the
different forms I'm saying from QL just to simplify or you move it over there
you up to you the good news it well in thing you can unit test good luck
unit testing what you do on this side talking about long-term storage can you
write directly can you yeah for long-term storage can you write directly into a
finals I can't directly do the question sorry because he's very
know as you're asking about the long-term storage yeah long-term storage and can
directly into finals as opposed to going into a previous and then going into a
finals so sorry is really noisy right directly into finals oh in
times yes sorry well effectively you are because the remote rights the
remote right interface is just an interface on top of whatever is the
back end so this was actually what we're doing we were using just for
simplicity the managed parameters that is actually based I think it's based oh
I'm not sure what is based on but it's not just you know standalone
parameters so the answer is of course you can of course you can keeping any data
like is for different use cases because you keep it there you do a long-term
analysis you train machine learning you do whatever you need but it's it's
parallel so your dashboard is fed by parameters you know how to do it you have the
other data I think you've heard it all you mentioned that the
permit is sync implemented the remote right version one one are you planning
to also implement the support for the remote right version two which was
introduced with permit is 3.0 oh we are planning as soon as it will be you know
stable and really I think it's declared stable like we'll probably hear the
next talk but it is okay remember it's already declared state that there's good
I was we will we will we we could be updated