Thank you, so let's repeat for the stream. Today I'm going to tell you a little bit about
the site evideo.bg and how it was born and how we made it possible. The title of the
stock is scaling to 12k live streams and 12k you will understand from where it comes.
So first of all, who am I? I'm Vladimir. I'm mostly a participant in DevOps and security
guys. I like play and do automation and some tooling. I do some architecture and everything
that gets thrown by sites. I work for the company information services. The company itself
is the national integrator for the Republic of Bulgaria. We have different kinds of services.
We are a eidus qualified. We are also a member of Uritus. We have platform service, regular
infrastructure services, on-premises, clubs, everything. We also do stock as a service for the
public institutions in Bulgaria. One of some of our major projects are in health care, transportation,
financial and judiciary branches of the government. The problem that we have, there is legislation
and the legislation says any and all ballot counting processes and activities should be recorded,
streamed and dark-eyeed in real-time. The requirements for that simple question that simple
problem are first of all, there is a legislation and the legislation says, just live streaming
should be done recording, filling of the forms, counting of the ballots, basically the whole
process that starts when the poll closes. The recordings should be available online. The recording
should be archived and it is absolutely mandatory with several small exceptions and the exceptions
are mainly for specific settings like for example elder care facilities, medical facilities,
gel facilities and similar. Basically where there is sensitive context, we are not doing
any monitoring, but it is mandated in the law itself. The law came into force in April,
in February, a little bit less than two months before the next election. The whole system
had to be designed, deployed, created, operated in that short time. The devices that are provided
are going to be operated by the polling station chairman, at least that is defined in the law.
The person people operating it are usually not really trained and there is no bonuses rewards
but also no penalties if something doesn't happen right. Which basically means untrained personnel,
uncertain conditions, the only thing that we can control is the device itself.
The scope, we have close to 12,000 polling stations in Bulgaria,
all will be streaming or at least, should be streaming. They start mostly simultaneously,
they stream for two to six hours on average from the election day itself,
and those were the things that were known from the start. On top of that, we added some more
some more of the requirements. We want a really nice audio channel so you can hear the audio,
which is usually quite important. We also wanted to do a parallel recording or local storage,
so we have a backup in case there is no connectivity. As I said, this needed to be
live and operational in less than two months. The solution for it is simple, nothing strange,
nothing ordinary, just the scale and the timing were a bit problematic. So, first of all,
we need a camera. Mobile phone will do, in fact, the mobile phone, it might sound strange to some
people but otherwise it's a very fully featured device. It has a built-in UPS in the form of battery.
It has usually a decent camera that you can use as a monitor. It has microphones,
if you don't know, the microphones in recent forms are really nice, really good, really crisp,
they also come sometimes with an array of microphones. The mobile phone itself has multiple connectivity
options, including words via USB-C, for example. More and more devices have hardware
accelerated colleagues, so it's easier to do this in hardware than to write the software and
back the CPU to the top. They're usually pretty simple to handle or almost anybody can
handle a phone. He usually does any day of the week. The mobile phones have a very, very well
accepted solution for remote management in the form of mobile device management software.
Additionally, we slapped a little bit of RFID attacks and QR codes, so it's easier to ask it
manage them. The devices that we chose, they were based on Android Go, the Android version was 13,
and they were quite inexpensive in fact. If we go back to the previous requirement that this should
be live in two months, try to find 12,000 falls with a lead time of less than 30 days.
I really challenge you, it's not a pleasant task. The other pieces of the puzzle that we're using
is mobile application of course. We need to do some kind of streaming, so we developed in
house application that's doing this. We are not using anything fancy, just of the shelf libraries
and our custom logic. The application itself needs to do the recording to, the application itself
needs to be configurable by some remote means. We chose to do this via the MDM. We decided to specify
QR code as a means of initial configuration, starting stopping streaming and other tasks.
The QR codes themselves are encrypted, and so the data is mostly secure. If you have enough time,
of course probably you could break it, but you usually have the QR code for something like
24 hours before it needs to be used, so we're relatively sure that it can be broken in that time.
Of course the applications need to evolve. We're constantly improving them with recently added
health monitoring, the application itself will hit the phone, the state of everything that's
happening, the audio, the positioning, the orientation. We fought it much more logging since the
initial implementation. What I forgot to mention, the system has been already used for
four different major elections. I know three, sorry. Three major elections that are covering the
whole country, several several smaller elections that are happening due to certain conditions.
One of the major elections was a combined election for parliament and the European parliament.
So it was, I think it was pretty well tested. Better encoding profiles were chosen, so in order to
lower a bit the CPU usage and the temperature. The performance has been improved. All in all,
we're developing it because, yeah, it's good to be developed. We also need a mobile device management
solution. We found a big plan that's open source. It provides the director for monitoring
logging of the devices themselves. It supports different profiles, but that's usually all of them do.
It stores the configuration. It supports messaging. It's as synchronous. It doesn't block
all the devices calling tweets and if it needs to push the push information the other way,
it also happens as synchronous thing. We can also do via the MDM, forced updates and
other different tasks. The MDM itself allows us to lock down the device itself, so it's not so
the user cannot easily break it. Of course, excluding dropping the phone itself on the floor.
Some results, you have run already eight different elections that I said. Both combined,
both single, small, large. The graphs that you see on the top, it's just the statistics of
using the website that's caused the results. The bottom picture is a screenshot from our internal
control plane. The top row is listing the active sections that have been active for the
election itself. The bottom row is the current active ones. For the public reception of the
system itself, at least we want to think that's very nice. But it enforced a stricter process
controls. There are more than 50 official complaints in the court for which the sole basis.
Well, not the sole basis, but the main basis is the video monitoring that we're doing.
There is already one of the cases, has reached the final judgements and conviction.
Many of the cases are still in investigation and more or less constantly in new cases are coming.
So, all in all, I can say that the system has increased the water engagement and the public
engagement in the process itself. The pictures that you're seeing are the top one is from the
already finished court case, that has reached the judgment. The bottom one is again from
very possible manipulation for the process itself. That's also being investigated.
Of course, you cannot do such a system without testing and of course I guarantee you the testing
will not be enough. We did quite a lot of testing and then some more and then again some more.
We were doing testing in the development process on a freshly provisioned environment
in a clean in a clean state. In the polling places, we also did some testing.
We did performance testing continuing to wire out some details and I guarantee you when
you deploy a system in the world, all the testing will not save you because the world can
throw at you very different stuff that will break you. Now we come to the infrastructure,
that's the basic stuff that we are going to need. So, the calculations are brought to the
simple 12K in-bound streams, megabit and a half for each total of 80 gigabits in-bound,
outbound and certain. We have seen several thousand online viewers but that's frankly said
not the thing that we are monitoring due to the way the infrastructure is deployed.
We unit 1220 to 40 terabytes of storage space for the recordings of a single election cycle,
at least for the big ones. You need sufficient connectivity to all Bulgarian telco providers
because as we already know, we are using just a phone and the phone needs to be able to talk to
its mothership. We are not using Wi-Fi, we are using 4G for the connectivity.
From the infrastructure itself, we do require quite nice performance,
especially on the storage sites. We decided that we want to use
it's going to like this large scale or largely scale infrastructure and for that we need it
relatively well established provider. We need to do quick scale outs because frankly,
we can provision infrastructure preemptively but there is not a real point to do that.
The later you do it, if you do it in the cloud as we do, the less money you will pay.
We needed quite good support, at least mostly in terms of networking, troubleshooting and security.
We require multiple locations so we can easily fail over and of course,
security and details protection are almost for this kind of world we live in.
So, the components, if you cannot tell from the picture, it shows GCP,
the components are relatively simple. We have a control plane that's running on the compute
engine on the bottom. We have a cloud test kill in the form of postgres that's our DV layer.
We have a video on the month that's passing mostly through cloud services.
The storage is on persistent disks. We use a shared cache,
reducing effects. We have on ingest and terestream clusters that are again running on top of compute
engine and then we're using the firewall options that GCP provides us.
Probably the first interesting note is the ingest notes. We try to run in with
multi-rtimp and then we set out on media antics. Our main goal usually is to try to use as much
open source software as possible and to contribute back whatever we can. In our case for the
application server, we're using engine x and topon resting. For our team p ingestion, we're using
media antics now before we were using multi-rtimp and for processing control and some communication
we're using rediscellary and FMPEC. The streamer note is extremely simple.
Engine x and open resty serving static files even though they are
HOS generated. It's absolutely fine. Basically we're streaming HOS.
The protection of the assets themselves so not to allow
easy way to do those ourselves is done in a layer approach. We're doing caching on the HOS chunks
in memory for improved performance and to lower the note from the ingest notes.
The control plane is in-house developed mainly in PHP. It handles internal eventing,
provisioning of the elections themselves, provisioning of polling places,
defaults, controls, who goes where who can stream what keys should be used and so on. Basically
that's the thing that is making the process possible. The other things around it are just talking
to it, getting data and doing it's bidding. The last part of the process is the data retrieval
because as you remember we have also a local recording and sometimes there could be no online
recording because there is no connectivity, any kind of connectivity. The data retrieval itself
is super simple. Unpack the phone, plug it to a USB-C and application written in
open Pascal. Pascal. Application written in Pascal just downloads whatever is needed from there.
Quite simple rules. Again we have a very involved processing chain that's mostly concerned
with making absolutely sure that the files are good, right and proper. If they are broken files
we try to automatically fix them. If not they go to broken and somebody has to fix them manually.
The files need to be arranged somehow in a file structure. There are a lot of manual checks,
a lot of process of manual processes and does the reason why we have to verify this. If you have
a 12,000 devices in a single room, go find number 42. If you don't have RFID, it's a total mess. If
you have RFID, it's a gain mess but not total. Just takes a little time. Other interesting pieces
of the puzzle or things that we learned, don't reinvent the wheel. Try to use platform services
if you are using cloud stuff. Do test, do automate and always have a very good backup plan.
Implement pre-flight checks. If possible, automate them, run them twice and then get somebody
to look over them manually. What we use, Google Cloud Services, Ansible, Terraform to stand up
the infrastructure cloud flare for DNS and some white caching. That's it. Google Cloud Platform,
the GSP provides us very nice services in the form of firewall caching and global distribution
of facets and if you have such use case it's quite nice. Of course they are issues as always.
The provisioning of the devices is manual, the upgraded self is a gain manual. Some things
can be automated but most no. If you start too many devices at single, at a single point of
time, all the telco providers will be very happy with you. You can see phones going to the
base station for miles and miles and miles away in a city environment, which is not really nice.
Even though we do training videos, we print materials that are used during the process themselves.
The people don't use them, don't read them, forget them, turn the phones upside down. That's one
of the reasons why we implemented self monitoring of device positioning. If the device is not
positioned properly, it will talk back to you. The first time you hear this, it's quite scary.
People really like the devices. The package that we ship is phone stand and instructions. The
stands are some Chinese brand and they require very deep technical knowledge how to operate them.
Sorry that that's literally true. They require technical knowledge. On the process of downloading
the files from the phones, choose quality cables. That will save you a lot of grief.
Truth-server-vide stickers with nice antennas. We can recommend the UHF ones, not the NFC ones.
And that's mostly it's what next. We have a hundred and twenty-terabytes of videos in Bulgarian.
That's some options that we talked about. I don't know if you can think something else.
Talk to us. We'll try to make some deal to get to you that data.
For example, yesterday I saw that the VOC guys are trying to do live transcriptions with AI.
Maybe we can try that too. Questions? The QR code leads to a feedback page.
That's my email address. Go for questions.
Yes.
For how long do you have to keep the videos and you have to keep them available for people for the
whole time? The question was how long should we keep the HEF to keep the ERT facts basically?
The streams and the recordings from the device itself. The law does not strictly define that.
If you want, I can try to find the line in the law that starts at the whole process,
and it's less than two sentences.
Next.
Yes.
I didn't follow how RFID helps you out when you have to have the devices in one move.
How RFID helps with too many devices. So if you are searching for a device,
and it's in a big pile of other devices, the RFID helps you to narrow down where exactly to look.
The folks, we pack them in boxes, large boxes, 60 phones per box, in their own box.
So basically you have a stack, and then okay, this is in this box, pull it out.
In this section of the box, let's start taking out the device.
The other way around this, pull every four and three, the barcode with the unique number on it.
Okay, next. Yes.
So the question was, back up plans, what kind of back up plans, right?
Okay, so first back up plan, walk a recording.
Second back up plan, keep the life recordings on the cloud instances for some amount of time on top of that.
Third back up plan, keep the recordings in a bucket, because string from a bucket is kind of a requirement for the
serious service that we use.
And then after you download everything down on the local storage and check everything,
just after that delete everything. And yes, we had to use that.
Yes.
Two more good questions. One, but there's no penalties if someone doesn't turn on.
Is there something on the first slide? Is that a question?
So they have what 27 or 20 important requirements of data? How do we be the multi-sync activist
watch it? You have 50 court cases? Did they call it themselves in a room for one more time?
Okay, yes. Okay, so first question, sorry, I forgot the first question.
Penalties, yes. The first question was, what happens if there are no penalties?
Well, there are no penalties, nothing. People can not turn the phone, can break it,
can do whatever they want. I think that that will be changing shortly, though.
The second question was, how do the democracy activists watch this? In at least what we know is
two ways. The first way is about downloads everything. There is one specific person that we know
is doing this. The second option is, the polling places, there is statistics where there are
suspected forgeries, fraudulences and so on, and those are the most monitored.
Usually that happens by an organization that says, okay, we need 50 people. You are going to
monitor this five, this X amount of stations. Otherwise, as we said, 20 to 40 terabytes of
video, that's roughly seven or half to eight years per election cycle. So single person, it's hard.
Otherwise, we found out that people mostly watch the polling places around them.
To see how the process works, and there are a lot of people that are doing that and just jumping
through the video, and some things get captured just by chance. Okay, there was another question.
I'm not sure. I have talked with some Polish guys during my work, not specifically on that project.
They were telling me that something about this was thought about in Poland, but was not accepted.
Otherwise, sorry, I don't have information.
I have a couple of words about storage. Did you use something as thick and potable or just huge
files share of each note? The huge file share, not going to work. I always won't work.
We used local persistent disks, and after that, we also told them to GCS, and from GCS, we get them down.
Do you have primary pick-up stream, or just a primary pick-up stream, or just a primary pick-up stream?
The device is the foes themselves due RTP, and then to the outside world, we do HLS,
any Anto streamers can reach any Anto in just notes, so it doesn't really matter where you hit it.
How much does it cost?
Estimation about the whole project, not sure.
Okay, let's try to do a simple estimation.
Hmm.
Okay, 5 to 10 million, maybe. I will really depend on...
Well, thank you. Plus, the hardware that will be installed, is every election after this.
It's much cheaper than the cost. It's enough to be able to buy another devices and develop some.
Okay, so let's repeat it for the stream itself. About 2.5 million Euro, plus the devices cost.
The costs are very high upfront because development testing can solve and so on, but after that,
running kit for each and every cycle, it's much much more cheaper, up until we need to replace the devices.
The devices themselves, the Android foes, where something like
100 Euro,
64, I'm completely out of numbers.
60-old Euro port device, so the devices themselves were even not that expensive, but that was one of the reasons
why we were looking for such. Do we have a time for another question?
Okay, so go.
Yeah, the phone from the devices, they do not listen if you're with battery spawning,
because I would expect that's gonna happen after.
Did we, so the question is, did we notice any battery-related issues?
Up until now, not really, we found during the unpacking and downloading the device data,
we found several phones that were quite hot because they were left on.
We had total failure rate of less than half a percent, maybe, even less.
Something like less than 50 devices have failed, and some of those have failed by being
repeatedly hit, or fallen, or rock, or something like, basically, physically that.
Otherwise, no, not really, but, well, you have to have in mind that those devices are not
actively used and abused.
Well, that's what I would expect to be from, because when you keep the charge battery in a storage
and leave it, then you kill as well, and, you know, I'm not so good at battery soap,
I don't really know how battery is behaved.
I recommend to our splash by slowly.
Okay, will you check? Anyone else?
We have a lot of connections, but there is no time to go.
Anyone else?
Yes, sure.
How do you manage the inventory of 7.2?
How do we manage the inventory?
Well, thank God for the RFID.
Otherwise, the inventory management is built into the control plane.
It has an inventory module.
It's nothing too fancy, nothing too special.
Probably, if the person is asking about physically managing the inventory,
that's another question.
It's not so easy.
But 12,000 folks are also not such an extreme volume.
It's something like 8 to 10 cubic meters.
Yeah, sorry, we didn't took a photo of this,
but on the next cycle, I promise I will have a photo.
Anyone else?
Okay.
And you mentioned you used WiFi, you have a project.
Did you have some issue of this mobile project?
I mean, some problems with your connectivity in some places.
Well, yeah, okay, so the question is,
did we have any connectivity issues, specifically with 4G?
Do you know how many different labs,
well, labs, companies, sites, there are that are claiming best network in something?
A lot of, well, there are at least three because each and every
token provider is using different ones and everyone is saying
that's what we have the best network.
That's it.
On the first cycle, there were some issues that on the second cycle got corrected.
What did the operators themselves, the telcos themselves are doing?
I don't know, but probably they are doing either prioritization
or just physically moving cell towers.
At least some years ago, I know that each and every telco provider had at least 12 mobile
cells to deployment either due to demand or brokerage.
Okay, anybody else?
Then, thank you.
Thank you.