Okay, thank you so...so hello everyone. My name is Resvanne. I work really bad in the
to give her with two-door here and Seoul there, even though he's not mentioned in the slides,
he's overseeing this kind of like a guardian angel of sorts, sometimes, and we're going to
talk to you about AI for meetings. So, first and foremost, we're currently living in an AI
goal rush, obviously who sells the shovels, gets the most money, and also there's like maybe
a danger of trying to find like some use cases which are not very, so lots of people try to
invent stuff to do with AI. We identified like the use cases, the most used, the most useful
things, let's put it this way, for us, for meetings, in our opinion, it's this, and obviously
for having them, you need transcriptions. I realize you won't be able to do summaries on a
respect action points, do sentimental analysis or whatever. We've been doing, we've been using
the Google APIs since 2017 to do captions, live captions, but we're unhappy with the quality of
the results, so the accuracy was pretty crappy. So, in 2023, the Open AI was released with
the SPR in 2022, and in early to 23, we also created the PRC, making it, making the
SPR before, in real time. What were the challenges here? We needed the SPR to
transcribe a stream, not a file, because SPR was built to transcribe an entire file. It needed
to do it in real time, and it also needed to have good accuracy. How did we accomplish this? I'm
going to try to speed up, because we have quite a lot of grant-cutter. We have a client who joins
the meeting, opens the website of connection to our server, and starts pushing all your chunks.
You see it on the upper left corner, then we have like CRVAD, which is a vertically detector.
Which drops all the other chunks, which do not contain any voice, and then all these chunks get
added to a buffer. At one point, we add a header to that buffer, transfer a minute into a normal
way file, and then feed it to whisper for transcription. This obviously has a problem, because the
buffer keeps growing and growing, and you need to cut the bastard, and we do it
when we do it. Basically, we turn two types of transcriptions, back from the server, an interim,
an intermediary, which hasn't solidified yet, which is still prone to changing, and the file
transcription where we are almost sure that is the file one. How do we do that? We do that by analyzing
the returned results from the whisper, which as you can see on the lower, that's basically
on the line above is the buffer, and the one here in emergency cap, and we search for the biggest
gap between words, and we cut there. This kind of works good, but it still lacks one of the challenges
there, the accuracy part, because obviously if you cut the buffer, then you will lack context again.
So, I don't know if you know it, but whisper allows you to provide an initial prompt. You could
use that to put some acronyms in there, so if you wanted to just to guess the proper terms,
and it will help you, but you can also feed the previous final transcriptions, so it knows what to
how to translate, or you encourage it to describe it in a certain way, and this is what we do.
Each time there's a final for a meeting, so the files from more participants, we feed those
files into the initial prompt, and this is also limited to, I think, 4,000 and something tokens,
and it gives it the context for the transcription. And that's it, that's very much it,
and there's already the captions. I still have, like, a minute left, I guess, right,
and maybe, yeah, and I don't know if you can see it when you know how it here it is.
This is how it works. So, as you can see, this is pretty fast. It could be faster, but we're
pleased with it. So, yeah, and then this gets, with me, start a presentation again,
and close the other stuff. Oh, but you do. And everything you see here,
gets fed to an LLM, and here to where it's going to talk to you about this. Thank you very much.
Yeah, that's okay, I'm just a second. Yeah, you should close that, maybe.
This goes the meeting.
Oh, it's in a browser, sorry.
Yeah, it's gone. Yeah, it's gone.
There's a, it's closed, it's closed.
Nice, I'm closing. Oh, it's, it's, it's, it's, it's the fucking, oh,
the other was showing here, so yeah, that is confused a bit.
All right.
All right. So, yeah, we're going to chat a bit about the summarization, and two years ago,
when we started to develop SkyNet, we had to find ways to enhance our meeting-related products,
which were most of all, were based on GZMit. And we had two main considerations, which,
one of those was the fact that, as I mentioned, we had the transcription at our disposal,
which was already like a core feature in GZMit. And the other one was the fact that we weren't
liking an experimental stage where we were experimenting with, as we weren't, we weren't sure where we
would be going, and the resources that we had allocated were very low. So, basically, all the testing
that we did was on our, on our max. And this is why we, we felt that's providing summarization
would be like a good, efficient, low-cost feature. And this is also because of the fact that
you don't really need like a big model to summarize things correctly. You can get away with,
have, we're trying like a 8-bit on Lama model, for example, and you would get like this in results.
And in the end, for example, today, we still run an 8-bit on Lama model. We didn't feel like
the need to change it. So, just a few, a quick overview on how summarization came to be achieved,
and how we implemented it. So, the first, the first method would be like the basic prompt. So,
in this case, you'll just take the whole text that you want to summarize. You added like a prompt
summarized and send it to every send everything to the Lama and you're getting a result back. Now,
this can be, most of the time, it's going to work, but once you're dealing with larger
payloads, and if your models are not maybe, do not have like a large enough context window,
you're running into this issue where the whole payload will exceed the context window of the model.
So, in that case, you kind of need to do the like a secondary approach, which is what we've
implemented through the map in this method. And this basically works by splitting the whole initial
texting to equal chance, and sending each of those chunks to the LM such to be summarized separately,
and at the end, you're going to just have like intermediaries summary, which you'd mesh together
and send for a final step to the, well, this might not be very efficient. There's like a third way,
but the presentation should be shorter than I initially thought. So, I'm going to skip that.
So, onto some architecture considerations. So, where the way we implement this kind of is,
so whenever you're requesting the summary or action items for that matter, you're going to get back a
job ID. So, this job ID is due to the fact that we behind the scenes, we keep like,
already skew. And so, we keep this already skew, and we basically rely upon the clients to
implement some sort of long-poiling mechanism in order to clear for the results. And there are several
reasons why we did this. And the most important is the fact that, as I was mentioning, we initially
started to run a restricted resources. I mean, not even now we're still running like on two
eight-end GPUs, actually three eight-end GPUs. So, they're not like the top of the line GPUs.
So, running on a restricted resources as you may may know, the lower specs, the lower specs
hardware you have, the higher number, the lower number of requests you can process. So,
this, of course, leads to incoming requests being delayed, in being executed. And this is just
a problem that's being aggravated whenever you're encountering larger payloads. And you would be
forcing like client timeouts and you'd be facing bottlenecks on your side. There's also the aspect
of redundancy. So, whenever you're dealing with this stuff, you kind of need to think about
node failures, which is something that has often happened in our cases. So, this might be even
due to model hallucinating and going on infinitely and you would have to stop it forcibly at some
point. Or maybe you're just introduced a bug, or maybe just the library that you're running,
model on is just crashing in some place. And this is why you plan to need some
some way of being able to rerun the same, the jobs that have failed or the jobs that have been
have not been processed, but were taken for processing on the failing nodes. And to avoid a client
regime mechanisms. And we also had like recurrent big spikes of usage due to rush hours on meetings.
So, you can see in this graph. So, there are like about three, four hundred requests per minute,
minute during a turning-minute lifespan and this was also something that we had to consider when
we designed this small. Now, onto some quick comparison on Lama CPP versus GLM,
we initially started working with Lama CPP because we were mainly experimenting on our local
hardware on our max. And this was like a very useful tool because it was very optimized to work on
various architectures, very, very CPU architectures. And it also provides a lot of
quantizations. It can also provide some quantizations that allow you to run your models on to
lower end hardware like a phone or a tablet. On the other hand, VLM does the opposite. So,
it does like a heavy optimization for the GPUs case. So, it's like the, if you want to go
see us about this stuff, you're going to have to switch to VLM, which is what we did.
So, I wonder. Okay. And besides that, there's also the major aspect of dynamic matching,
which VLM offers in Lama doesn't. And what this means is the fact that you're going to have
you're going to take advantage of the full context window of the model because you're basically
running your request in parallel, as long as it fits the whole context window of the model.
And you can see this in action here. Yeah. So, on the left side, we have SkyNet, which runs
10 requests in parallel. And after 25 seconds, it gets like, this is like emulating the
long polling use case, but it's only does it like once. It's just doing a request at the end
in order to check for the results. And most of the requests are, I mean, five of these already
have about 50K characters, so they're not like small payloads. And you're still getting all the
results in time. Now, if you were to do this sequentially, you wouldn't be able to achieve this timing,
at least not on this set up. So, we, as I mentioned, we were running on an 810 GPU with a quantized
intake Lama 3.1 for the update A-VDM. And some numbers, we've switched recently to Grafana from
Wavefront, so we only have numbers starting September. And we've processed since then about 1.5
million summaries and action items. So, this is it. If you want to check out our project,
this is the URL. Thank you, Wendy, if you have any questions.
So, this project here has both the transcriber and the LLA. So, they're residing in both
you can, it's a module, as all the care puts it. So, you can enable either one or the other
functionality or both at the same time. So, yeah, please check it out, and we have a question.
You know, all the process is welcome. No, nothing is going to happen. No, everything is
happening. Of course. I mean, you can configure it to go to OpenAI or Azure, but this is something
that you're not going to do. It's not going to happen. As I was saying, so you can configure it to
run to either your local sense, yourself hosted model, or you can configure it to run through
OpenAI or Azure, if you would like anything else.
All right, thank you.