Hello everyone, so that's the third year we have this wonderful room and for them and that's
the second year I'm going to talk about LMS in Sponsor 3, so from last year talk when I was
talking about using LMS for Sponsor 3, year has passed and I would say that I'm still a bit
skeptical about fully replacement of Sponsor 3, because it's still a bit unstable, it's still quite
costly, so you can ask multiple times and get different opposite results and what's more important
there are some privacy concerns that are not solved and for sure there are problems with explanation
why this message got rejected or not, because you can't simply get into LMS that deeply
to understand why this decision has been made, but for sure we have added more support for LMS in
Sponsor 3, in particular we added LMS context, we added the search context, so it's
in general possible to represent to LMS not only current message but also history of previous
emails and it was quite useful, but it's still one thing, but what has changed in the past year
is that development with LMS has changed dramatically, so it's now completely different thing,
some people are thinking about it like replacement from assembly to compile language and when
we previously has written code manually, now you can just write some prompt and get the code from
it. So basically, in 2024, what did we have? We have code completion, we had compile it,
we have that was quite good, so basically you write small things and get help from a system
so that can check your code and suggest something good or not, but it worked in this way, so it was
mostly assisted coding the outer completion, for sure it was mostly for one file edits and also
it could do some documentation look up, so you start to type and you get documentation about
this and this function, so in 2025 it's completely different stories, so basically you have
your LMS fully assistant that writes code, that can design architecture, that can do multifile
refactoring or things about it, writing tests, testing stuff, debugging things, debugging issues,
fixing them, making reasoning, making commits, doing good workflow properly, so finally I get
proper commit messages for example, that's a big, big, big action and for sure it can do lots of
projects, so literally my current workflow is the following, so I have two console tools, a
code and refactor droids and I'm using them for different models, so I'm using git work work work
trees for doing stuff independently from the main branch and different models can work with different
basically work trees and do their own stuff there isolated from the main environment, it's very useful
because after all you get results, you review them carefully and you select what to get and also
it's good run different subvegenes because it's a big say of tokens, big say of projects and
if you work on some huge projects it's good to split it into small parts and refactor them independently
in different models in different sub trees, but I guess many you are developing the LMS after some
experiences and end up with something similar, so the basic the main gain and I think the main advantage
of all these assisted coding is that you can prototype fast, so I have lots of ideas and one of the
biggest issues is that some ideas are too large to implement, so I'm going to explain it a bit later
about some examples of these large ideas, but now you can just do this, this and this and if you don't
like you can just throw out the code and you have no regrets of it, that's a big thing
and you can actually explore different things because if something doesn't work you can always go back
and build something else and again you don't have a little regrets about miss at the
opportunities about miss time and about miss movement of your fingers to write this code and that's
very good, another big advantage is really better documentation, English is not my first language
and I'm not very good in describing things and that's I think a problem for any author of software
because you know your software better and you imply a lot of things that are not apparent to other
people and you get angry if you get stupid questions but they're not stupid because basically it's
your knowledge and if you apply your knowledge to this question it's stupid but it's not stupid
because you're not explicitly defining it and one biggest help was automated boilerplate stuff
for example the website's terrible without telling him he needs no uncibly need to
to program in YAMO you need to do deployments and list deployments and respite line is
good absolutely fixed by L and so if I get my server crashed or revoked or hacked I can easily
rebuild them now because previously it was like secret knowledge of that there's nowhere to
comment it and you have no documentation nothing like that because as I said I respond to this
that there's a hobby project and mostly building it to something bigger building it to the
fully open source project was a bit painful and mainly about site projects such as documentation
analytics and stuff like that. Real disadvantage is that you are a bit out of context so you
have multiple models that are doing their stuff and sometimes you don't even understand what you
are doing now so yeah sometimes some conversations can be very difficult so I remember one issue
got into reply from plot that it told me like you know it's from the better please advise me where
to go so that was a very open recession and this was very very long thing and sometimes it
starts to do very simple solutions so quite recently I've found some bug and tried to debug it
and closed decided to let's compare pointer to function and if this pointer has a lot of pattern
it's bad point so of course it's bad pointer but the question is not if it's bad or not because
but the question is why it's bad things you need to we need to worry be very clear about that and for
sure it's impossible to fully build large projects without human attention because AI attention
it's a mathematical thing and it's quite limited I would say so LM is not an expert in
response the LM is expert probably in writing code but not in years specifically yes and for sure
you have the main expertise yes and one thing that it's quite important for the open source
as a whole that actually your own value is a bit lower now because many people can actually
check the same things so if you have a question about responding I think yesterday we asked
one question from code and it found a wrong solution but when we fixed prompt something like
check the documentation and do it properly it created a proper solution with like a multimapo
these two conditions like get this from and get this recipient and get them matched very simple things
like a simple rule that many people are doing in that day to day job to filter some specific
spam or do some specific policies but now it's possible to do this code but unfortunately not
directly so if you ask it to do like write me a rule first from day to do this in this and this
usually it's it's wrong it's just wrong so actually what are not problems and what is usually
addressed well and development is that code below is not a problem because if you write prompt carefully
if you're a view carefully if you ask them to refactor bad places it's okay same as for LM slope
so you can actually ask before commit to fix and clear everything it's okay hallucinations is
again a problem but you know that it's it's crap and you can fix it again you can review and actually
you can ask another model to check the documentation and check the code and do review and find
these hallucinations and fix it so it's not a real problem so here is a short list of different
prototypes that I've done over a year and not many of them are parts of our sponge itself because
there are lots of things around how to manage documentation how to build new site how to build
analytics how to write client integrations in different languages which is also quite interesting because
for example I don't know go but many people will ask me about native clients for go and it was
very simple to kind of a trust client to go client with all creepster stuff and stuff like that
so it was quite good finally for a lot of them for example finally they have interfaces for
our internal structure and that was quite useful so okay I'm fine about the first part and now I'm
going to tell the detail about some of significant projects that were done over a year one thing that
I was thinking for maybe like two years so far was adding fuzzy detection for HTML structure
the problem is that many sponge is built from templates and these templates have different texts
different basically addressing like deer somebody deer different text and stuff like that
sometimes they have different personalized personalized links with the same domain but different
hashes different queries and the idea was to build fuzzy hash on top of HTML structure only so get all
tags extract features from that like waterfalls, waterfalls, water styles I used and what's more
important that they also include so-called call-to-action domains again a cultivation thing is actually
our wood is clicked by user so like wood domain get clicked from this email with some
button like subscribers, subscribe, go, whatever because there are lots of links that are not
relevant so for example you have a foot or something like facebook and policy and stuff like that but
this links are relevant and we only check call-to-action links then we use the same algorithm for
fuzzy crushes that we use shingles it's like me and hash trick so it's quite simple and true
and then you find similarities between different shingles and compare how close it does
some match things so you have some hash with shingles and you match them either directly if it's
had exactly the same structure or a bit indirectly if we have some common things but not different
but what's important that we use call-to-action for hash on everything so if call-to-action changes
it's a different hash the reason for that was to distinguish fish from normal email because sometimes
swimmers are just getting legit emails and just fix the main and that's it but we need to distinguish them
from normal ones so why it's important because we can do proper white list thing with fuzzy
crushes because we can get some brands shipping all the confirmations one time codes alerts and that's
white listing part and that's important to have legit emails that are based for non-sanders no
for example let's imagine like social network like Facebook link and then they have different
content but they're templated and always the same and that's good to find stuff like that for
white listing because for group sponsored and system white listing is even more important than
black listing to be honest but for sure it's okay for black listing as well so we have some real
life testing and it really works well with templated emails and they still exist and they're still
widespread some examples of that. Lotures comes fishing template based fraud stuff like that
so in general that was very good project and I would say other if I did it but full code on my own
it would take maybe two or three times longer to implement now it's up and running and it's okay
yes and city was just like a benefit because now we can get the real links and distinguish them from
but from insignificant links. How do you get those? Oh sorry I don't know how to get back.
So significant links are usually detected by belonging to specific classes like button
or they are highly visible by CSS and as they are not in footer so we are trying to detect
these by heuristics but again for sure it's set of sets of heuristics and it's possible to cheat
them somehow but now Spanners are not trying to break it because probably it's just to you.
Another project that was surprisingly failed to Google some of code project and to be honest
I'm not quite sure how Google some of code is planning to proceed in this year for because of
all of the phones. I really don't understand. So historically we had only Spanners' home classification
and the idea was basically to extend these to multiclass and for sure the mathematics behind
by his and theorem is not strictly married with binary classification you can do one versus all
which is actually used or something else and it's good to check not only Spanners' home but also
mark different other labels such as transactional and mail social and then emails personal emails
and so on and so forth. And yes now we have multiple class and this actually is wrong. So I personally
don't suggest to make Spanners' home in different other classes so I personally suggest to have
different classifiers. One is binary for Spanners' home and one is binary for other labels.
Same thing so basically you get corpses and trained them via common language so this is the whole
architecture basically so you get message you extract and split it into big grams, add some
meta tokens and then you have cash lookups on side of radius so it's all implemented as
already scripts and on radius you just get all classes there and you get multiple results afterwards.
So basically again two codes it's one code. So you call it once with all pipeline of tokens and you
let write the scripts to get back weights and then you get this weights and basically get
resulted from example it's Spanners. It's okay. Yeah so you can you can for sure do this so there is
all tooling but for sure it's not like out of the box it's like a brand new feature and it takes
some time to get into let's say mail call. If it's a mail call it gets extensively tested by
many users. So it's good thing. Yeah okay I have not much time to talk about it but one thing that was
a long standard project was to convert fuzzy to from UDP to TCP just basically to dodge
headscanner, port scan, broken logic and in October it was big incident that affected thousands
of users over the world and even I had to write a mail like you break the internet in the
half of the world. Yeah and the idea was basically to switch it to TCP and it was also quite
difficult and so for one side it was simple like let's convert some protocol from UDP to TCP
but in general it was very difficult because you need to think about lots of things like how to switch
like how to fall back what to do if your TCP connection is over how to frame things inside of
TCP stream how to multiply between different connections so it was quite a big project but I hope
in future it will be much much better. I've started this as my own project but I never finished
the client part why because it's it's it's just a mess. You start to implement it and at some point
you just lose all hopes to make it any how usable for production. I think it's a common sense
when you're trying to refactor a big chunk of code and at some point you just give up. Yes
and the final project hopefully I have five minutes left and that's one of the most interesting
because not only because I've implemented it in the most recently but it was a talk in in
the matrix room where my previous experiments about using embeddings for a classification will not
so successful but I built more prototyping around it and what's more important I've revoked
this part so previously it was two layers never let me be without normalization layers and
this I think softmax at the end but now I have dropout layer and normalization layer fixed it from
railroad to yellow change it arosines normalization so a lot of tunes around my own classification
engine and it was very good so basically what we're doing now we get displayed depth part like
if it's a HTML it's multi-part alternative we select a HTML part if it has some larger
HTML attachment and nothing else we also select the attachment if you have all the text and
the select text so basically select text part then we do so called smart three-minute text so for
example if you have a long chain of replies and I've seen messages with megabytes of replies
we trim all quotation we trim all replies the trim all features like with best regards name and
stuff like that so we basically get the meaningful part of text then we get all other possible features
for example like symbols and metatocans and for text we generate embeddings like part of LLM so
it's just a compressor part and instead of text it outputs embeddings so that's the most expensive part
else then we get all features from emails and we can use these features as a normalized
vectors we fuse them all these features together normalize weight them and feed it to local
automatically or manually learnt multi layer network with all this architecture and the results
were very very good actually so the only thing is that LM embeddings are quite expensive so that's
the only drawback but it's worse to consider that not only for classification but also for storing
these tokens for field like full-text search semantic search whatever so tokens are important
and what's more important is that this information is not personal data so you can store them
and train on them and do all things afterwards without content of emails and that's where
a good for privacy protection at least I believe so you can't actually get the message from tokens
you yes so there are some results with it too testing independently with some of our
sponge users so I've used it on my own corpus and that person has used it on his own corpus
so I would say that the numbers are better than bias so bias is a couple of percent worse but what's
more important is that this architecture I would say it's more more scalable and more
interesting for future use and counting that embeddings models I usually move the language
on multi-language case it will definitely outperform bias because bias can't distinguish between
languages in general yes so I will also test it this architecture on CPU and yes GPU is
ten hundred times faster especially if you do some sort of batching especially if you
optimize models and GPU can handle multi-language models and what's more important you can actually
even speed it up more by splitting it by languages so for example if you have mail from one
specific language you can feed it by GPU and leave the rest to whatever you want or just skip it
and this can be quite efficient yes but for personal service I think when this feature comes
again to mail cow or stuff like that you'll be able to test it more extensively on CPU it's still
viable but when I try it on my MacBook I decided to rent GPU just to preserve my cool ups
yeah and one important thing for embeddings is the best table so when you when you get same
message you get same embeddings and yes it's it's much easier to deploy a part of embeddings
network land full oh and and I think that's it
so if you have any questions
what do you mean yeah maybe next year I won't be giving presentation that's my so for example
this one the slides were created again by Claude so it started Apple script to build the slides and
key note because that's the slides creation tool that I'm using and all diagrams are also kind of
created by a lot of like types into diagrams that are presented here maybe next year I won't need to talk
at all so maybe we have like last year of human human communications I don't know
and other questions so in bias it's not it's bias it's the same kind of mathematical
operations so when you reload you reduce weights of tokens in all other classes and increase weights
of all tokens in one class so basically that's really learning and then if you adjust probabilities
and operate on feel the pipeline of bias and classifier it will move to another class and then you
just rely on smart exploration to remove all tokens and remove insignificant tokens and stuff like that
but between different classifiers if that was question between spam and transaction for
example we can't move message because for sure it will be a no-op classifiers are self-isolated
between classifier so classifier produces results and if you have two classifiers you have two results