So, this is a short version of a talk that I usually give about my journey from old school
cloud developer.
I contributed a lot of code to open stack, went to Kubernetes, yeah, yeah, yeah, okay.
Recently I got bitten by the molecular biology bug and then that started in bioinformatics
and doing a master of research in London recently, right?
Of course, I don't pretend or expect to become a common expert in biology, but I like
to bring old expertise I have in building clouds and stuff like that into this domain, right?
So the project I did as part of this is what I'm going to highlight very quickly here.
To begin with, we have two problems, which are, let's say, slightly relevant for this room,
but it just gives an idea about what we're trying to achieve.
So one of them is antigen specificity, meaning given an antibody sequence, will it bind
to a given antigen?
Just for simplicity, for example, the SARS-CoV-2 spike protein, that all of us are for
25 million with from the pandemic, and it's a binary classification problem in machine learning
terms, so meaning like, is it going to bind, yes, no, okay, in biological terms, it's not binary,
but we simplify here.
The other one is the school part of predictions, so given the whole antibody, which positions
in that sequence of amino acids, right, are going to bind with the antigen, meaning
the form body that we are trying to prevent to cause a trouble, right?
And this one is a token classification problem, meaning that given a sequence of tokens,
in this case, one for each amino acid, which one of them has that positive level or negative
label, okay?
It's, in doing this, we are comparing a large number of, pretty large number of existing
models that we're going to fine tune for this particular task, okay?
Well, you know, that resulted in around 600 tasks, okay?
For both sequence and token classification, most of them requiring GPUs, so it's not something
that you'll just run on your laptop, it's not that it requires a good amount of processing
power, and that it requires also a good amount of orchestration, because all those tasks
are changed, right, some of them, parallelized, of course, and we require a lot of attention
in the ordering which random, so if you do it manually, besides taking forever, there is
a very, very high chance that you're introducing human error.
So both in terms of repeatability, but also reproducibility, about the researchers, it becomes
a nightmare.
So I went out and looked, hey, what can I use to automate all these things?
And I looked, of course, for directed, acically graph solutions, which are very common, right,
two very common ones out there are, for example, Apache workflow, that they used here,
and another one, very common in the scientific world, is next law.
Who heard about Apache workflow, to now?
How many of you heard about next law?
Okay, so both of them are very popular.
And then, of course, that thing is just the orchestration layer.
You need to choose an underlying platform to do that, right?
Just learn, actually, at something which are very common, right, in the scientific world,
in the HPC world.
But coming from open stack Kubernetes and so on, from of course, Kubernetes in the significantly
more natural way to do this, and I wanted to see if we can work very well also for this
type of HPC use cases.
So here it's a quick overview of how the antigen affinity pipeline looks like.
There are two of them, of them, right?
Again, this is a very short version of the talk, so I'm just focusing on one of the two.
But you can see we have to begin with a lot of tasks that are needed to prepare the data set.
So we start with a row file, a row archive file, basically with all the sequences.
We start clustering in them, very important because we don't want to have duplicates,
which in terms of sequences gets also quite complicated, so I'm not going to do the details.
We need to split training of all the additional sets, then we have a completely separated
test data set, which has to be independent from the other for data leakage considerations,
which has to be clustered as well, potentially under sample if needed.
And then we want to have a control in which we simply shuffle all the labels, so we want
to see which are also the prediction we get on that.
And then for each of the models and we have a lot of them, for those of you familiar with
the domain, Antibiot, Antibiot, 2, yes, and 2, and so on, yes, and 2, maybe familiar with you guys
to come from the metagrop.
Some of them are very large models, the biggest one I'm used here has 15 billion parameters.
And for all of them, you repeat the stuff which is in the blue box, the yellow, smaller
boxes are the task which required GPUs.
So the core of that involves finding tuning models, getting for intunes potential ways and
stuff like that, okay.
At the end of it, we have completely separate tasks which are reports for intuninar and
if everything goes well, you get an email with the results.
So the idea is that you start this thing in the evening, takes around six hours on a server
that I used with two, eight hundreds on that, two and a village hundreds.
And by the morning, you get an email, right?
It's also made in a smart way, so that if a task doesn't need to be run, because I already
have a process, for example, that particular part of the pipeline, it doesn't repeat
the whole thing, right?
And the pipeline is made to be run automatically when every have changed in the data
sets, changing the code, and what not.
Now I'm trying to do a live demo, which is something relatively rare in a line in talks,
so this is the Apache Airflow interface, okay.
I take, for example, one of the prediction tasks, and I clear it, okay.
This will tell, if the network works, it will tell Airflow to start riskadowing that
particular tasks, so not all of them, just that one, because it's, we don't have the
time.
That will spin that, of course it will contact Kubernetes, to the operators behind and
execute or have, and Kubernetes will start scheduling a container, right?
The big difference here is that, compared to SLR, for example, Kubernetes doesn't really
have a scheduling solution, which works very well for this type of task, right?
That part of the work is, of course, entirely of loaded in this case to Airflow, which
does a pretty good job with that, keeps on retrying, basically, and scheduling a maximum number
of tasks based on what your configurations are.
It's already finished, and I can look here, so this is a big advantage, for example, compared
to things like next flow, because all the output from my containers are coming out here,
right?
When that is finished, it goes, it goes here, and starts, it sees that it already collected
everything else, so it's going to go through the, through, the next task, which in
both, for example, running all the reports and everything, here, this is the R code,
generating an RMD, and at the end of it, it's going to send an email, okay?
So if all went well, I'm going to have an email here, which just arrived with all my results.
I also simplified it for simplicity, running only two models here instead of all of them.
I click on it, and I get all these nice metrics, you know, which show me comparisons
between all the models, okay?
In this case, against only two models, because simplicity, but this is all the whole thing,
looks like comparing, comparing all of them, you see, all the models on the X-axis,
and the rest, so for example, recall, FPR, F1, average precision, and so on, so it's very
useful enough for running this stuff.
Moving on, in terms of architecture, you have three main components, you have an F-Lome
rest of the PI, your DAX, which are running, you have a schedule and a queue, which runs
on the work, which is a third component.
If you compare it with next flow, next flow will basically be the scalar class worker
part, right?
We saw this thing.
An important consideration is relatively to storage.
We have here every single container, which gets bone, running each individual task, needs
to have access to single share storage.
How do you do that?
Well, in Kubernetes, you have so-called CSI's, you know, the driver's specific for storage,
we need a pick one that has read right many, meaning that you can share the same PVC,
right?
The persistent volume claim, two, all of those containers, which are running in parallel,
so that they can share both the code, the storage, and so on.
Two great examples in that case are a set for production, or NFS for POCs, right?
So, both of them work pretty well, and they solve the problem, of course, set is more.
It's better for production sinkers, use case.
Optimizations, you have a cool them.
Again, here I'm not entering in the details, but I'm using timeslizing for this particular
case.
I'm sharing my two GPUs between all those models.
But you can do meek if you want more security, more multi-tenancy, or you can use
NPS, but it's currently in a experimental stage in the Kubernetes device plugin, right?
And then for scaling, since you want to share the model across multiple GPUs, you want
to have, of course, having phase accelerates, which I'm using from Mmodos, and deep speed,
which basically allows you to split your models across multiple GPUs in across multiple
nodes.
The three degrees there, one, one, two, three, one allows you to do less sharing, but
it's less problematic to configure, and the third one allows you to do more distribution
of the data, but it's more complicated to configure.
But you saw here, uses the level three.
Last but not least, I think that I will explore more in, for the talk in the future, because
I'm currently porting this pipeline, or so to next floor.
You often people ask me, which is better, if lower next floor.
In reality, there is no better option.
It's just that very shortly, one is Python, the one is Ruby, one uses a, let's say, very
opinionated DSL, the other one is just Python code.
One has an easy learning curve, next floor, it's a bit steeper.
Next floor, it's more HP-serient, if you want, but most important, next floor has an extremely
strong, and if community, which is an F core, on the biology side, which is not present
of course, on air flow, and in my opinion, for example, coming also from the open stack
experience, community is where more important than code, if there is a community, you
can fix the code, right?
So that's why I think it's very interesting to port now this pipeline to next floor,
and see what's coming next.
Thank you.
I'm done with this.
We have time for one question.
Yeah, while the next speaker can line up, one question, yes, that's the next step.
Okay, that was a quick question, I should repeat it, am I going to port this after
the next floor, so when F core, yes, that is the plan, okay?
All right, thank you so much, guys.