So, this is my talk on Brown and which is a new CWL implementation that uses Geeks.
CWL is the common workflow language.
It is used in running scientific workflows on HP systems for data analysis.
I will say more on that later.
So, my problem is that I am an academic scientist and I have mountains of data to analyze.
In my case, this is usually genomic data.
And when I say mountains, it is easily some fraction of a terabyte or maybe even a few terabytes.
So, the HPC, if you are not familiar, this still uses batch scheduler.
So, you have a bunch of compute notes and you have some node on which you submit jobs.
And there is a queue and it gets scheduled on the compute notes at some point of time when you get high enough into the priority.
So, it is a very traditional, maybe even antiquated system of computation, but that is still the way it is done.
And each of these jobs that you submit, each job is typically just a shell script, a batch script.
And each of these jobs can take hours, maybe even days to compute.
So, you really do not want to repeat those jobs if you can help it.
So, some cashing should of these job steps is helpful to prevent re-computation.
But the problem is that in the scientific workflows, you might want to tweak parameters that are input to the command set you are running.
Or maybe you want to try a different version of the software tools that you are using.
And or it could even be a more complicated problem where you were well, your tool is the same, but some library exchange or maybe the compiler exchange.
So, it is hard to do perfect cashing where you can be sure that your cash is not still, that the results you are pulling out from the cash.
This is actually the results that you want.
So, people typically don't have any cash in, that is the, that is very common to, it is very common for people to stuff in all their parameters into the file names of their results.
And, you know, things like final, final, any equal to one and then, usually there are many more parameters.
So, stuff like 10 parameters into a file name and then it is very hard to do that consistently even if you are the only person doing the data analysis.
So, this is a problem.
So, about the common workflow language, I use something called the common workflow language to describe the jobs that I want to submit to the HPC.
So, the common workflow language is one of many workflow languages, but some of the special features of the CWLR that it is, it is standards based.
That is a YAML, it is a YAML specification.
So, it is easy to, it is very machine readable and it is kind of easy to write a few.
At least it is easy to generate from programming language.
So, I can show you a typical workflow looks like.
So, this is a really simple workflow for spell checking.
I mean, it is not really a scientific workflow, it is just a tar example I am showing you.
So, the blue boxes you see on the top, they are the inputs and the blue box at the end is output.
And these are all the steps each one of these yellow blocks in between is one step of the computation.
So, one step corresponds to one job that runs on the HPC.
And these steps can be connected in any graph like form.
So, it does not have to be linear, it can have these parallel paths that come together at some point.
It is a DAG.
Yeah, so here is a more realistic workflow, this is actually a workflow that I use it, you said work.
So, and I will be showing a demo of this later today.
So, it is very similar to the spell check workflow, it has it has inputs, outputs and steps in between.
Just the tools that are used in between are actual bioinformatics tools.
So, I will come back to the slides.
So, the HPCWL is a standard based and it is a strongly typed language.
Each one of the inputs and outputs that I showed you in the picture, there is a type attached to it.
And each one of the steps are set and types of inputs and gives you certain types of outputs.
So, you can type check all these, you can type check the entire workflow even before you execute it.
So, that is CWL. So, how does Robin and which is my CWL implementation?
How does it actually run CWL and how does it use geeks?
So, I mentioned earlier that the cache-go-in-stale is a problem.
And one good way to solve that is to use geeks.
So, geeks lets you, so if a familiar geeks and so every binary in the store is attached to the long hash that that encodes all the dependencies that were used in building that binary.
So, essentially that hash gives you a unique identifier for that tool.
And in geeks you can use geeks expressions and a concert called program file to generate a unique script with the hash.
That is, that has this specific, you know, scientific commands that you want to run.
So, so, once you have that hash, you can build a cache which has those hashes and that cache is guaranteed to never go stale.
So, let us get on to a demo and maybe things should become clearer.
So, yeah, so I have this CWL file which is the workflow that I am going to execute.
So, this is just the, this workflow, but I am showing you the actual YAML code that describes the workflow.
You have inputs, outputs and then various steps.
So, all that is just CWL and I don't want to go into the details of CWL right now.
Then let us look at and this is an inputs file.
So, I showed you in the picture that there are three inputs, chromosome sequences and threads.
So, all those inputs are defined here as a simple JSON file.
So, sequence is sequence labels, chromosome and so on.
So, now let me actually execute it.
So, let us time it as well.
So, Ravnan, PNG nom.CWL and specify the inputs files.
And it also takes a store argument.
The store in this case is not the weak store.
It is separate, it is basically the cache that Ravnan uses.
I just borrowed the geeks terminology here.
So, let us run this.
So, you can see that it is executing the workflow step by step.
And yeah.
So, it says output is another JSON file which is JSON tree.
Which contains the output file in this case, which is dot path file, which is a sequence thing.
By informatics sequence thing.
So, it took 30 seconds to run this.
This is a, I made the sequence really short.
So, I could show you a demo.
So, let us run it again.
And now this time it should just pull straight out of the cache and without having to recompute.
Yeah, 8 seconds.
So, you can see here that it says that this script has been previously run.
Just retrieving the cell from the store.
And you have the same output.
So, that is basically all there is to it.
It is a matter of combining a CWL with geeks to get this to get a perfect cache.
And now each time you want to generate those results.
You don't have to remember what parameters you put into the workflow.
You can just, you know, ask the system to produce the results.
And if it is already done, it will just keep to you.
If it is not it will be completed for you.
So, you may be familiar with this problem.
And using make, like make is all nice and good.
But if you change the make file or if you change something else, you might have to do a make clean
and then rebuild everything all over again.
So, essentially defeating the purpose of make.
So, this is just like that, but it is like a perfect make.
So, that is it.
Thank you very much.
So, any questions?
Yeah.
How does it relate to the CWL language like in the new project like,
which is an official page about CWL?
So, CWL is a standard and that is separate.
It is got nothing to do with geeks or ignore or anything else.
It is a standard workflow language.
I just implemented the spaceification.
Yeah.
And how does it compare to?
Because there is a, then the geeks project for the CWL implementation.
So, you mean other implementations of CWL, right?
Like CWL too.
Yeah.
There are several CWL implementations.
CWL is a reference implementation.
Then there is a style and a reward dose and so on.
But none of them have a perfect cache.
So, CWL gives you some caching, but, but if you, if you like,
all of the workflow in some way, you essentially have to throw the entire cache and
Recomplete everything.
Yeah.
Questions?
I suppose you are going to be integrating this with things,
which hold dates from other sources.
You have your geeks forged projects.
Are you sort of looking at ways of collecting dates and all of the transmitting, right?
Oh, geeks for this, it is a very, almost time to listen.
15 minutes.
Okay.
I can show you CWL.
Yeah.
But that is a different topic.
Yeah, this was meant to be a very quick lightning talk.
Yeah.
So, I was saying that it is easy to generate Yamal from any programming language.
And so, the CWL workflows are supposed to be written as a
CML files and I really don't like writing Yamal files.
Even though something will do like it.
I don't know.
So, I have a different project, a different, so I built this domain
specific language that generates the Yamal files.
So, this is where I built on Gile.
So, it looks very schemy with expressions and all that.
And I describe the inputs, the command that is running, which is this tool called
Samtools.
And then there is an output file.
And in the other field, I specify the packages that are required for that step.
So, packages in this case are the geeks packages.
So, you have a bunch of steps or commands like that.
And then they are all connected together in a workflow description, which is what you see
then.
So, I used to both them together.
But it is really up to you.
If you really like writing Yamal, you can just write Yamal and use
your own and then the geeks part.
So, there are actually people who like writing Yamal, which is a bit strange, yeah.
Yeah, it's starting to have to cover 15 minutes, so.
There are no questions we could just end earlier.
Yeah, you have a full CCWL talk online also, right?
Yeah, yeah.
I have a CCWL talk in the first time.
Yeah.
I have just thought about the question.
But the question about that was from Murray.
Your link, why do you want to do this?
Sorry.
What kind of computer?
Sorry.
I was going to ask a bit of the big questions.
But I was from Murray.
Do you do by your computing in your job?
Sorry.
Is it just examples for...
Yeah.
I do by your mathematics in my day job.
Okay.
So, do you know my data?
Okay.
Yeah.
Thank you.
Yes.
It's too stretch, right?
Yeah, yeah.
I can't imagine running jobs on HPC without Geeks or without a good cash.
Because usually, by the time you get to the point of writing a paper, it's been months
or maybe even years.
And you don't really remember what it did a long time ago.
And it's, and keeping notes doesn't really work that well.
Yeah.
I wanted to talk more about the controversial, not controversial.
You said at the end, this is just like a better make.
You mean, would you really get rid of make, if you could, to replace it with, like...
Geeks or RNs set up is actually based on hashes.
Yeah.
So, you can't have these sort of conflicts when running on a thousand notes.
And within one second into false.
And people have drops, make make for the reason, because you get all types of notes,
not deterministic behavior.
Yeah, I've seen notes on HPC, which have like,
sometimes people forget to set up your NTPDments on them.
And the time is like 10 minutes, 15 minutes off.
And then strange things happen.
Yeah.
So, you have a question?
No.
You look like you have a question.
Excellent. Thank you, everyone.
Thank you, everyone.