All right, can everyone hear me well? Perfect. So today we're going to talk about the
end of core community. It's basically community how to make your next-door workflows run
correctly reproducible and easily and hopefully make it easy to write your next-door workflows.
But first of all, what is next flow? Who view knows what next flow is? Can you see
raise a fence? All right, perfect. For those of you that don't know next-door, next-door is
basically a workflow manager. So a language runtime and a community like it says here,
but mainly a workflow language manager, you can use this to change all your bioinformatics
tools right off each other in a parallel way, make your pipe and reproducible and be able
to run it anywhere. So reproducible is the first part I talk about here. This is the most important
part. If you write the next-door pipeline, you want it to be able to run everywhere and have
its creates the same output. It also needs to be portable. So next-door pipelines are heavily
based on container environments and konda. So you can run them almost anywhere without any
actual changes to the software environments you're working. Let me just do a technical adjustment
here. It's a little bit higher. Okay, let's write that. Is this better? Is it better?
Okay, and then the last part, scalable of course, you want to start writing your next-door pipelines
from the local laptop, but my five samples, for example, some very small things, and then you
will have to well want to immediately scale this to like five million samples in the quote, for
example, and this is all possible using next-door. But of course, such a big and scary thing,
is very versatile, and this highlights the need for standardization. Standardization is introduced
by the end of core community. So in of course, it's basically the community of like-minded people
who write next-door pipelines, I want to make pipelines usable for everyone. So this starts, of course,
with a set of standards. We have a whole list of guidelines. I won't go through all of them right
now, or I will need like two or three more talks about this in the future. But to main parts,
are the documentation, the templates. We use a common template for all pipelines, and use of public
containers to make sure that everyone can actually access all the environments during the
pipeline. As the best practice, this is probably the most important one, and then of course,
there are a lot of different best practices. Of course, we allow only a low one pipeline per
data type and or usage. So this means that there isn't any overlap between several pipelines.
There's only one pipeline that can do your actual analysis for you, so you don't have to go looking
at three or four different pipelines to see which one does your analysis best. We also use stable
text to release our pipelines and run it with CICD to actually test our pipelines. And of course,
they are closed already. This is pretty native and next to right about now, but end of course,
really push it through. Cooperation is maybe even more important. We all work together on pipelines,
and there's no single pipeline that is owned by one person. All pipelines are owned by the community.
It's of course open source. Otherwise, we wouldn't be here, I think. And then to make the work
easier, we have a lot of components, which are parts of your next supply plan that you can actually
use and reuse in other pipelines without having to write these components yourself. This really
speeds up the process of writing next to pipelines and reproducible. So to talk a bit more about
these components, we have about 141 pipelines right now. I think 60 to 80 of them are released.
I forgot a number, but something around that. Others are still in development, but all pipelines
are really being worked on. Then modules, we have a thousand and 700 modules. Modules are basically
a wrapers around buying from edict tools. So for example, some tools, few, you have a module for that,
and you can just pull it in your pipeline and start to use it. And around 100 separate flows,
which is basically a chain of modules that you can start using, which is a specific analysis.
We test our pipelines using minimal test data in CI, and also some full tests for some pipelines.
We use the NF test framework for this. I won't go deeper into that, but you can still come
ask me at the end to get some more information about this. We lent all our pipelines to check
for consistency between all the different NF core pipelines to make sure that if you read one
pipeline that you can understand all the other pipelines. We use a schema to validate or inputs
of the pipeline. So the parameters, for example, your sample sheet, where you define each
specific option for each sample, I use the NF schema plugin to do this. And then tooling,
which Julia will talk about more at the end of the stock, can be used to tie the stall together
and to make it even easier for you to write these pipelines. And to use them, of course.
So a little bit of statistics to show you a bit of the scale of the community.
We have around 12.000 slack users right now, as you can see in the graph here,
it's still going up pretty well. And a lot of active users also, so you can say 12.000 slack
users, but we also have a lot of active users of those. We have more than 500 contributors,
which means a contributor, we count as someone who made at least one commit to any of the
NF core repositories. There are 175 plus repositories, 141 of them are the pipelines,
and all the other are more infrastructure related. Then total of all the repositories we have
around 100,000 GitHub commits, which shows you the progress we made since the inception of NF
core in 2019. Over all the repositories combined again, we have almost 8,000 GitHub stars
and 790,000 GitHub views. So to go over a couple of our pipelines, the most important ones,
we have our nasic, which is the basic expression analysis or an API plan.
ATXC, which is the ATXC analysis, speaks for itself, I think.
Simply see if there's some applicants sequencing analysis. I don't know what it is, but
there you go. Then, Sarac, which is closer to my heart, there's the variant analysis of short
read to DNA data using a series of different tools, etc. Then, fetch NES has used to fetch your
metadata or fast queues, so you can create sample sheets to run other pipelines. So this is basically
a start of point for a lot of people to start using NF core pipelines. And then viral recon is mainly
used to analyze viral samples, and this was heavily used during the COVID crisis, so it's always
a nice thing to have. And maybe a special mention for protein fault, which was able present in a
couple of talks from here. So, oh, pay attention. Then, in of course, socials, we have our Slack
GitHub, Macedon, Blue Sky, LinkedIn, and YouTube, where you can follow us, join us, ask us anything
you want, preferably next door, and of course related, but we answer all the questions too.
So to recap a little bit, and of course, it's a community of people that work together on
Excel pipelines. To do this, we need a set of standards and best practices to actually apply
on these pipelines, so everyone knows how to write one pipeline for a array of different subjects.
Then, of course, we need the modules and separate flows to create our pipelines, and some specific
ways to actually run our pipelines. And to do that, we have the NF core tools, which Julia is going
to present now. So, a big round of applause for Julia, please.
Okay, hello. Okay, thank you. So, I will focus on NF core tools, and for this, I
prepared, I wanted to invite you to come along in a small text adventure with me. I promise that this
looked like a nice terminal formatted, but this is what we have. So, please try to not be to this
directed about this. Okay, so, in NF core, we offer tools to help both users and developers.
So, let's start this adventure, imagining that you have a bunch of data that you want to
analyze. For example, if you work in a core facility, in this case, you are a user.
And we offer several comments to help you. So, let's imagine that you need to run your pipeline
in the HPC of your institution, which doesn't have internet connection. This is not a problem,
because you can use the download command that will download the pipeline and all the containers
required to run this pipeline offline. Talking about the HPC in your institution, we also have,
and here I introduce this concept of special repository that I called for the sake of this
presentation, which are repositories that collect resources that can be used in any of the
next flow pipelines. So, the first one, NF core configs. Here we store config files, where you can
create a config file for your institution, and then everyone in the institution can use these
for any of the next flow pipelines. And this will typically have things like, for example, the
maximum GPU CPUs that you can use, maximum memory, or other parameters that you need to specify.
And once you have your pipeline downloaded, you have your institutional config, what you want to
do is tweak a little bit the run for your specific needs. This means setting the input parameters
of the pipeline. For this, we also help you with the great parameters file comment,
which is very useful to store this input parameters, and then you can use it later on, and it's
nice for reproducibility. Okay, so then on the other hand, imagine that you have already analyzed
your data, and maybe you found a small bug. Then you become a developer, and you are following
the developer path. For example, you can, if maybe you found a bug, or maybe you want to use
a different tool for these analysis. In this case, what you will have to do is to create a new
module. And here comes another of these repositories, which is the module's repo, and this one
is where we store the collection of modules and sub-workflow that you can use in any pipeline.
And that's basically a module is just a wrapper around a tool, so you have your software,
you will wrap it around the next language in an informed module, and then these modules,
you connect to create sub-workflow, and the sub-workflow you connect to create pipelines.
So, you are going to create your new module, contribute to the module's repo, and you will
start by with the great comment that will provide you a template, which is basically set of files,
but give you like a skeleton to help you build this module, and you will find different
to-do comments that will guide you through the process. Once everything is ready, you will have
to link your code to make sure that you are following the standards, and then it's easier for
everyone to rebuild your PRs, and maybe also contribute back. And finally, of course, you will
have to add tests to your module to make sure that everything works as expected.
So, once you have your module created, or maybe the tool that you wanted to use already
had a module that in the repo, you want to use this module in your pipeline, so now you go and
contribute to the pipeline's repo. And here you also have a bunch of comments to help you, for
example, you can install the module, you can remove the module, or if you were already using
the module, but you might some changes upstream, you can update it to get these changes.
Okay, then in another hypothetical situation, let's imagine that you tried to find a pipeline that
was suitable for you, but you couldn't find any. So, then you want to create a new pipeline.
The first thing that you will have to do is come and talk to us, and for this we use this repository
called Proposals. You will open an issue there, and there is where we discuss, and we check that
there's not a similar pipeline in the in NF Core, in such case we will enforce collaboration,
but if everything goes well and it's approved, then we will create this repository for your pipeline.
And very similar to the process that we were using for modules, you will start creating
the pipeline from a template that will give you again, like this is skeleton for the pipeline,
and has some, some of the features that we use in NF Core, for example, the CI testing and some
examples. If you are more interested on knowing about this template, we have this recorded by
CI start, when we go through all the files expanding a little bit more, then you will also have
to link your code to make sure it follows the standards, and there are a couple of additional
things in the pipelines. The first of all is this raw create file that will be created automatically.
Raw create is an open source standard that is used basically to pack the pipeline with its
metadata and its structure, and this is used for provenance tracking. Then you will also have
to update the next log schema. This is a schema that follows the JSON schema format, and this is
what we use to define the pipeline parameters, and then we use it to validate the input parameters
on each run of the pipeline, which with the NF schema plugin, and also use it for other things,
for example, to generate a nice documentation in our website. And finally, when you have your
repository in NF Core, you will regularly receive template updates. This means that an automatic
PR is going to be opened in your repo, giving you some updates that we use to deliver,
maybe some bug fixes, or most importantly to keep up with the latest next log developments,
and next log features. And here I wanted to do a quick sneak peek of how these works in case
you are interested. So, basically in the pipeline, we have the main branch where we do the
cyber releases of the pipeline, then we have the development branch, and all the pipelines have
this special branch called template. Whenever there is a release of NF Core tools, we push the changes
to the template branch of all the pipelines, and this is what we use then to deliver the updates.
It is also important to mention that all of this that I have been talking about, it's not only
for NF Core pipelines, but for any of the next log pipelines. So, if you want to have your
pipeline in a private repo, you can still use a template, you can still install NF Core modules,
or for example, receive updates in this case manually with a sync command.
And last but not least, we have this small sidecast, which is that of course, you will want
to add tests to your pipeline, we're using the NF test tool. And here comes the last one of these
special repositories, which is the test data set repository, and here we store some data files
that you can use for testing. And we also provide some commands to help you look into this
repository and see if you can find a file that you can use. So, now you've completed the NF Core
adventure, congratulations, and the only thing left for use to join the community, and just
wanted to make a last summary take home message of all what we provide, which is a pipeline template
validation of input parameters and files, CI testing, LinkedIn configuration, and finally documentation.
But, most importantly, a group of people ready to help thank all over it with you.
And thank you very much, if you want to catch up with us later, we are going to be having
beer, so I just find this.
So, that was perfectly to time. We have a little bit of time for questions, okay, if you'd like.
So, maybe one or two questions anybody, or just post, yep, please go ahead.
Maybe before the questions, we are hosting a hackathon at March 11th to 13th. It's online.
You can everyone can join, feel free to join if you want to contribute to NF Core or anything
next or related. Go ahead.
Yeah, so the question was, if we have a, now it works way better. If we have a process to add
this data to the data set repo, basically you will open, so we have branch for modules,
and a different branch for every pipeline. So, you will open a PR there and try to be organized
and update the rate me with all the required metadata and information. And then someone from
the community reviews your PR.
I'm a bad statistician, we're sort of free to decide your community, but if you're
you open to biostatics, by the way, for example, yes, definitely, do you want to answer?
Okay, yes, so basically NF Core started in the genomics field, so we have a bunch of genomics
pipelines, but now we are expanding and we have one about economics, we have earth science people,
we have probably some examples that I forgot. So, definitely, yes, whatever it can be added.
So, the question was, the question was, if we can, that modules were only used for tools, and how can we actually connect with
reference databases, for example, that was a question, right? Yeah, so next look and actually
has some pretty built-in functions to access data from reference databases, for example, there's also a
possibility to write your own plugins for next look and to make it possible to access this data and
pull it into the tools you actually want to run it on. There are any modules to access with reference
databases, even though it just happens sometimes, it's a bit of a gray area, but we prefer it if you
can just make next all actually pull the data in for you. I'm going to add something here, so
so there are some themes you will spot through this, the program today, and one of them is
aerocreate, next flow in F core, these are becoming standard only for this part of the world,
and I think probably at least half of the world's bi-informatics pipelines and other science pipelines.
So this is only going to get more standardized and more effective, and all of the reference databases
are actually looking at interoperations with the processing so that we can do fully provenance,
reproducible, and reconstructable systems. So yeah, this is just the beginning.
Thank you. Thank you.
So, and I think, okay, this is going to be very quick.
Just going to point one, it turns out that our ship likes to meet the pipeline, who owns it,
that make this a lot of dirt. So the question was, who owns the actual pipelines that you publish
to NF core, and the short answer is everyone, so it's a community building, actual pipelines,
so any pipeline in the NF core, community is actually owned by the whole community. Of course,
there are all of people who are considered to be the lead developers of the pipelines, we have a little
bit more to say about what happens to it, but the main consensus is that everyone owns the pipeline.
So we're going to disconnect your laptop and do the great transfer.
Okay, so I should have a third hand should not. So whilst this is going on,