Well, welcome, everyone, it's really nice to see how many people have turned up and joined
online, probably.
So my name is Lasto, and I would like to tell you about using Nix for reproducible
by informatics workflows.
I have a pretty unusual setup.
As you can see, I have no R installed, no Python, and also none of the usual bioinformatics
tools that you might have.
So how do I get things done, other than asking my team to do it?
Well, I just enter one of my work folders, and voila, I have R451 available, I go in another
folder, have a different version of R, and if I ever need to compare some sequences,
I can just prepend the comma and blast away.
It hasn't always been like that.
Some years ago, I was a biologist looking for free software to do some statistics and plotting,
to do it properly, so I found R and learned it.
So some scripts, of course, and I realized my scripts started breaking when I upgraded
R or some packages.
So I thought, okay, we need to control this somehow, so I found RM, that is a great software
to do so, more recent than RV.
It controls R and R packages, but not anything else in your system.
So every time I try to run it or restore it on a new computer, then I have to think a little
bit, quite a bit.
So then, for other workflows, I use Docker and CWL, so of course, the question comes naturally,
why can't you just use Docker for your analysis work flows as well.
And it turns out you can, and it actually works really nicely and reproducib bit, Docker images
are reproducible.
The problem there, I think, is that it doesn't scale too well.
And what I mean by that is, if you have just one workflow, it's fine.
If you have multiple tasks that you are trying to do, then you cannot just cram everything
into one Docker image, right?
You need to build some kind of composition of Docker images with base, with R, Python, and
then one for bulk RNA-seek, one for single cell, and one for flow cytometry, for example.
So you can do this with the from statement, of course.
But then you end up still slightly more packages than what you need for each of these.
You cannot create an image for every single project that you have.
And also you have installation of different things, as well, many places in your tree.
So I will still not satisfy with the granularity, and that Docker is not really atomic in that
sense.
So I went further and I found Nick's, which is, I will quickly introduce here.
So it's a package manager that controls everything.
And this is what really drew me to this, is that it's not only R, not only Python, it controls
all the system dependencies and everything.
You have packages for all kinds of software.
You can build software truly reproducibly, and it's isolated, and it's atomic, so everything
I needed.
It did change my life completely the way I work for the better.
So how does it manage packages?
It's a monorepo that has basically software recipes written in the next language.
Each recipe has the source, all is dependencies defined, and how you build that particular
software.
And then of course all the dependencies are also contained in mixed packages, so it's just packages
all the way down to zero.
And you build something, it ends up in the slash Nick's store, so it doesn't interfere
with anything else on your system.
It is also dependent on the unique hash that is calculated based on the recipe itself,
and the dependencies recipe recursively.
So if anything changes, even an upstream package version changes, you get a new instance
of your software, and these different variants can leave next to each other peacefully.
And that's what you saw, like I could have many different versions of our on my computer.
You don't have to build everything yourself, the rescaching built in, so most of the time
you end up just downloading the binary, but sometimes you can also build it yourself.
It is huge, so Nick's package is the biggest repository out there, which is a bit of a cheating,
because it contains all the language-specific packages, including by a conductor and
cron, but as a bi-informatician, I think we don't mind too much about this.
And these are all built with a single Nick's expression, and I think about 80-90% of the
packages work out of the box.
Some needs special attention because of how Nick's is, so it's because of the immutable nature
of Nick's, or that you cannot download things during installation, or you need to point out some
system dependencies.
So for that, we have a pretty nice community, you are welcome to join the Matrix Channel.
Around our bi-oconductor really cycles, we get together and hack away at night to make
those new packages work again, but the ones that are breaking, of course.
And if you want something, if you are not happy with the really cycles, you want something
bleeding edge, then you can check out Bruno Rodriguez's, our stats on Nick's, where he has
daily snapshots.
Right.
So if you are not yet into Nick's, even after today's talk, you can do this also with
purely with R. There is the rig's package for you. You can just use a function, define
the R version, the packages that you need, and you can have also system packages, ID's,
you name it.
You can also convert an existing RM into an next expression, and so this will just
speed out an next expression for you. It's a file that you can then check in to get
for example, and with Nick's shell, you just enter and you have your environment that you
defined.
Yeah, I could talk a little bit about workflows, but I think for the sake of time, maybe
I will just quickly mention, you can manage workflows, there are two initiatives that I know
about, bionics and rig's press that you can try to, a similar to targets, for example,
or next floor.
But what I would like to talk a little bit more about is Docker, because I might have given
the expression that Docker and Nick's are somehow mutually exclusive, but they do work really
well together, actually.
And this you can do in two ways, you can either define Docker image environment in an
next expression that you see here, for example, build layer image, and then you add the software,
you want in your image, and then you will get a Docker image as a product.
This is more reproducible than Docker files, which are not reproducible.
So I like to use it to make my Docker images.
It also produces quite slim images, so it's less at a surface, right?
Another way that I found useful, if you are, let's say in your workplace, you are tied to virtual
machines or containers, you must use this, then you can just install Nick's on top.
Now you have a single image to maintain, and then Nick's can pull in the necessary dependencies
for each of your tasks, right, that you, for each task, you can define a shadow-snips file,
you define what you need for that task, and Nick's will take care of it.
So let's meet the non-DocardMazes anymore.
So what are the benefits of using it?
I like the fact that I can rest assured, I'm going much faster than before, that you can
rest assured that you will have the environment that you find, and you will have it five years
later, or 10 years later, you can also share your environment with your colleagues, and
you can be sure that they can run your code, CICD pipelines or containers equally.
And that is because you can store your environment as part of your code and fire it up anywhere.
I also enjoyed being part of the ARP packages maintainer community, so I've learned a lot
from them as they are quite knowledgeable in the Nick space.
But I can say also is that learning Nick's is an experience, it's a journey, it's not,
I can compare it to the experience when I first installed Linux, so you kind of realize
there is something else out there, something very interesting and very cool, and it changes
the way you think about software.
So with that, thank you for your attention, and looking forward to these questions.
Thank you.
We have time for two questions, but I would like to make a small comparison, because Nick's
is not the only player in the game, and we also have Nick's, and afterwards later on we
have a small lighting talk about Greeks, so it would be nice to compare.
We would have loved to have together, but you could have.
So issue of questions, and by the way, if the next speaker could get in line, so you can
just prepare the next talk.
I have any questions?
So the question is what was the original use of Nick's?
Actually, I was surprised to find out that Nick's is really old, so I think it's about
20 years old now, which means it predates Docker, but it's never been that popular,
I think now I see a bit of an uptick in popularity.
What was, I think, the original purpose was reproducibility, so how to build software in
a reproducible way, and actually you can use it not only for software, but you can
also, any kind of digital product, let's say.
But at the original purpose, I think it was that, am I correct?
So you say that one of the problems of using Docker images is that you have to have an
environment with the many dependencies, many different tools, but what if you will use, for
instance, it seems like by a containers and you type to optimize your processes, it's
like what we're doing in a port, that we have a module that runs a single pull or a couple
of tools, but not type to have everything, I see it in a block area in it, and then you
don't rely in the community to have this image, I don't want anything in this make make
is the same approach, so why didn't you see Nick's one days already at Nick communities
you've seen is the game?
Okay.
Can you repeat the question?
Yeah, absolutely.
So the question is, I tried to summarize the question, I think it was, why not use publicly
available images for single workflow steps, right, containers that are defining environments
for workflow steps.
So I think as long as I was working with workflows, like for example, I don't know
are any sick mapping, I read mapping, I would say, it was okay because then you can really
just pass it on, when I started to feel the need for Nick's was actually more downstream
analytics, so when you are using various arpecages for different tasks, and yeah, you
have to build the container with R in it, and then this set of packages and another container
with another set of packages, and yeah for each project you need just a little bit different
environment, so that's where I started using it, but yes, and actually you can, I could
see it working really well together with NF Core or Galaxy, if it would support Nick's
defined environment, right, in the meantime, we can build a Docker containers with Nick's
as well.