I'll be talking about what's new in SPAC version 1.0, which was released last year, and
also what's to come in the next release, which is actually version 1.2, because 1.1 was released
also in the meantime.
Before I start with that, I'll do a short introduction.
It will be slightly different than usual, and the main point about what's new will be
like your repository split some news about a new installer, and the compilers as dependencies.
My name is Harman Stoppels.
I'm based in Zurich and work as an independent software developer, and I'm involved for about
like 6 years already with SPAC.
You can find me online usually with this avatar, essentially it'll help her.
I want to talk a little bit about what's unique about SPAC, but in the context of Nixon
Geets, because there are very well represented at this conference, and instead of going
for the usual, like SPAC is this flexible package manager, and you can mix a match versions.
I will want to talk a little bit about how it relates to Nixon Geets.
For those who don't know Nick's Geets and SPAC, they're all based on the doctoral teases
of Ilkotolster, and he concludes basically two things in his teases, that one, the best
way to describe a silver installation is the hash of all its built instructions.
And second, the best way, or the best language for that, is something purely functional.
Oh, sorry.
It is actually purpose.
So in this group of package managers, Nick's and Geets, they pride themselves as purely
functional package managers, and they boast high ideals, like purity and reproducibility,
and that's great, like don't say grung.
And that seems to suggest by extension that SPAC being written in this mundane language
Python must be a non-functional package manager, and that is admittedly sometimes true.
But, I would like to make a point to the contrary in this talk, saying that also SPAC
can lead claims to higher ideals, and that gets a bit philosophical.
So I thought I would stay in the realm of this philosophical discussion, and to make my point
talk a little bit about package ontology and philosophical realism, namely Plato's
allegory of the key.
So for those who probably everybody recalls from high school that in this game have a group
of captives sitting with your back to a fire in which objects are passing, and it
has shadows in the walls.
Now in SPAC's contribution to the lexicon of philosophy, those objects are called abstract
SPACs.
So for example, you can think of this the command line where Cp2K depends on Q2 at 14 is
passing by.
The conqueror, it casts a shadow of this installation request and casts it in great detail.
The shadows are called concretizations, and they are basically dependency graphs where everything
is resolved, the package divergence, the variance, the edges between the nodes, and it
would, for example, be Cp2K at a specific version, built with GCC, whatever, with
a very specific version of Q2, etc.
So these shadows can be installed and run, but ultimately, they are a projection.
So if you are in the cave as a captive, and you kind of bump the versions of those shadow
objects, like in other words, you are curating a particular silver stick over time, you
are kind of playing with shadows.
SPAC package mentors, however, they transcend from this cave and they work in the world
of ideals.
So the package recipes they work on are not like a specific piece of software, but they
describe how this over could exist in any possible reality.
So they are like the platonic ideals of that software.
So it captures the essence, and instead of saying, like, I depend on Q2, I would go to,
you say, I depend on Q2.
All right, now to get to the point.
So what's unique about SPAC is that if you consider like a temporal axis, that unlike
Nixon geeks, SPAC is not mutating, but additive, like the entity side by, for example,
and exists beyond time, whereas in Nixon geeks it mutating concept over time.
So in SPAC new versions are added, and all ones can be kept.
And by doing this, we keep multiple crowds happy.
So on the one hand, there are like the conservative people, the ones that install old versions
of software.
On the other hand, there are people who want to be bleeding at you, like rolling release,
or software developers who want to try out the newest versions of their dependencies,
and we can keep these folks happy at the same commit over package repository.
And also, we have things like variants, which allow us to avoid community wars over like
what are the proper defaults for a certain package, how to build it.
So ultimately, we keep various, like in this world of ideals, we keep maintainers with different
interests working together, so that back in those caves, users can cast their shadows, they like.
So speaking of package repository, so what we've done in SPAC version 1.0 is already
a little bit old news, is that we split SPAC the tool from SPAC the packages.
I think also Nixon has done this a long time ago, and this was highly requested, because
first of all, it allows you to bump the package repository in newest without pulling
in the latest books that we have developed in SPAC, and otherwise, sometimes you want
a new feature, but keep your packages at this person commit.
So you can now do this in SPAC, and that was, of course, a nice PR to put up a removing
half a million lines from SPAC the tool.
The glue layer between this is the package API, which we promised to keep stable, and
the package API is basically just like the Python first line of a package that says from
SPAC the package import all, that API is versions, and we hope that not deal with version
bumps, hopefully at all, but, and the API is kind of seventh stone, so if you find something
that regresses, you should hold as a countable.
The second thing that is a bit ongoing, so it was not, it did not make it to the one that
already is directly, but it's a new experimental installer, and it will talk a little bit
about a new job server, the public's job server, and how we used it.
So best show what it looks like, so if you run SPAC install something, you will first
of all notice that now multiple packages are building in parallel, as you can see on the
right-hand side, and there can be many in parallel, in principle, there is just one flag
to control the parallelism, and that's kind of like make dash j, if you're used to make,
and we want to have package parallelism by defaults make it a bit easier.
There's a basic terminal user interface, and you can toggle between like overview of what's
building and the logs of a particular package.
And we hope that in the spring we will have this as a property fault.
A little bit about a public job server, I guess it's, I don't know how well known this is,
but it's an ancient protocol, so from a previous century, and it allows composable parallelism
across processes.
So it's been long supported in GCC, but recently there seems to have been like a remissons
of this protocol, like last year somehow everybody got interested in it again, and I think
it was the motivator was mostly ninja.
So somebody had created a PR for ninja, job server support, nine years ago, and last
year the time was ripe to merge it.
It caused like three or four acts of ninja at a build system, at a build tool.
I cannot go into all details, but there's very interesting blog that kind of shows like
what it took from a container to deal with these types of community requests like, why
can you have a job server?
But it was very useful for ninja because sometimes ninja calls ninja, and it would cause
a very high load on the system if there was no job server support.
Much faster, it went in LVM, it went from issue to implementation within a couple
months.
It's not released yet, but and a spec had like the first implementation also in November.
I think Gen 2 had a blog about it, and the Julia community was considering it in the
issue.
So what is it?
It's basically simplest way to think about it is like you have a server that has
a pouch of coins, and that's actually a pipe, and anytime you take a coin, it means
you read a byte from a pipe, you claim that like I'm doing now something in parallel.
And if you do this properly, you create like this process tree, where if you start with
n-1 bytes in the pipe, or tokens in the pouch, and everybody takes one token for every
parallel job, then the leave notes are just n leave notes, assuming that they do
are like sequentially executing or like using one single thread, one single processor, then
you get it's like an estimate for the load on the system.
The interior notes in a graph are idle, and it made their like truly idle.
So people criticize this protocol, it's super simple, but like one bad apple that arguments
suppose that somebody takes a coin from the pouch, and decides like I will never return
this like to keep it, ultimately the job server has no clue who took that coin, and you
continue your build with limited parallelism, so it will just slow down to build.
Well, first of all, I don't think that's very, it doesn't really happen, we don't see
it happen in practice, and I think the Gen 2 people have also been doing lots of build
with these, and they're still using it, so I don't think that's really common things
happen, otherwise usually the build just stops anyhow with a build failure, for example.
But if you really are scared of implementing a client for your job server, then you can
also just use make, because it has both a server and a client, and that's actually what
the GCC does, so there are a link-time optimization executable, actually generates a
make-fall runtime, and then calls make on it, I don't know if that's great, but it works.
You don't get, like you get the composable parallelism that way, spec used to promote
is before the new installer for power users that you could generate a giant make-fall
or your dependency gravity or about install, and then run make on it and make it basically
like the driver that starts the job server, and you would get the parallel installations
like that.
In practice, it was not great, and that's why we have a new installer, for example, sometimes
you don't need to install all the early dependencies, which is part of the dependency graph,
and then this graph is not as static as you think it is, and there was also a lot of overhead
of running make-to-runs back to install a package of a very short install that was just
too much start-up and shut down overhead of Python.
Other people claimed that this product was not modern enough, and by modern I mean 2008
when the gold linker first introduced like a fretpool in linking.
So for example, if you have a fretpool with many short-lived tasks, then like reading from
a pipe and writing to it, that would be a lot which is purely in C. So people thought
it was not possible to combine this ancient protocol with modern parallelism.
But LVM seems to have pulled it off, so that's a quick look at how they do it, and
finally enough it's being discussed as we speak, so maybe this slide is already not current
as I speak, but basically what they implemented in a first iteration is they create a fretpool,
it does, for example, linking jobs or short-lived tasks, and they start all the threads and
then they block all but one on acquiring a token from a job server, and only when they get
to token, it's going to start doing a parallel work.
It's not yet completely finished, I would say, so I hope it works in the next LVM release,
but we'll see there are some open issues, and actually the Gen2 people were on that,
and I started also commenting on it, so I'm now getting updates about it.
I hope that the next LVM release has a proper implementation.
Last thing, or maybe second to last thing, I want to talk about is compilers as dependencies,
which landed in spec version one.
So I think this was kind of like the catalyst over the point at which you said, like, probably
now it's time to make one that overlays of spec if we have this, and I think it was
actually introduced at or first alluded to by Todd Camlin, at false them in 2018, so
it's like very, very long ago.
In spec 0, not x, we used to have compilers as attributes on a node, so for example, you
would have installed open as LVM 3, and then a compiler would be just like some property
on that node, and that was a bit of a simplification of the real world.
Now we do it like this, open as a cell is a node, the compiler is also a node, and we
even add a couple other nodes that are kind of necessary to make the model work, namely
the runtime libraries as separate nodes in the dependency graph.
So this is what you now typically see if you do spec install something, and I've colored
a couple in green and the other one in gray, so the compiler is now a build dependency in a
principle, you can also uninstall it, and your software would still continue to work,
which wasn't the case in the past with spec.
So how does it work?
In spec, we have now better concept of languages as entities languages like C, C, XF, and
4 trends, where we have right now their kind of like virtual packages, and the compiler
packages, they are providers of these virtuals.
So for example, a compiler could provide 4 trend, and the package depends on 4 trend, and
then our software links these two together.
It can be all conditional, so for example, you can have like an incomplete installation of
C, without a 4 trend compiler, then it just provides C and C, XX.
The compilers as packages, they can inject further dependencies into their parent, and that's
how we deal with G, C, Runtime, like Lip Center, and C, XX, etc., and G, Lip C as dependencies
in the graph.
Now we have some special logic that you can like these runs of their bits, special, like
you can mix different G-Lip C versions, and it's you run time versions in the graph, and
that allows us to do like compiler mixing.
And we try to be nice deal with like sometimes they're like a breaking change in the API
of G4 trend, like Lip C4 trend, and we try to ensure that you don't accidentally get
like half a stack compiled with all G's and other half with new, and there would be
API issues there, so we try to avoid that in this over.
We added some new syntax, and I think this is something that is probably unique to
the package manager right now that you can properly talk about mixed tool chains.
So for example, you could use Clang for C and C++, and GCC for 4 trend, which was like
maybe less so now, but it used to be quite popular combination of compilers, and we can
actually model this.
So on the command line, you can actually say like I want to install this package, and
then the percent means depends on, and then it goes like language equals compiler, or it
is actually more generic, it is virtual equals package.
You can use this syntax everywhere, so we can now, we also have finer granularity, when
you know like this package here, you cannot use a C++ compiler from GCC.
You can specify a conflict like this to add that constraint, and we added some, like you
can specify in config, what your compiler tool chain is, where you can say like you can
shorten this a little bit, so you can, for example, talk about install things with Clang
with G4 trend as a dependency.
I think I still have one minute, so another thing that we're concerned about, or actually
use it or concern about, is that spec was pretty slow, so spec is probably different in
the sense that there is a resolver that is doing what otherwise would be done by a community
pushing like this ecosystem forward, so the server was kind of slow, and also the fact
that we used Python did not make it much faster.
I think I won't let go over all of these, but they're like various targeted PRs in the last
two months to make spec as a command line tool a bit more, a bit easier to use and faster
to respond, and I think, so most of it comes down like trying to make the dimensionality
of the problem we're solving, smaller, so the search is a shallower or, and we're trying
to reduce allocations and that kind of stuff, and we use some new tools that are in Python
free.15 that are actually quite nice, and I can recommend, for example, the new sampling
profiler, it's actually quite nice to figure out where your bottlenecks are, and I think
and with this slide, that like over time you can see that there are some, like, this is the
package repository where we solve our largest set of packages with our solver, and every now and
then we bump the spec version there, and then you see this drop, and we hope to get a few more
crops so that the spec install, it doesn't feel like it's stuck for a minute. I'll end there,
and I'm taking questions from the audience.
Yeah, 3.15 is not released. Oh, the question is, isn't 3.15 Python 3.15 not released yet,
and that's true, but you can already use it. In fact, you could probably spec install it too.
So the question is about compiler bootstrapping that people would spec install
GCC, and then mark GCC installed by spec as an external or, like, as a configured compiler.
So I didn't touch on this, but this is irrelevant in spec1.0, because if you install a compiler,
then next time you install something with spec spec, realizes that, hey, this GCC package is
available in my store, so I can reuse that. So with compilers being ordinary packages,
they should not be very different from, like, if I first install C-make and then install the
package that uses C-make, the solver would also realize, hey, I can reuse this installed C-make.
So the question is, can you use back without, like, any links to the operating system?
I would say we do, like, until the level of the lip C library. So there's, I think, different from,
like, nix and geeks, spec ultimately still has this dependency on the system version
of usually GCC. But otherwise, that's like the only required external package.
Yes?
spec was a 0 version based for the pages, so maybe not a bit close, and you have spec1.0 since
that's similar, isn't she? Have you seen any impact on that on how development is done,
that people need to be more careful to keep that if they are stable, because you are used to having,
let's say, the freedom to create things. And now that you're on the 0, you will, I'm
going to mention that you want to be sure that things are stable. Does it have a specific
impact system of how things are built?
So the question is, whether one that already is, are there any concerns about, like,
are we limiting ourselves with, or is there impact on development? Is there impact on development
has done, now that there is a one that already is? I was actually a bit scared about this, so
notably with defining a package API, and if we did it pretty quickly. So we split the repo,
and then, okay, now we'll see, like, maybe we forgot about all kinds of stuff that we want to
change in spec, and that's now no longer possible. In practice, I would say we did not really see this.
So our day-to-day development on the tool side is not really impacted. Sometimes you have to look
up, like, am I touching public API here? But in practice, we still, like, for example, I was
worried, like, we cannot do performance improvements, but we can. And actually, we just started
with that in one that, oh, because we felt like, okay, now we have something stable. We can do more
targeted performance improvements, for example. So it feels like we're still quite flexible.