So, hi everybody, I'm here to talk about my NYXOS Powered Home Lab.
This is, I was in a really large auditorium and I was like, wow, I did not think that
this many people were going to be interested in NYX, but there's actually a lot of people
here too, so this is awesome.
Okay, so first of all, I'll tell you what my path is in NYXOS, because I probably took like
a difficult path, and I'll go through this part quickly so we can talk about the interesting
stuff with what time I have left.
I think probably a lot of people start with NYX and NYXOS, probably with like a NYX shell,
because you are dealing with packaged dependents like conflict dependencies and your dependencies.
I ran into that, home manager, if you're trying to like consolidate your dot piles, maybe
as another popular entry point into the NYXOS system, I don't really have opinions about
flakes, but I like didn't do this, right, like maybe this is something I actually do a lot
now, because I work at a company that does click how stuff, you know what I was trying to
tell you about.
So what I do is all the time now, but on my first day with NYXOS, I was like, no, I need
to solve like all these problems, all at once, and yeah, if you want to hear more about
click else, I'll be talking about that later, so the question I was reading with is like,
I want templated VMs for my home lab, and I don't want to have to store multi gigabyte
images for all these templates that I have, and I want to be able to have a lot of them,
and I want to be able to version them, and like use text-based definitions like I'm used
to with like a Docker file, or a Docker composer, or something like that.
I wanted to share parameters between different VMs, and also between the different layers
of my VMs, right, like there's some information in my workloads, and I was using Ansible
to configure Prometheus to monitor my hosts, but maybe I wanted some information about what
ports were exposed by those workloads, or which ones were running in Docker, and maybe
I had C Advisor, and I wanted to get that information into my Prometheus config, and with
Ansible, that was just kind of way too much work, right, it was very difficult.
So I came across, and I saw this sort of the solution, and a way that let me define
my VM images using something similar to a Docker file, and the other challenge with Ansible
way is like keeping track of like, why have I actually applied to each different VM, right?
You have these plays, and they're not as reproducible as we would like, right, because
they're dealing with the state on the target host, and there's just all these things,
right, they're making it so much kind of jiffy-hote.
And Telephone doesn't really fit like a home-ribed context, right, like that's for provisioning
infrastructure, not for configuring hosts.
So why next to us, if first me to use get-ups, it was self-documenting, right, but the next
us, configuration, sub-documenting, and it was portable, reproducible, like much more
so than a multi-gega-by-image, and it was using familiar tools under the hood, where
like I could see that I could use OCI containers, I could use system D services, so these
are things that I'm familiar with, and it was just a nice, declarative way to interact
with them.
And why do I love NICS?
Well, like I mentioned here, it's one language, I have one object tree for everything,
so anything that's a parameter anywhere, I can use it somewhere else, no matter what part
of the layer, what part of the stack it's in.
I will functional languages, so NICS just kind of made sense to me, I don't think that's
true for everyone, right?
Like I think some people just kind of think in that way, where you look at a pure function
and it makes sense, and some people like, no, I want more steep.
And then finally, the NICS module system is probably my absolute favorite thing in NICS.
So for those of you who are not familiar with it, like NICS modules is what's used in
home manager and NICSOS, and it's probably when you've written your modules, maybe
without even knowing it.
But what I love about it is how it lets you override things, right?
So every other configuration tool that I've used, whether it's like how am I Ansible
or what have you, right?
When you're trying to make versions of things or like make an instance of a templated
thing, it's always end up in this challenging situation, or it's like, I can't
override the specific thing that I need to override without going up to the thing that
I'm like, depending on, and or if you do want to override things, you end up having to
be really, really verbose.
Like, I'm going to override this file, this config file, so I actually need to recreate
the whole thing, even though I only wanted to change one value in it.
Or you have to, like, in careers you used something like customized to do that, like, it's
challenging.
NICS module is awesome because it all just gets murdered, right?
Because it's just a tree of configuration options that all gets merged at the end, so
you can override anything anywhere, and you can specifically override just the thing that
you need to override.
Okay, so I'm going to quickly talk about how we use NICSOS Generators to create VM images.
I'm going to talk about some of the NICS configuration, NICS configuration that specifically
is useful for VMs.
I'm going to talk about how I use NICS, a NICS-flake to manage multiple hosts, where I can
just drop and you can dig into a folder and then apply it.
And then finally, we'll talk a little bit about just, like, how I manage my workloads and
services.
Maybe it'll give you some ideas.
Yeah.
So, okay.
NICSOS Generators.
So, and anyone here familiar with this command NICSOS Generator, yeah, okay.
So you can, you can give it a target for that.
Here, I'm using PROXMAX, and actually the output of this is going to be PROXMAX back
up now, that you upload to the, there's a directory that PROXMAX expects its backups
to be in.
It's like, bar of using something.
So, STP, the output of this, which is going to be, uh, simlinked in the folder that you
run this command in, and also in your next store.
Um, and you get this image that you can then restore a backup fund.
And you can use really any complete configuration that next to that.
So, I have kind of a base template that I use, and most of all has, like, it enables
to remove updates, which we'll go through, and it has my SSH key for my default user.
And then I can provision as many VMs as I want using that image, and then use NICSOS rebuild
and target host to update those with the specific configuration that they need, for whatever
that VM is supposed to do.
So, let's talk about the VM specific configuration.
Um, everyone for all of module's path, and anyone who's not module's path is especially
variable that you get in an XOS configuration that points to the actual path, but, like,
in NICS packages, we're all of the modules are.
So, if you look at that git repo, in NICS package, right, and if you look at the NICS packages
git repo, go to the modules directory, and then go to the profiles directory.
You'll see there's actually a bunch of different profiles for different situations, like
there's an LXC profile, um, there's, I think, in ESXi1, um, and since ProxMox, under
hood is using KVM and Cuno, we're going to use the Cuno, um, guest profile.
This basically just includes all the hardware configuration.
There's basically replaces your hardware configuration file for VM, for the most part.
You still need to define some volumes and things like that, but all of you drivers and
everything.
You get with this.
And then we can just, like, turn on the Cuno guest service.
This is nice when you're running in ProxMox, or any other Cuno hypervisor, because it, um,
most importantly, there'll be a lot of safe shutdowns, right?
So, that's really nice.
It also reports some metrics in state and other things like that, like the IP addresses and
stuff that's useful to see from the hypervisor.
Perfect.
So, uh, what goes in our VM specific configuration?
Uh, we want to enable the bootloader.
That's nice, because then we can roll back to the previous config.
I don't like to have a lot of extra volumes in my VMs.
I like everything to just be on one volume.
So I use this, uh, node V, setting for the bootloader.
So it's just on the boot drive.
It's not on a separate volume or device or anything like that.
If I are, yeah, let's just keep it simple, right?
This is the most important thing to put in all of your VM configs.
Uh, I leave this out sometimes and it always bites me.
Uh, what this does is automatically resizes the partition when you reboot the VM.
So if you resize the volume underline the partition, right, next to us, we'll see that.
And when you boot up, and it'll take up the space that you gave it.
Um, I tend to start some VMs right with very small volumes, like eight, ten gigabytes.
And then, uh, if you're using it as a next build host, that can get filled up and
really, really fast.
If you're not garbage collecting the next store, right?
So just like always keep that on as a safety net.
And case you need to increase the size of the volume and recover from something.
Um, this is the other more important part that goes in all of my next configs.
This is, uh, what allows remote updates.
Um, so this we were saying is like, what users other than route apply updates remotely.
So that we can use just a regular pseudo user.
And then we're just enabling the X command and flakes, which is useful for remote updates.
Once we've got that enabled, we can do this.
Um, this is, this is completely replaced like 90% of my Ansible commands.
It's, it's now just this right, update and it's config and apply it.
Okay, please get to something maybe a little more unique, not directly from the documentation.
How do I manage the manager of hosts, right?
Like that, at one point I had, you know, maybe 20 or 30 mixed OSBMs running, um, I've
been simplified, but it, it's a lot to manage all of that.
And, um, you don't necessarily want like a flake for each one, especially if all the hosts are
kind of doing the same thing or based on the same template.
Um, so I had this in my, in my primary flake.
And we're basically just iterating over every configuration file in a directory and turning
it into, uh, an XOS configuration that we can drop into our flake outputs under the
XOS configuration's key and, uh, then you can just drop any new file into that top level
directory and get a new target of the new flake target.
Um, speaking of flakes, right, this, I originally, it was kind of following some of the
patterns that I see in other people's like that files, repos and there are some really
popular people, uh, people's configs right.
And they tend to have this structure where you have like your user's directory, your modules
directory, your host directory, and they're all importing from each other.
Now that's great.
I still use that pattern, like, for Nick Darwin on my laptop, because I like to have
everything in one repo for my actual work machine.
But when I was managing a bunch of VMs, this started to get unworthy because I wouldn't
know, I would want to update maybe some file that was being depended on by all of these
other files and, uh, it would create breaking changes, or I would lose track of, you know,
which version of what I had used where.
So the way that I deal with this now is I've split things out.
My dot file, secret to users, that also contains like, got repositoring, each one of these
circles is separate repository, so that repository also contains, I can think for my
natural work stations.
But then I have a separate repository that has sort of my reusable modules and templates
that I might import into a server specific config.
And then I have a whole bunch of repositories that contain host configs.
And all of those import from the modules repository, some of those import my users and secrets,
so that I can, right, use those in the host configs.
But I try to keep these, um, separated by use case or domain in my life, there are a lot of
different things that I do, a lot of different hats that I wear.
And so, uh, wait, a repo per collection of hosts will call it.
And then by importing those other two things as flakes, it allows me to version it.
So I don't, I'm not, if I, if I update one of those upstream things, I'm not forced to update
all of my hosts at the same time, I can do it, you know, whenever I have some other
thing that I need to change.
Um, I was launching the modules, I think they son, I hope people were able to enhance
everyone knows what this does.
But they, yeah, just, just use these liberally.
They're awesome.
Make default.
So you can write a thing, make force, allows you overwrite a thing that wasn't made
default.
Um, one important note, as you sort of get a proliferation of mixed configs, and I didn't
do this at first, I kind of had system.state version in my templates.
Now I still keep it in my templates because that makes them complete and I can build
them and test them, which is nice.
So I also, like, do explicitly override this in every host specific config, otherwise you
will break things when you update a template and forgot that host was using an older version
of system C, um, and, and also, yeah, commit your configs, um, the next source generate
command that we looked at on like the first slide, it does not put the configuration.nix
into the image that it builds or the outbreak, which I will for you choose, it's not going
to put that configuration that was used to generate it into the output.
So it's on you to save that somehow, like, in your, in your, um, get repository.
So let's talk about workloads and how we get workloads into these reposts, uh, services,
right?
Um, this is just a really simple example of how we are, um, how we can use the module
system right and use the, this make enable option, um, to create modules, so rather than
having, like, rather than copy pasting stuff between nix configs, you can just throw
anything that you ever do into a nix module's repo like they showed, and then you can
make it so that it can be toggled on or off.
And then on all of your host, you just import that module's directory and toggle on the
things that you want to be on.
Um, so, okay, so this really saves you from, from having to keep track of, like, what
code gets copied.
Where?
Because you just, it's just, it's just configuration, where it, like, you're just getting
back to that same simple way.
So, uh, where's the, uh, uh, uh, uh, uh, uh, we saw the services that you knew that enabled
earlier, right?
This allows you to, like, create something that simple for your own custom defined services.
Um, and you've been available, where you can see at the top, I've got, um, some ports
that I might allow myself to configure in the future.
But I just, I sort of wasted those in case I wanted to make those configurable.
Um, and we knew, use similar concept for Docker services, right?
So, um, here's a Docker stack that, um, uses that make enable option, and then, if
it's turned on, right, we're going to, oh, this code got kind of, garbled.
Um, sorry, that's hard to read.
I'll, I'll upload a get repo with, with better examples.
Um, but yeah, this is similar if you step the way enabling portiner with some specific
options, right, and then we're actually defining it as an OCI container that then gets
enabled.
Uh, yeah, that's an option.
Okay.
And then I, he said, uh, where I'm actually using tail skill now, so it's where I go
under the hood.
Um, if you do one user record, there's some documentation on the next, the next was website
rate that shows how to configure it.
It's a lot.
It's actually like, a hundred or so lines.
So I would recommend definitely like just copy that into a mix module and make it as easy
as this, right?
So if this is dot tail skill, about enable equals true, you can make your own little
services dot wire guard dot enable equals true.
Um, tail skill has a, this is another thing that just demonstrates the power of
excellence, right?
Tail skill has, um, pretty extensive documentation for how to set up like a router, right?
So this is a thing that basically allows anything on the subnet that this host is on,
to be advertised to the rest of my tail skill network or a wire guard network.
Um, it's this, that, right?
Like what all that can think that there is depending on, you know, how you're networking
is set up.
Well, in mix OS, you can ignore all of the tail skill documentation and just say, use
routing features, both, which means that it's going forward.
And it's going to accept, um, advertised routes from other posts.
Okay.
So real quickly, in one minute, lessons learned, um, I guess that I used to have a lot
of hosts.
And one of the things I kind of learned along the way was, um, like, I needed to build
all of these things for myself, right?
And, uh, part of what I was trying to do was avoid using Kubernetes, but it turned
out I accidentally built myself a Kubernetes.
Um, so keep it simple, since dramatically reduced the number of hosts, just so that
I don't have to deal with that much proliferation.
And I'm, um, for, for actual production workloads, I'm actually moving away from VMs and
just running the XOS on their metal and keeping it simpler and not worrying about things
like high availability because it turns out that the biggest downs, the biggest cause of
downtime in my home lab is me messing with stuff.
So the more I can separate things out into hosts that I just don't touch and they just
do their job better, uh, and that gives me higher availability than a true replicated
setup.
Um, so yeah, like, you can use proxbox VMs.
I think there's a great one you need to do a quick experiment, um, Nixon, microBMs are
great for a quick test of something, um, and containers are great for workloads when you
don't actually care about security or segmentation, right, because they're a lot later
weight.
Microwave VMs in Nick are also pretty lightweight, um, I actually do this when you have
really tight coupling between the layers, when you're doing this stuff that you don't
want to update often like a Raspberry Pi that's like sitting in a closet, this is my home
lab version of an edge, right?
Um, when you want to do really quick experiments, right, at this point, I've sort of developed
like a whole host of templates and modules and things that I can really quickly turn
on or off so that I can quickly test something against another tool, and, uh, you know,
for a family of VMs, we're just like, I just need this thing to do with thing real quick.
These are all great use cases for this, um, yeah, I don't think I've got time for questions,
but come to the usacan.io, booth.
So yeah, so TASKA needs a secret in order to get started.
Um, so I have an Ansible Play that starts TASKA with my secret, but I'm still using, but
So you could also use something like age next and actually encrypt the secret way in your next
effect.