Yes, already. Hello. Thanks for being here. My name is Ludwig. I'm the founder of Geeks.
I'm the co-maintener of Guy together with Andy. I'm here to talk about the shepherd.
Who in this room has already heard about the shepherd?
Everybody, wow. Who could have thought? Okay.
So I'm going to steal, you know, talk a little bit about what the shepherd is.
Many of you already know, that's great. And I'm going to talk about what it's like inside, which probably is less well known, we'll see.
First of all, did you know that the shepherd started actually a long time ago, 22 years ago, I think?
So that was the actual announcement of the very first pre-pre-release pre-alpha release of the shepherd, which was the name DMD.
And this is the real text of the announcements. So you can see this is coming from someone who really enjoyed the scheme, right?
You can only see the parents straight in the announcement. So this is great.
A lot of things have changed since 2003. So in particular, you know, there was that initial piece of code for DMD.
And then in 2013, we revived the shepherd DMD shepherd. Why do that? You may ask, right?
It was already a little bit silly to, you know, start a new distribution.
Geeks, you know, that geeks things you may have heard about. Why start a new in its system? I get to that in a second.
So that's one of the main events I would say. And the second event is we had a one point or release just in December.
That's quite something, right?
Yeah. So why start that project? I mean, why keep working on an in-it system when we have cool options available?
Well, so that geeks system thing I was talking about, you know, it has a kernel. All right, that's a Linux kernel.
But you know, we like parents. So as soon as possible, we like to have, you know, that initial RAM disk. And of course, we would use scheme inside that initial RAM disk, right?
So what happens next, you know, you need to have so many systems at that point. And this is where the shepherd comes in, right?
PID1. And we obviously also want to have our favorite language in that piece of code.
And then, well, the rest applications, you know, sometimes you have to use applications, not written in guide. That's okay. That happens. But that's the idea, right?
Okay, then more time. This is a scary problem to talk. Please bear with me.
Um, all right. So this is, this is my laptop, okay? I'm running geeks system. And so I'm running to shepherd.
So what does it look like?
Well, if you've ever used system D, for example, it kind of looks similar.
I mean, at the command line level, at least. So you can earn the heard status command. And you see the available services.
There are lots of them. We could have a three-style representation of those services. I agree. I mean, system CTL does a better job here.
We're going to improve on that.
Anyway, you have the list of services, as you would expect.
And from there, you can say, well, what's the status of, let's say, the average demand service?
Okay, and you get pretty much what you would expect. So the, if this status, you see the main PID,
you see the command line, what it depends on, you know, the requires line says it depends on a bunch of services.
Um, you see it's log file, and you see things like things that have a demand world with standard output recently.
So you get an idea. It's not totally happy with the network situation or something.
Anyway, and there's one thing here, the custom action thing, which is more unusual, I would say.
So this service of a demand has a custom action. So you can not only start it, stop it.
You can also use the configuration action. So let me show what it looks like.
So do heard configuration, have a hidden demand.
And it just prints one line, which is the fine name of the configuration file.
So I can, I can actually view the configuration file like this.
Okay, that's the thing.
All right, going back to heard status, we can see that there are different types of services.
And in particular, we have timers, which is a relatively new feature.
So for example, we have a GC timer, which has nothing to do with snippets.
But it does have to do with garbage collection in the context of gigs.
And the log notation timers. So I can query the status of log rotation like this.
Again, I see it's up, et cetera.
And I get information about upcoming timer alarm.
So when it's going to run next, things like that.
Okay, so this was for PID1, my main shepherd instance.
But I also have shepherd running as a user.
So if I type heard status without sudo, I can see my services running as my user.
Because I'm using gigs on them. So these are services running as my users.
Some are stopped, some are timers, some are started.
And one of them is the repal service, that's Winger Bell.
That's to read a wrong print loop service.
And so of course, I can actually connect to my shepherd instance, if I manage shepherd.
Okay, this is this is guy running inside my shepherd instance.
And I can say, please choose the shepherd service module.
And show me the GPG agent service.
This is my GPG agent service.
And I can do things like show me the status of that service.
Here it is, things like that.
It's even more fun to do it with PID1.
I want to do it here. Okay, but you get the idea.
All right, so this is what the shepherd is from a user's perspective.
And of course, you first need to configure it.
So what does the configuration look like?
You have a configuration file, which is actually a scheme code.
Obviously, this is an example where you define the SSHD service.
We're just saying, in this particular case, where we're forking SSHD in demon mode.
It has a PID file.
If it terminates to our leap, please respond it on my behalf.
And then register the service and start it in the background.
And this is my entire configuration file.
Of course, in practice, you have more services that we get the idea.
But I can also say, well, I would like SSHD to be started as a 90D style service.
So if you're old enough, you probably know about I need D otherwise.
Well, it's a thing that allows you to say, please listen for accept connections on these two end points here.
And whenever somebody connects to these end points,
then starts that SSHD program.
And it will take care of the rest.
So that's how it's implemented currently on GIF system.
We have a 90D service.
The good thing is it starts instantaneously when you boot your system.
It doesn't actually spawn SSHD.
And SSHD is only spawn when you actually connect to the machine.
It supports, well, other types of services like timers as we saw before.
So this is an example of a timer, which specify a calendar event thing.
In this case, you're saying, I want this update DB command to learn every day at noon and midnight.
That's an example.
So the calendar event thing specifies constraints on when you want to learn to command.
And I can say I want additional actions like a trigger action to trigger the update DB thingy.
Whenever I want the sudo, heard trigger update DB.
It starts getting interesting when you do other things.
You know, you can do other things and just start processes or listen for incoming connections and stuff like that.
Because this is all scheme, right?
So you can say, well, my start method here is going to do something I want to do.
Which is, for example, to mount a file system.
And this is how I would do it.
So I specify a custom start and a custom stops method that mount and mount a file system.
And you know, sky is the limit.
You can do pretty much anything.
And actually on GIF system, we use it a lot to do.
Lots of different things, you know, beyond just starting demons.
So that's the idea.
The whole thing, you know, that whole idea of how you design the system is about blurring the line between users and developers.
Because in the end, you know, when you start customizing your system, the higher your shepherd configuration file,
then effectively, you are sort of becoming a developer.
You know, there's no clear line here.
And that's the whole point of this design.
All right.
How does this all work internally?
Well, when I started hacking on the shepherd and DMD, I thought, this is very simple, right?
I mean, the thing is just going to start programs.
And then it needs to wait for a sick child.
And when sick child happens, it responds with program.
And that's it, right?
I mean, it's not very complex.
Yeah, bear with me.
I thought I would do the diagrams with TXE, but it's just too complicated.
So yeah.
So it turns out, you need to do that in shepherd, but you need to do more things.
So for example, you want to be able to use that third command to inspect the status of your services,
or to modify them, to start them, stop them.
And so you need, at some point, in the shepherd, to have something like this, where you listen
for incoming connections, you accept incoming connections.
And when you get an incoming connection, then you need to process that connection
in a reader commands that the user sends, like a status command, for example,
and maybe multiple commands.
And at the same time, you need to keep accepting new connections for new,
heard programs, for example.
And there's more than that, actually.
So when you start looking about the details of how you can start a demon,
well, for example, one thing we saw before that, you might have a PID file
that you want to wait for when you start a demon.
So yes, you start by 4x plus xx, you know, to start the program.
And then you have to wait a little bit and see if the PID file comes up, right?
And so you have two solutions in that case.
You have to wait, you specify, you know, some period.
Let's say I will wait for at most 5 seconds.
And I'm checking if the PID file comes up.
And if it comes up, I'm going to the right hand side here, the PID file is here.
So the service is starting, I'm done.
Okay?
But if the PID file does not show up within 5 seconds,
then I'm going to the left hand side.
I don't have PID files, so I must probably do some clean up, like,
terminating the process that I initially started and that was supposed to create that PID file.
And at that point, I should change the status of the service to stop.
So it's getting kind of more complicated than I initially thought.
And I mean, more generally, you know, services might take a bit of time to start.
So it's not just stopped or started running.
You have probably at least two additional states, so starting, you know,
the service might take 5 seconds to start.
That's okay.
And stopping because it might also take a little bit of time to stop itself.
And so you need to keep track of all these for each and every service.
So in the end, you have all these things, and that's only part of the story,
because I'm not even talking about any these type of services here,
like I mentioned for SSHD, I'm not even talking about socket-based activations,
timers, things like that.
So you have all these sort of activities that you need to bring in parallel, right?
And so, you know, usually when facing that kind of situation, you're like,
yeah, I'm going to use threads.
But that's not possible.
We want to be able to use fork.
And, you know, it's actually quite little known,
but you cannot use fork in a reliable way in a multi-threaded program.
That's the thing.
So you need shepherd itself to be single.
And so you need, you still need to have all that concurrency.
It'd be single-threaded.
So how do people do that generally?
I looked at the system deco to get an idea.
It's really in C, but I think that's the same in many programming languages and frameworks.
Essentially, in this case, you have to,
I mean, if you look at the code, this is a code for restarting the service that stops
too early, you know, responding the service.
So you have to, you know, the service stopped.
So you wait a little bit and you restart it.
The code looks like this.
You cannot recognize those loops.
You know, it's totally different.
There's a state machine here.
You're in a function.
You're living the function.
You're going to come back later.
It's very hard to see how the code actually relates to the diagrams I showed before.
In the shepherd, it's very different.
Overall, thanks to fibers.
I get back to that in a minute.
We are able to have code that directly resembles the loops.
I showed before, the sequential pieces of programs.
So for example, this is the code that handles sick child in the shepherd.
It's slightly, but very slightly simplified view.
You can check the actual code.
It's an actual loop, right?
And here, we're just consuming a signal, you know, data coming from a signal,
left-defile descriptor.
And the thing here, so a port in scheme, if you're not familiar,
is a wrapper around the file descriptor, essentially.
And here, we're just reading from that file descriptor.
And you know, it might block because I don't have a sick child available right here and now.
But that's okay because fibers is going to do that corporatively is going to suspend this piece of code
and to save the continuation of that code for later.
And when sick child actually comes in on that file descriptor,
then it's going to resume that computation and that piece of code, right?
This is sort of like, you can think of it like corporative scheduling.
Yeah.
Same story for the, you know, the listen loop, the loop that writes for incoming connections from clients,
like the heard command.
It's just like this, you know, we accept incoming connections
and from there, this accept call unified block forever.
But that's okay because Shepherd is actually going to suspend it and resume the continuation.
When incoming connection actually happens, okay?
And from there, you know, I have that connection and I can spawn a fiber.
That's going to process the client request, the client commands.
Same for the, you know, how you terminate processes.
So typically when you terminate a process in the Shepherd or in any in the system,
you basically want to, you know, send it sick term and hope that the process is going to be kind
and terminate itself properly, you know, within a few seconds.
And if it does not, then at that point you send sick kill to terminate it brutally.
And the thing is this function here is actually quite simple.
I mean, there is no state machine, you know, you just see the entire floor in one function.
And that's a key thing, you know, you write for a message that tells you this process terminated.
And if the message doesn't come in within grace period seconds,
then you know you have to send sick kill.
And this is it and it's very, very everything.
This is wonderful.
I mentioned the states of services.
So individual services have to have those four states possibly.
And so each service actually needs to have something, you know, that carries that state.
And so in fact, each service internally in the Shepherd is associated with a loop like this,
that closes a very little state of the service like its grand status, its value,
and information such as when the service was previously restarted when it changed states,
you know, so you can have debugging information.
And this is pretty much an actor.
I've host in the room, it's familiar with the actor model.
I mean, this is everybody's familiar with the actor model,
because this is pretty people keep talking about it, right?
This is pretty much like an actor, right?
Every service in the Shepherd is essentially that loop that we just saw before.
And we can send a message to it to inspect its state.
For example, when did you switch to the running state or what is your value?
What is the PID of your process, you know, things like that?
We can send message and we can also, well, send message to inspect the state or send message to change the state of the service.
And this is what we have, you know, a whole bunch of actors essentially.
That we can talk to to inspect and modify their state.
This is, I mentioned the actor model, but maybe Christine would be upset if I were too much conflating,
you know, the actor model and what this actually is.
Again, this is using fibers, so it's actually the concurrency credential process is model,
which is not quite the actor model, we'll see there are a few consequences here.
Sweat and tears. I'm presenting, you know, a very beautiful panorama of the landscape of the thing.
It's not always been easy, I won't lie.
So, one thing is fibers is great, you know, because it hides the whole idea behind that you have scheduling and corporation, et cetera.
But there are a few gutches, a few pitfalls.
And so, one of them is, you need to be aware that when you use channels to communicate between fibers,
channels to channels have a get message and put message operation on them, but it's blocking.
So, well, it's not blocking it that you have to run the view.
So, in this example, we have Alice and Bob trying to talk to one another, so it's a fairly common pattern when you use fibers.
Bob has given the channel that Alice is listening to, and Bob is doing, you know, put message to hide to me with the reply channel.
So, that's a fairly common pattern and the idea is that eventually Bob would listen on that reply channel to get the reply.
But in this case, Bob is not a great friend, you know, he's just saying, bye bye, instead of actually, you know, trying to listen on the reply.
So, what happens here is that Alice, when Alice does a put message on that reply channel, there's nobody listening on the other end.
So, Alice will just block there forever, you know, that's kind of a problem.
So, you can easily fall into that kind of trap, and even for example, in cases where, you know, Bob throws an exception because something went wrong.
And so, the consequence Bob will never listen on the reply channel, but Alice doesn't know, and that is still there, you know, trying to send that message to Bob, but Bob is gone, you know, so that's a problem.
So, the lesson here is you must not miss a run the view.
And more generally, you know, shepherd is extensible, so you have situations where people might say, hey, look, I wrote my own service with my own start and stop method, and it's great, but now for some reason, hurt status hangs.
Well, we have a problem.
More generally, you know, for this kind of problem, we'd like to have something like this, so I don't know if you use GDP before and 23 programs.
Yeah, usually when you're in that situation, it's a bad day.
But in GDP, you can do threat apply all BT, and that shows you the backtrace of all the threats, right?
So, you get another view of what's happening in your program, and we don't have that with fibers, so you never really know what your fibers are doing right now.
So, when debugging, you have to be, you know, thoughtful and creative.
And we've had a couple of other issues, so this is this civil-dool guy reporting a bug against fibers, a memory leak, you know, a memory leak, that's kind of a problem.
Actually, that same guy actually implemented the leak before or so.
But we got a second one, a few years later.
This one was a bit less serious, because it only happens with very specific circumstances, but still, you know, a memory leak in PID1, that's like, not great.
So, well, I mean, like any runtime support libraries, I guess you need to take care of it, right?
All right, thanks for your help. So, what's the status of this thing? Well, there's been this Linux magazine headline, which I found summarized the situation pretty well.
System D fix is a bug, while facing a new challenger.
Yeah, so we have one point of open running. It's being used on all of your gigs machines, Gix people.
It's great. I think it's, it's pretty stable. We haven't had any serious issues so far.
It's being better integrated in Gix system and Gix home. So, for example, replacing that old and clunky log rotation service that we had with the Shepherd's log rotation service,
replacing mcrone by Shepherd timers, which are, I think, more convenient and replacing the old styles.
This log D by the Shepherd's, this log D. Fiber's, despite what I said, before it's a solid foundation, any time you want to have, you know, that kind of concurrency.
There are a few pitfalls, but really, really, it's pretty nice because it hides all the complexity. There is no callback hell. It's great.
And it needs love. I'm trying to give it some of my love as you can help.
The Shepherd itself is very small, so it's not a system to replace them because system did so many things these days, right?
But it does, you know, the core functionality of managing services, you know, spawning demons and doing other things.
And in only 7K lines, of course, so I think that's really cool.
And if you're missing anything from the Shepherd, well, let's just go ahead and hack. You can hack in your config file as a start or you can hack into Shepherd itself.
And you're welcome to do so. What about the future? The future is next talk.
Juliana is going to talk about the integration of the Shepherd and Goblin, so I don't have to talk about the future.
And this is it. Thank you.
Any questions?
David.
So the question is, how did we address the run-divou problem in Shepherd itself? Do we have some sort of inbox and outbox?
No, we don't. So we still have synchronous messaging, so the basic channel thing. We just pay attention.
And that's the whole thing. I agree. It's kind of, yeah, you have to be careful to be available any time you're sending or receiving a message.
So essentially, that means you spawn a fiber if you're going to do something that might take a bit of time. That's the thing.
There was a question here. Yes.
Would you say that the fibers are just, do you need it? Just to work around the limitations of the positive stress?
No. So the question is, are fibers used here just as a workaround for the limitation of positive stress and fork?
No, no. I think it's really useful for any time you have concurrent sequential processes, lowercase, as is the case here.
I mean, you have different activities to run concurrently. I was going to say in parallel concurrently.
And so if you want to do that, Shepherd is a great foundation for it. Whether or not you're talking. That's a different story.
And actually fibers let you choose whether you want to have multiple threads, post-extreads in your process or just one. So you can mix and match.
Any other questions? Yes?
How do I hack on the Shepherd from PAD1?
So the Shepherd has an extensive test suite that is not running as root and the Spirit1.
So for the vast majority of situations, it's enough. You don't need to hack it as Spirit1.
But then we also have system tests in geeksystem running in a VM where when making changes that only have an effect on PAD1, you would first test it in a VM.
So we have system tests, for example, that make sure that the root file system is properly and mounted when you shut down.
And that's something that's very specific to PAD1 and that you need to test in a VM, basically.
Another question. Yes, please.
Would it be possible to replace Android's in its system with Shepherd?
I don't know, maybe I'm not familiar with Android, but maybe maybe I don't know.
Okay, let me know if you do find out.
Okay.
It could be an option, okay.
Are we done? Peter, what's the status here?
Are you have a question?
Does Shepherd support any kernels besides the Linux kernel?
Yes, the Shepherd can run on the hurt.
And it has a small portability layer because the hurt doesn't have signal FD, for example, which is the signal file descriptor thing.
Yes, and it should work on any public system actually.
Just not tested.
Awesome.
Thank you.