Hello everyone, I'm Juliana Sims. I work at the Spritly Institute where we're working
on a lot of cool social web stuff. My colleagues are going to have some talks in this room
after me so you can hear more about that then. This work is funded by the NGI-0 and trust grant
through NELNET, so it is not actually Spritly Work, but it is using Spritly Technology.
I'm going to talk to you about how I am. I'm buttering Ludovic's precious shepherd.
Now, I'm integrating the Goblin's library with it. I feel like I had a better segue than
that, but yeah, so I'm porting the shepherd to Goblins. Who here is heard of the shepherd?
If you haven't, go back in time 30 minutes and you'll learn everything you need to know.
So yeah, I'm porting the shepherd to Goblins. I don't really need to explain what that
is and its system, process manager, written in Giles game uses fibers to implement a concurrency
system derived by the actor model. Goblins is a Giles library that implements the object
capability security model, which provides security guarantees as the name would suggest,
and it also uses the VAT model of concurrency, which is similar to the actor model with
some variations. Again, my colleague Jessica Talon will explain that a little more, I think,
later today, so I don't have to go into too much detail there.
And I'm just going to be talking about what this project is, what its goals are, some benefits,
where it's at, where it's going. So, first of all, what we're actually doing is discussed
in more detail. I have a design dock up. That's a GitLab repo I use because of, I like
the force push and savannah makes that tricky. savannah is where the official shepherd repose
stored, but the Goblins port design dock, if you see all the way down at the end of the URL,
you can go either to the GitLab repo under Spritly or the official savannah repo on the
WIP, Goblins of FI branch, and you can read the design dock for more detail. It's kind of
a living document. I think a couple of the things I said in there, I've already decided I want
to change, but it'll give you an overview at a little more lower level of what's happening.
And basically what's happening is, I'm taking all of the bespoke shepherd actors that
Ludific has worked so hard to make what you're very nice to work with, it should add.
And I'm changing the multi Goblins actors, and I have a little joke here, but calm is a
Goblins thing. So, instead of using CSP and implementing actors directly, we take advantage
of the Goblins VAT model and our actors, and the concurrency are managed by Goblins.
There are other code changes necessary to sort of translate between how Goblins does things
and how traditional scheme does things. I'll mention a little more later, but that's the
core of the work is just translating code into a slightly different format. And this has some benefits.
So, first of all, it makes a lot of bugs, Spritly's problem. A lot of the concurrency issues.
Ludific was just talking about our supposed to be handled in Goblins, and so if we run into
some of those at least, I've realized not all of them, but at least some of them, that's a bug
in Goblins and not in the shepherd. I think I hope that it will reduce the required knowledge
for developers to contribute because you won't need to understand the actor model and concurrency
sequential processes and how it's implemented. You'll just need to understand how you use Goblins,
and I'm hoping that will help. And it will also provide a distributed debugger. I threw that
point in there at the end of Ludice's slide when he was talking about debugging. I don't know
exactly what that's going to look like yet, because I'll get there. And yeah, hopefully it'll
produce more ergonomic code because a lot of Goblins' asyncronicity and concurrency looks just like
synchronous code. So, here's a little bit of an example of the code itself, and it's just an example of
the code for starting and stopping a service. I think the shepherd equivalent, it's in the service
controller. I'm merging different parts of the code base. So, here's the before, and I didn't test
if you could see all of this, but as you can see, we have, this is a branch for a match statement,
so I just pulled it out to give you some context. We're creating a channel, spawning a fiber to
tell another part of the program that the service has started, and then we're doing some
bookkeeping announcing to users what's going on. And so you have all of this code for mostly bookkeeping,
and then we have this code, which does pretty much the same thing. Instead of using a channel,
as we see here, and during get message and all of that, we instead use the Goblins concept of a promise,
and so we spawn a promise, and then we return a resolver, and the caller will,
is handling the actual, starting of the process, and will fulfill that, and so that's an example
of what this code will look like in practice. This let on form is a macro around the Goblins on,
I think that's a procedure, that handles promises, and I think I talked about promises a little bit,
they're like JavaScript promises, again, hang around, you'll learn more about that,
but their mechanism for concurrency and asynchronous code evaluation. And so this will not just
change the code base, right? It will have benefits for users. One of the local level things,
so if you have your machine and you are a single user and you have your shepherd,
one of the benefits is that it will give you the ability to control your user shepherd services
through the system Damon, the way the shepherd works right now, you spawn a separate shepherd
Damon under the control of the system Damon to manage your user level services, and
this work the certain aspects of the Ocaps paradigm enables you to just control the system
shepherd, but only the aspects of the, or the only, only the specific services that you spawn
yourself that you know about. And this will also give us the ability to have different shepherd
Damon's on different machines talk to each other. That's sort of the headline feature, the thing
that I'm most excited about and got most excited about in this project. And this will use the Ocaps
and protocol, which Goblin's implement, and it allows you to abstract away the network layer,
so you can connect your, your shepherd Damon's, your machines, over arbitrary network boundaries.
I mentioned plan 9 because if I don't mention plan 9 in relationship to this project, I get sad,
because I think plan 9 is very cool, and that's a big part of what inspired me to want to work on this.
And it will also facilitate existing deployments of the shepherd on, as the example here, says,
high performance computing clusters, HPCs, which I usually call super computers, and I think that's
incredibly cool. So we're going to, uh, sprightly is going to implement a, uh, UNIX domain socket
net layer, so that the existing networking infrastructure that HPCs use will continue to work over Ocaps
and, um, I sort of have this, like, last point of pointing towards things this can be useful for
that it isn't currently deployed for, because, uh, I'm not aware of any system quite like this,
and so, uh, one, one idea that comes up is orchestrating your, your networks of servers and managing
services on remote machines from maybe a central, uh, uh, machine, because you'll be able to start
and stop and register and control services from any machine, and the really fun thing is that
the user local benefits and the network benefits are the exact same thing, because Goblin's
allows you to abstract away the network and it allows you to write code that operates locally
and over the network and looks exactly the same. Uh, so I'll get to showing you that code in a second.
I should probably have put that first, but just as a click overview of where we are on the project
right now, uh, not quite as far as I would have hoped to have been, life and the world and things have
slowed me down, but, um, so we have the core service registry ported to Goblin's, we have, um,
the, I shouldn't have said core with that one. We have the service registry ported, which is what
actually keeps track of the service object. It's a bookkeeping device essentially, but it's a
great point of abstraction, um, for a lot of different functionality, and we have the core service
actor, which represents the actors. I've merged the controller and the service. I think I'm probably
going to separate those again, which I'll again discuss later. I still, in terms of porting this
core service actor, I still need to get actions. Um, Shepherd has this cool ability where, you know,
you type her status or you type, um, I'm blanking on examples, but you can send commands to the
services and Shepherd allows you to define arbitrary commands and these are called actions.
So I need to port the functionality for that. I've been putting it off because I kind of want to do
some trickery using Goblin's. We'll see if I actually do that. And I also need to do the actual process
management. It was funny that Ludific was talking about how hard can it be to spawn and forks and
processes and keep track of a signal, um, I've been thinking the same thing and I'm probably
overest of any myself, but you know, and then I've started documenting these code changes and what the
API could look like for users. So we're about to see a rough example of what the API could look like
for users. And this is some code I'm using to test that things work. I've borrowed services
from the Shepherd's Test Suite and torn them out since I don't have the full system set up to
give you a quick run view of what's happening. This is all, um, boilerplate. It's going to be
handled in the Shepherd Proper. You don't have to worry about it. And we see here we're registering
some services. We're only worried about the test service and test two service today.
Essentially test service creates a file and puts, uh, I think, food in it. Test two service
changes the text in food to bar and when either of them stop they delete that file.
And so here we see, so we register the services and then, um, once they're registered, we ask the
service registry to send us a list of registered services and then we operate on to the services.
This API is strongly subject to change. It's sort of, uh, as I'm trying to figure out how
these various pieces will fit together and doing that so you can ignore the details of that bit.
It's not important. And then you see down here, once we've registered everything, we have
letons again because we're doing this asynchronously and we're starting service two, which depends
on service one. And then we're stopping service one, which should because it is a dependent of service
two stop service two. And the arrow operator is really important because that is an asynchronous,
uh, message to these service actors. And that arrow means that this could be over the network
as well as locally. And then we have some promised stuff to deal with waiting for things to print.
So let me show you this in action. It's not, uh, super exciting, but
as you can see here, this is the list of services. We have four services, a little bit bigger,
absolutely. Is that good? Baker. Um, so we have the list of services there and then we have
a little message telling us that we're starting these services and that we're stopping these
services. And it's all working over goblins. And these services could be on this machine,
as they are, in this case, where they could be on a completely different machine just as easily.
Um, I think I'm running lower on time than I thought I would be. So I might skip this part.
There's some issues around the port with fibers and goblins, translating promises to values,
goblins loves promises. They're very similar to JavaScript promises. The actual um,
the Ocaps promises inspired JavaScript promises. Um, some issues around compile time.
Object construction because garlics to compile things into bytecode ahead of time. And if you
try to define an object, which is a lambda at the compile time it causes issues.
World events have made this difficult. Um, I probably was wrong about being able to merge the, uh,
actor controller and other parts of the actor. And then there's a lot of things I have to not do.
But I think it would be cool. Uh, Ocaps likes process isolation. You want to spawn your
services in a cold dead world, as we say. And that's sort of out of scope serialization,
which is a feature goblins provides where you can serialize your actor objects. And then you can
revive them. And it's the same actor object. You can upgrade them. All this stuff. That's out of scope.
Um, I tried a little bit to switch replacements to using become, which is a thing where an actor
becomes a different kind of actor. That didn't, um, prove necessary capability grants is where you
handle objects to other places. That might come in some, but anyway, I just get really excited about
problems. I think it's great. Uh, so the next steps finish up what I've started on, essentially,
and refine the public API, uh, make sure that, you know, you don't have to use add-up to
actually access your objects that there's a, uh, traditional scheme values API that you can use.
To do stuff, um, that's the talking about service references, how users are going to be able to
refer to their services in their configuration files, um, getting problems to the values. And right now,
like I said, I've copied like parts of a test file for testing all of this, and I want to get proper
testing. And then in terms of functionality, the things that are left, we need service actions,
we need spawning and monitoring, uh, integrating the CLI, and then it'll just mostly be a matter of
exposing the new functionality the goblins provides, and I don't know how complex that'll be.
But we'll see. So that's it for this talk about chef and goblins. Um,
spritally is going to be talking more about their work in this room after me. So stick around if you want
to hear more about that. And, uh, thank you to Enelmet for funding this work. Thank you to the organizers
of this room. And, yeah. Do we have time for questions? So, yes.
Yeah. Yeah. So that was just, uh, stay repeat the question? Yes. Um, because you can, uh, control
and monitor services on different machines, you can have dependencies across the network boundary.
Yeah. Yes. Yes. So, uh, the, the question was given that you can have machines, uh,
uh, dependencies on different machines, could you end up with different shepherd damans having a different
perspective of the network. And the answer to that is yes. That is something I'm thinking about
constantly in the back of my mind because I'm not sure exactly how to handle that. You could
imagine, for example, that you have an application which needs a SQL database. And you don't care
what the SQL database is. You have some abstract API you're using. It could be anywhere on your
network. Um, so how do we make sure that you're getting the right kind of SQL database? That's sort
of thing. So yeah, this is something I'm actively thinking about. It's one of the big issues that
needs a little bit more design like on it. Any other questions? Oh, it looks like we have
you get switched to the microphone. Um, yes. Um, um, I may have such a problem with this, but um,
with this part, are there changes, shepherd's public API? Okay. So the question was, will this
port change shepherd's public API? The public API will remain backwards compatible, as well as
the protocol for communication between the herd and shepherd command lines and the shepherd
damen proper. So all of your code will continue to work exactly as it has. Um, uh, I would like
to expose a new API that allows you to take advantage of some of the nice cities of goblins, but
I might not. We'll see. We'll see how things go. Yeah. Yes. Yeah. So the question was about my
mention of plan nine and asking if machines will be able to use physical parts of another computer,
the example is an audio card. Um, the answer is that this work will not enable that, but you could
potentially use it to set up some sort of system that would do that. So that's not within scope
for this project sharing physical resources, but um, I can imagine some ways you might be able to
make that happen. Yeah. I don't see any more questions. So I'm going to hand it off. Thank you
all so much. Thank you for that today.