All right, this next session clearly does need no introduction, because you're all here for a reason, so Christopher Dolby.
Okay, we're traded. Okay, cool. Hello, everyone. I'm Christopher. I'm going to tell you a bit about the project.
The project I work on and also the company I work for, which is head scale and tail scale, a bit about dynamic, and I spend a lot of time picking the correct English world complementary, because complementary is very close.
So head scale is kind of this open source complementary, no complementary thing to tail scale, but first a little bit about me. So my name is Christopher. I'm from Norway. I live in laden and I'm a software engineer at tail scale and I spend half of my time dedicated to open source and more specifically head scale, but so not everyone might know what exactly the two projects are.
It's not really about a project in per se, but I'm still going to tell you like very quickly and overview of the thing. So tail scale is like overlay network solution. It allows you to connect nodes together and it makes sure you get like direct connections.
The way that works is that you have a control server, which is a software as a service hosted solution, where all your nodes essentially exchange keys and information they need to be able to connect each other. Head scale on the other hand is kind of this host it yourself, run your own infrastructure, open source clone of this product, which allows you to do essentially all the work yourself, but you also then have all the power yourself.
And we're going to touch a bit upon like why that it sounds a bit odd to be allowed to work on these things. So we're going to go through why hopefully that makes sense and a little bit about the community and stuff like this. So head scale first of all, we're a pretty pretty sizable community now. We have like 32,000 stars and get a we have a discord with 6000 members that have a lot of community moderators, people help each other.
We have a bunch of very active users who contribute to like new people, all people who have a lot of if they have challenges or exchange solutions.
We don't collect any data because it's a bit like anti the spirit of what we stand for, but we did once find that you could actually search like certificate ledgers for like domains that start with head scale. So we know there's at least 3,000 installations out there,
but presumably there might be more if you call it like the VPN dotter or something like this. So it become rather successful.
It all started back in like 2020 where my friend had one fund, you kind of started the project, that's like, oh, tail scale is pretty cool.
I want to figure out how they do all this stuff and try to kind of like play around with the debug options in the client and then kind of see can I can I make it work if I didn't use the control server.
And now the few commits and like it's it rolled all along a bit like the beginning of Corona, you know, when I had anything to do.
And then it kind of picked up again in 2021 where I also got bored and we used to work together, so we had contact and so on and then we're good friends and I started committing a bit as well.
It's 2021 and I check in and kind of like, oh, it's not really working. So I brought tests. There's a talk about this a few years ago in the go room, so check that out.
But overall it was already starting to get quite a lot of traction because people found that it could be used for many use cases.
And then in 2022 I got opportunity to join tail scale because they kind of recognize that actually this project is quite nice to have out there in the community.
So that obviously in the open source world and in general, just like software, there's a lot of skepticism and things like this.
We saw a lot of kind of sentiment both directions. So a lot of this is kind of like hacker news and discord and read it.
Things that I found for like the presentation. One of the quotes that I really liked is like, oh, I'm actually more likely to use tail scale because this this focus of kind of like supporting open source makes it.
So I'll use the sauce one, but I know the other one is around and it's a good good will presented by the company and that's that's kind of cool.
But at the same time, there's also this thing that head scale depends on the debug function or debug feature that allows you to kind of override the control server.
And of course, that is a feature that could go away if so desired.
So people who are more on the pessimistic side, they will kind of go in and say, oh, we see eventually a certification disk in this can go away.
And that is, it's a truth, but the client is open source. You can fork it if, you know, you're going to start doing something like this.
So I want to look a little bit about the reason why this is actually quite a nice synergy and it works very well together. So why would tail scale fund head scale.
So tail scales relied from very early on in the marketing model, which is kind of like bottoms up. So you, they give away a free plan.
At home to do their home labs and stuff like this. And then they go to their company and say, like, oh, why am I using this legacy VPN concentrator? That's not really pleasant.
And then maybe eventually these champions will bring tail scale to work and head scale kind of had a very similar similar way.
So like, I also was our hobbyist. We love running around infrastructure, but maybe at work you actually don't want to run your own infrastructure.
So you would still run head scale in your home lab and then you would go to work and just like maybe you should just buy support.
Or maybe you try head scale and use head scale to companies as well. So in, in general, it had a very similar notions. It was kind of like just in addition to the same, the same kind of marketing motion if anything.
In addition, it's kind of like a bit of a bonus that like tail scale wants to build an overlay network for everyone.
And one of the cool things is that if you are multiple implementations of this of the protocol, you kind of have to make sure that you're adhered to like good practices. You document it.
You make sure that you are backwards compatibility. You are more concerned to make sure that you don't break things. And this kind of helps people at the engineers of tail scale kind of sharpen their focus and code reviews goes another round.
In addition, we kind of had like good communication with tail scale engineers. So for all these, it was quite cool because a lot of them are very pro open source. So even in the beginning, we got like invented invited to design documents for a new upcoming protocol change. So we could implement it and be ready to when the client was launched.
And there's also a couple of other implementations as well, not only head scale, there's also iron scale, there's one called wire fire with like a bit smaller communities, but still pretty cool that multiple people are being able to do this.
In addition, there is this notion that like a lot of people, these days, one wouldn't be locked into a vendor if they don't have a way out. So head scale kind of work as an escape patch. Like if you, you might be happy to use tail scale for an hour, but what if something change?
Well, all the same network parameters, at least the ones we managed to implement so far, will still be available as long as you use you have head scale as a backup. So for the time being, you might be happy to actually use this software as a service.
But then you switch on later. So it's kind of like, it works as a way to keep for tail scale to make people more comfortable if they were to be like skeptical towards it.
Which is quite interesting as well. And one of my favorites, this is quite new. I was installing like Incus OS, which is this quite cool distribution for Incus.
Which is like Alexey and VM management type of OS. And then they implemented tail scale recently and in the YouTube video, the maintainer calls out that actually, he's got happy to implement tail scale support even though it's not fully open source, just because when he implement it, he still can fall back to head scale.
That means that it's like still sticks with the ethos of the project, but you kind of get both the commercial popular support of tail scale, but you also can use it with your head scale.
So you kind of become more, it becomes more interesting for all the open source project to implement tail scale support even if it's not like strictly open source, because the clients are sort of the service I'm more less interchangeable.
But then there's come to, yeah, then there's like running the project after you kind of go from free to or unpaid maintainer to kind of salary it. So how do we kind of do it?
Over the years we kind of figured out how we can improve. So these things that make it run for us.
And most of the times we do the right thing and sometimes before short, because there's a couple of things that changes when you when you go from unpaid to paid.
So for example, so now I have a lot more responsibility, not necessarily directly, because there's no boss in the open source world that comes and tells me, but like I am paid now to work on this all the time.
So what I kind of like loose the ability to, oh, I'm not going to bother with this, let someone else do it because I am kind of the main contributor, that's at least the most time.
And if if no one picks it up and even the hairy issues, you kind of have to go and do something about. So if something goes wrong, then you kind of have to do more first line, you're the first line when it comes to urgency and these kind of things.
And at some point there's quote from Nolan Lawson from quite a fun blog post, interesting blog post.
At some point you earn up against issues that are more social problem than technical problem.
One human just doesn't scale and I'm kind of half a human because I only were 50% in addition to some time that still comes out of my evenings.
And you have this problem of like features, code reviews, conflict of interest.
You're always made, when you're paid, you're kind of always paid and you can't forget it because if you forget it, then conspiracy theories may arise.
So any decision can always be perceived as protecting telescope, which is a bit of a challenge.
So we had this issue where at some point I was grooming the, I was a bit tired, I was grooming the issue tracker and I started closing a bunch of things that we thought we would never implement because it was quite complex.
It was kind of out of scope of this project to do a few people and all of two providers, for example, which is quite a complex thing.
I know how to do it and also it requires quite a lot of security on it.
So we closed it, but I made a mistake. I closed it with this very, very undescriptive message.
And of course, that's not for a year and then suddenly someone found it.
And it was like, how are you closing all these issues to protect telescope's interest?
Because this will eat their market and then it will take it down.
And you can kind of make this argument, but I don't think it's really realistic when like you have a 50-end organization,
like half of the developer.
But the other thing is like keeping all these issues open in the tracker at all time is also like a mental toll.
And the kind of thing that came out of it was this long post about how potentially I was planted two years before I was employed by telescope to embrace and extend an extinguish to project from the inside.
Because, you know, it could be true, but that is quite a lot of planning and I think that it's a bit challenging.
So there's a couple of these things that you kind of always have to be in your tiptoes and that makes everything a bit more tiring.
And I found that that is a challenge that you sometimes need to do it.
We had the other one which is quite interesting.
It was like a simple flag missing from one of the command lines that no one really got around to.
And this because we have to draw on this like, oh, is this like this software play to make sure that this never gets implemented and that it's always a little bit worse than telescope.
And I was like, oh, never mind, there's actually a PR for it, it just doesn't merge that.
And that is, you know, you have this like the 2% of skeptics that has this recurring pattern where like they assume malicious intent and then they make an accusation.
And a accusation might often be wrong or not necessarily is rooted in something.
And then they kind of move on with like, acknowledgment of like, if something was actually wrong.
So it just sits in the issue tracker forever.
And you kind of have to deal with this like amounting thing, which is a bit challenging.
So you're going to have this, I like this henland racer friend of mine tiptoes, which is like never attribute mouse.
That which is adequately explained by stability.
And that's not to say that these people are stupid.
There's a lot of things that are just like mistakes and misunderstandings. And that is more of the truth to it, I think.
On the personal side, some of these events are kind of like frustrated with me so much.
And I took a step back and kind of like, instead, I just, I have a 50% part of my job where I can work on telescope.
So I would just work on more telescope projects, which means that ultimately I spent less time working on the open source project, which I think in the big picture is actually quite a loss.
Because it takes like a, it's a project, we invested a lot of like personal time over the years.
And there's no, we want to best for the project, no matter what.
So yeah, it's a bit of a challenge.
So I'll talk a bit about how we structure it.
And I think we do it quite well.
We become a lot better at doing it in the open.
So we have like a roadmap as planned by maintainers.
It's mostly informed by like reaching future parity.
And also like what's fun to work on and it's kind of things.
And then we see, before all of the issue trackers see what is very popular and things like this.
Usually people will come with like, there's a person posting a very, very businessy thing like skim,
V2 groups in support for enterprise Microsoft stuff.
And then we say like, oh, this is probably out of the out of scope.
Of course, that is a business thing.
So you might get told that, you know, you're ruining the project from the inside again.
But luckily, there's not a lot of upwards for skim, V2 support because most people don't need it.
In their home thing.
But you get like all the features from TAVS get that we never got to implement.
And over time we're going to get to all of them.
But it still takes time is like half a person.
And so on.
TAVS can't really inform anything.
But I saw I have my time.
I spend the time and I plan everything and I plan it with the old maintainers.
And so on.
And it works pretty well.
I'll even shut down slack the days I work on ahead of skiers.
I don't get disturbed by like other company things.
And a quote from the one of the founders is like,
they never really open source TAVS get because by the time it came to that was so many kind of like,
it was built for a very specific set of day-round for deploying it.
There's a lot of like customer if else that happens with early customers.
And it wasn't really practical.
And on head scale was already there.
So might as well just support that instead.
So sustainability.
So one other thing we say no to sometimes getting in maintenance.
Or like in into thing because like a lot of men.
There's a lot of maintenance to people committing things to your project.
Cold typically outlived contributors.
They would come like this drive by PR and then they disappear.
And then you have to fix all the bugs.
Not always, but often.
And this kind of like that's why we make sure that when people contribute,
it aligns with like the roadmap so we don't take in everything.
And we try to have this model we say like open source discussed contribution.
Because if people just come and drop by 2009 PR,
then it's very hard to kind of understand.
It takes a lot of time just to like beat it into shape to get it into the project.
Contributions.
We do get contributions.
Like I think I'm the main one.
There's a couple of other people historically.
We have an excellent documentation writer who is helping out and helping a lot with testing.
And then there's like the occasional things come here.
So all of them are LLM based.
4,000 line PRs that you kind of don't know what to do with.
And some of them are like this one fixes my problem.
I'm going to speed up a bit because my time management is getting here.
But at the same time, we never implemented the web interface because we're not web developers.
And it also makes it so that everything has to implement it in the full stack to kind of get it out.
So we made an API and I would like the community to take care of it.
And this is just the top for the first ones.
There's even a lot more of these UIs you can run with head scale.
And that was great.
We had a lot of people who wanted to learn Reactor, Swalter or whatever.
And they kind of made their own UI that now the community uses.
And there's a lot of there's a discord channel where they help each other.
And it's great.
Not everything has to be implemented in the project that is core.
So our community is awesome.
Reflection.
So it's not just us that that is like very specific thing.
We're like someone's making a semi-competitor like looking thing.
I found this, there's this project called WorldWarden, which is made by employee who works for BitWarden,
which is this self-hosted password manager.
It's not completely the same because it does it on this own time.
It's not paid by BitWarden to do it.
Maybe this is an encouragement for BitWarden to actually pay him for doing it.
But there's a couple of things that works in this way.
Another thing that I saw recently is this, in the ghosty repo, which is like Mitchell has a lot of new project.
There is this.
They look all the get-a-bishes and not only men's handers can add them.
And then they force everyone to go through kind of like discussing to figure out if it's a bug.
To kind of filter out low cost or like the, sorry, the high cost very low description type of issues.
It's a very good strategy.
I really want to adopt it, but I'm kind of afraid that if I do, then people will say like,
you're guarding the issue trackers and no one will come with features that will compete with tail scale.
So it might not be a thing.
But this is a cool model.
I hope more open source projects adopt this.
I think also, and this is a bit to the previous talk,
donating time, I think is a really good way for companies to go to open source.
Then the project can figure out what they want to do with the time instead of having like,
companies come in and force different things that I need implemented.
So yeah, sure, you get the future you want, but also you contribute a couple of other features.
So you can detach business interest from project interest.
Last slide is kind of before assuming malice.
I don't think the vast majority of people I speak to is like,
I think it's super awesome that I get to contribute to this project.
And I also think it's pretty awesome.
And it's pretty, like a lot of them have recognized that we don't get,
we wouldn't have gotten as far if this wasn't the case.
Even if there is suspicion, we still aren't much, much further than what we would have been.
Yeah, so this is really cool.
And I'm very happy this quote at the end here is,
it's not something you see every day from a VC backed company.
I wish more companies did this though.
There's definitely a possibility.
So I think I'm out of time.
So this is the slide for the questions.
Thank you.
Any questions?
There's not a follow-up.
So I want to clear.
I'm not assuming malice.
I took a minute to read tail scales patent portfolio while you were talking.
It's pretty clear to me that head scale has to implement the teachings of the patents that tail scale holds.
And since you're under BST license, users of head scale don't have a patent license to those.
Have you had any users who probably mostly be for profit corporate users who are concerned that they don't have a patent license to use head scale?
The answer I have to this is, no, I'm not aware of any of this.
And I don't know anything about patents.
If I'm going to be, I have to be honest.
The licensing is open.
We just follow the same license as the tail scale client.
But yeah, patent license.
Yeah, I will not say anything because I don't know.
This is out of my kind of knowledge space.
Thanks, cool talk.
How you keep up with the future priority because it must be very hard.
So I think the future priority, the core tail scale product is fairly simple.
We're not future part with that yet.
But I feel like I have a very decent idea of what is like the core and then what is like,
okay, this is a accelerated and nice to have some stuff like this.
So keeping up very hard.
But I think when we nail connectivity works always as it should.
You kind of have all of the access control features and these things.
It's going to be a lot easier to build all of the other things.
But I think that is also like, it could be in the section about how overtaking tail scale is not really realistic.
Because they're again 15 engineers and we're half.
I don't think they're concerned just because of this fact as well.
Even as there might be many people starting to contribute to a project is still going to be very hard to keep up.
So I try my best.
That's a good answer.
Yes, so I wonder if you can say more about the governance of the project.
Because you said that tail scale donates time and then the project decides what to do with it.
But you also said that nobody is your boss in the open source project.
Yes, so the way it's organized that is that I have a fork of the project and all of my code go into kind of this is the very technical part.
The fork goes in my branches going in my fork and IPR into the project and then the project is still in one's name which you will see and get up.
And so on so like it's still one kind of step away from tail scale in that way.
Other than that is like there's no one in tail scale that kind of I still have a manager right but like the manager doesn't really isn't really involved in the things around tail scale.
So it's more of a country the main the all the maintainers which is currently and blocking one.
We kind of sit and look at through the issue trackers see what's going on and then the goal is to just do the governments with a full roll map in the open.
So we tag kind of like all the future releases and say which what we're working on and then kind of people can see what we prioritize.
And then they could kind of challenge it if they want to in discord or in something else.
So if you work for both tail scale and head scale, what stops you to just copy tail scale code into head scale.
That's a very good question. It was actually one thing I was trying to put in but I couldn't fit it.
So we did like tail scale for sorry head scale for two years before I joined and had like access to any other code and the code is very different.
So like they're at the moment like all the APIs for like how to calculate the mesh or how to generate like the packet filter has already been written bespoke and there isn't really like anyway to.
It takes the same time to kind of like write our own thing now as to it would be to kind of like take a hammer and beat it into fitting in the other thing.
And I think the other we have kind of like built it from a bottoms up of like making it first work and then making it like more performance while the tail scale control service quite a lot higher performance and can handle advanced vast vast majority of more nodes and stuff like this.
So it's a very different style of code also written by better go programmers than I am.
So yeah.
Thanks for running off time. Thanks, Christopher. You come. That's for more questions on the side.
Finally?
Well, suppose anything.
Why was he sent to the Blue Arc
Yeah, but why can't you do that?