Hello, welcome everybody. Thanks for coming to my talk about attacking ballfields
with this workshop title that I'll explain in a second. My name is Vinci here. I'm an interesting
developer. I've been on the business for more than 10 years and I think I've had fun
stuff. I was a cool. I was a customer at the time and nothing else was there. I was so
feel of easier if you were a developer. It sounds complicated. It's like, oh, we can do it
here and so. It's about a second. So I'm not basically three topics. I want to explain
what I'm going to explain in a second case. We all know I think most of you probably know
what it is. I'll talk about this tool. I developed a cool tracker and it's history. It's
the more than 10 years history now. I started in 2014 and how I exist from the cloud in a sense
because in the beginning it was like cloud native serverless at the time. Now I'm muted.
Nowadays it's a standalone application that runs on netbistiefoundation server. So package
source is the netbistief package collection. It's not just for netbistief. So it runs, I put
here this number. I was slightly surprised. It runs on 23 OSS, including netbistief. All the
other BSDs, Linux, Mac OS, Illumos, etc. And all of these have multiple architectures. So you
might be using it on netbistief and B64 and you think you're in the majority use case which
is probably true. But maybe you have a pine go pro running a 64-bit arm chip and you can also
use package stuff. We don't have binary packages for all these architectures. There's some
sort of like core platforms that we provide packages for. But also we release quarterly. So the
last release was called 2024 Q4 was released at the end of the year and it is the 85th quarterly
release. There's some discussion that we might not have quarterly release in future but we'll
see about that. I think we will. And the release consists of like branching off the CVS tree, yes,
it's still CVS unfortunately. So you have a stable source of source I guess and you can build
your packages from there. There's security updates during the for the latest branch which is nice.
Now, package source does not have a CI-CD pipeline the way you would think about it. Like,
I don't know, I don't know how free BSD probably has something like, I don't know, they seem
way more professional in these matters than we are. Like, intuitively you would expect, you know,
somebody doesn't commit updates of package, let's say, and then there's some sort of
demon that builds the package and maybe it's dependency, see if everything still works.
We don't have that. However, we have sort of, in a way, outsource or distributed the problem,
which is that anyone including you can do bulk builds of either all of the 30,000 packages
or just a part that interests you. I'm using this software called P bulk and publish your results.
And then we can look at other people's build results. Like, I don't know, this guy builds 150
packages on this exotic platform you can see what is broken and whatnot. I actually looked into
porting Pudria from Fabius D, but it seems hard. Yeah. So P bulk is also specific to package
stores. It consists of some C and some shell. Basically, it first does a scan to find out what
depends on what. So you can do it in dependency order, obviously. Then it builds the packages or
some subset of packages. And it sends the report, usually via email to this mailing
this package source bulk. And also the report contains a link to a file called the machine readable
report that you usually also upload somewhere. And the machine read the report is a lot longer
and it has all the individual results. It's not very easy to read as a human, but it's useful for
software and we'll get to that. So that's what the email looks like. You see you have
machine readable version here. This is an official net-based build. You see out of the 29,354
packages, 143 failed to build. And then there's this list here which is sorted by the number of
packages broken by this breakage. So like, if you're, I don't know, say your Python build is broken
and there's a couple thousand packages that can't build, which show up at the top here. So that's
an important signal. And this is really, really important, especially when you're preparing
this quarterly release because there's a freeze. You want to treat to stabilize. You want to see
that at least the platforms you care about have no breakage or nothing important is broken in
the sense. But because these are hard to read, I started building this web app, so it's called
pull tracker. And what it does is it actually subscribes to the mailing list. Any mail that contains
the report is parsed. I put it to a database, so you already have a sort of metadata, you know,
the counts and whatnot. And you have the URL to the full file, it downloads the full file,
parsed as well, and puts individual records for each package. And then you can query that database.
So, so this is the current address where you can see and the relange side, let me see
release engineering, this is the source, my GitHub. This is what it looks like. This is the start,
so you can either start by selecting a platform and looking at the latest build or you can start
by, you know, going by a category to a package that interests you and see the latest results for that
package. Maybe I can actually do this little live demo. Let's see if I can do that.
One second. So here's a page of a build. This one is arm 64 build on a stable branch.
It has a little graph, like 26,000. Okay. And then I should explain the status as maybe.
Prefailed means it didn't even attempt to build the package because it knows it's not going to work.
For example, something that is not for this architecture or marked this broken. Then there's
indirect prefailed, which is when one of the dependencies is prefailed. And there's failed,
which means it's tried to build the package, but it failed. And there's indirect fail, which means
one of the dependencies failed to build. So you can't even attempt to build. And again, you have the
category list. And you have this thing here, which is relatively new. We have some what we call
Sentinel meta packages. So meta package is a package that has nothing in it, just dependencies.
And you can use that to make lists basically. And we have added a bunch of these, like bulk test.
Let's do bulk test go for example. It's one of the most familiar with. So whenever you update
the go compiler, you want to build the good bulk test go package and, you know, it's dependencies
to see that stuff still works. And so the status of these Sentinel packages is a useful signal.
You can see some of these are marked as indirect failed. And you can go directly click through to
the things that failed at the time. So like, I don't know, you'll go to MongoDB. And then you see
it MongoDB broke two others, both of them are the bulk test packages, because they're marked
as depending on it. One annoying thing is you have these links to the actual reports, but you
don't know which is the failing state. So I suppose it's probably failed in the build. So you can
click through and you can scroll through the bottom very quickly. That's a lot of text. It was a bad
example. I don't know. Let's not go there. Let's not go there. Anyway, so you can actually read the
build reports. The logs, the logs, the failed pegs are also uploaded. And you can try to debug
from there and maybe you know what the problem is right away. So that was the, there was a little
tour of the UI. There's a bit more to it, but yeah, those were the most important things.
Also, oh yeah, also on the build page, if we go back to the build, below this is the same
list that we saw from the mail, like the package is breaking most other packages.
You see PHP-56 failed to build and 151 other packages didn't work. And some of them also pre-failed.
In this particular example, it's a bit weird because these are net-based, he has X11 from
base or X11 from tech source. And if you set it from base, it will not try to build these,
so they show up as pre-failed. But that's an artifact you can ignore those.
All right, going back to my slides, going back to my slides. So this thing was started in
2014, and this is where I'll go sort of maybe, I'll say some critical words about the cloud.
So I started this as a serverless application using a platform called Google App Engine that was
the style at the time, I guess. All the back end is written in Go, and Google App Engine had
good support for Go and had plenty of APIs that are useful for this kind of thing. You can receive
emails, you can download files, you have a database, not a SQL database though, but a document database,
which were, okay, it's called cloud data store. You have memcache, you have this thing called delay
function, which lets you schedule expensive processing out of the request. I used all of those things,
and was nice, and I paid for it out of pocket, it steadily became more expensive because
one major downside of the cloud data store is, it will not let you do any queries that are
not backed by an index. So you have more and more indices, and an index means write amplification.
So you write, say, you know, 30,000 records for a build, but in reality with all the indices,
you end up writing maybe 100,000. That's very annoying. So you have, say, 300 kicks of data,
but only maybe 50 kicks in there are actually like build data that is useful, the rest is overhead.
But also, in the beginning, I was thinking, you know, who my user is, and I thought net
BST folks maybe don't have a full featured browser, so I'm not going to use any JavaScript. Everything
was rendered on the server. And in the end, I came back from that, you know, my own understanding of
like web UI is probably evolved. It turns out, there are advantages to like having a JSON API
and doing XHR requests, because one that your query doesn't block the rendering of the page,
it's very nice, you can render the summary maybe, and for the full results, you can wait one or
two seconds until the full results start to appear, and two is also very, very easy to do.
Especially, we've got, okay, so this thing I talked about in 2015 at a package source
con that probably not many of you were at, but so there's also an API, you could do,
you could use it as a JSON API, although it's not documented as such, but it's there.
And now the thing, the thing about the cloud, so Google App Engine, as these things like to do,
Google App Engine continues to evolve. So over time, it became less attractive to host your
application on Google App Engine, and I don't know if they even, if it even shows up in documentation
today, because there's new hotnesses like cloud functions, cloud run whatnot. I would not
recommend anyone use this. Let's put like this. And also, the data store is kind of annoying,
not only because of the right-emplification, but also it's slow. However, they fix that,
they actually use a completely different server software called the firebase data store that
was, they came to Google through the firebase acquisition, and added a shim on top that made
it behave like the old cloud data store. But that's very integrated, everyone's data on to it,
and it's much faster. So, okay. And also, as I said, the cost was starting to rise, because as
I was writing this, I added ingestion, but it did not add division, or at least not automated,
so like the amount of data continues growing, and the cost continues increasing, and in the end,
it was about $60, which I paid out of pocket. But then it continued to evolve further. And I got
an email saying, you are on Google App Engine Classic, and that should be read slightly immediately.
If your infrastructure provider tells you you're on the classic tier, that means they're about to
delete what you're using. So, you should migrate to the latest and greatest App Engine
flexible, or I don't know, like the new tier at the time. Oh, by the way, all those APIs that you're using,
they're gone. Like, there was no API for incoming mail, there was no API for fetching files,
there was no API for memcache, memcache, they say, we don't have anything above memcache, like,
I don't know. So, and there's a direct sort of associated to it. Like, if you don't migrate to
the new thing, you will not be able to deploy new versions after the next February or something,
so I started rewriting, and then it's hit me. Why am I doing this? If I rewrite stuff anyway,
I might as well not target a proprietary platform, but just don't stand alone. And that also,
that also meant I could adopt a SQL database, which it turns out is much better suited for
this kind of cross correlation query workloads. And because I was too lazy to configure
PostgresQL, my first thought was Postgres is great. I started with SQLite, and it turns out
SQLite is very good. So, it is not a toy. We still have, like, I don't know, a couple hundred
gigabytes of data in there. It does not break a sweat. It's totally fine. The one thing you kind of
give up is that you can't really serve, you can't really have replication, or only with extra
work. So, basically, you're kind of limited to one instance running at a time, which is totally
fine for us because there's not that many visitors. So, SQLite is actually pretty great for this,
and it's also much faster than what we had before. And then, and then I thought, now that it
can run on any OS, on any, you know, just like you just started on the command line, it runs
open support. Can I get the net-based foundation to host it? On of hand net-based users. So,
started that process. They asked, like, okay, but how much RAM does it need? How much space does
it need? We made up some numbers, and they were like, yeah, sure, it can fit in a corner of that
server over there. And so, that happened. And today, the old address, a book tracker.appspot.com,
hosts a page that says, this instance was shut down, and here's the replacement link. So, that's
that's the last deploy I did. I posted a static page that says, we're gone. I deleted the database
and now we're native. Quick overview over the APIs that we used. My absolute favorite one is
the incoming mail API. It's not forward. So, the app engine incoming in mail API, it'll transform
an incoming mail into a post request to this slash a h slash mail endpoint. And it turns out you
can do that in a single line and it dot forward with curl as your mda as your mail delivery agent.
So, that's very nice. For writing the HTTP server, there's the built-in one and go. So,
that's fine for SQL actually using ORM. If anyone knows what an ORM is an object,
it's a very simple thing. You write your SQL queries in a file and it generates functions for it
and types for the results and the inputs. So, that's nice. It's called SQL C. It's a little
coaching. Instead of a mem cache, I just cache data in process. You know, I have a map. And it has
the key to value and an expiry and the values are regularly removed. For chart, instead of the Google
chart API, I use something called chart.js, which is a single JavaScript file that's self-contained.
So, that's also nice. For the App Engine Log API, I use this thing called Log Rust, which
is one of those fancy-go logging packages. It doesn't have those. It's fine. Any other
putting would also do. So, what this logs is mainly debug logs. When there was an error connecting
to the database or things like that, in practice, nobody really reads the logs. The request logs
themselves are logged by the foundation in the Engine X reverse proxy, which sits in front of
this. And one also bigger reflection I had to do is that they hosted at a sub-directory. So,
slash pull trackers, I had to fix all the URLs to include the sub-directory because previously
was running on slash. And for the files fetch API also, how do you fetch the files just
making outgoing HTTP requests you've done? I actually develop both versions inside by site and
different branches of the same GitHub repo. And eventually, I just declared the branch
called SQL the default. And this is where development continues. And they all main branch is just
dead. And we also didn't do any data migration, because the data comes in by itself.
Like you start with an empty database, you let it run for two, three days, you have a couple builds
already, and you'll find. Nobody is that interest in all data, most people want the newest data.
Sometimes it's useful to go back to see like, at what version did this package stop building,
maybe? Like, was it caused by something else? Or was it caused by the update that had to
two months ago, if you have two months of data, then you can see it. That's nice. And yeah,
and it's there now on the net-based server for release engineering. And I want to talk a little bit
about the community involvement in this also. So, I'm a net-based developer, where this is
developed outside of the tree. Obviously, it only runs like with package source stuff. So,
there's no users that are not package source. And I'm also kind of still the only developer.
However, what has changed, especially recently, like this, this becoming hosted on a net-based
the order domain has actually given it a big push in like visibility and legitimacy. And all the
release engineers and all the people care about stuff not being broken, they use this. And they
tell me that it's useful for them. And they come back with requests, for example, the Sentinel package
table that was a recent request about half a year ago. So, yes, that's kind of a one-man show,
but it is kind of motivating because you get these testimonials back saying, hey, thanks for doing this.
It's very useful. You know, if in your project, you have somebody who does something like that
on their own, you should tell them that sometime. It's the probably one of the biggest motivators
there is. One other thing also, I started doing releases. I hadn't realized how satisfying
it is to not just deploy, you know, you can deploy anytime you want, but like to say, I'm going
to deploy this thing now and I'm going to call it the 2025.02 release. And then write a little change
lock saying, in this release, I fixed this bug and I fixed this bug and this new enhancement.
And then you send it to like some mailing list and then you get things back like, you get some
reactions back and so on. So, it's nice. It gives sort of a closure to having on some work.
And in the future, there's still a lot of stuff for improvement. I noticed that some of the
actually the oldest bugs that were there from 2014 are still open. That's because they're
basically requests for redesigning the thing, so it's hard. There's some requirements that people
wanted to see such as, we should follow the commits and highlight when a new package has not
been built yet, for example, or be able to diff, like, what's has changed between the net
beasty build, a week ago, and the net beasty build today. This would all be, I think following
package source is kind of hard, but like the diff is kind of straightforward to implement just
takes time, somebody has to do it, or search by maintainer. There's a related website that some
of you might have used, it's called Repology. It's, it also, it follows different repositories,
the packages, just one of them, it also follows free beasty ports and deviant and one not.
And it'll tell you which repo has which version of which package, and Repology also allows
searching by package maintainers. So, I can go into Repology type my name and see if the
packages I maintain are up to date, or maybe there's an open security patch that they haven't
applied. So, something like that would be useful. Also, if any of you, I don't know, open beasty
folks, maybe, or, or, previously, I don't know, or some substrol, want to use something like this
for your builds, you should come and talk to me, we can, we can do things like that for one
to. And I'm just going to, I guess, be working on it. It's kind of low intensity, it's maybe,
I don't know, few, a few hours of work every quarter, it's, it's fine.
And that was kind of all I had. Thanks very much. The slides are online on my blog.
And again, the URL is here. Are there any questions?
Okay. Do you want to say a latest question on the latest, why can it's for the
thing? So, can you repeat?
Do you want to get about the latest release on the latest data?
Yeah. What makes best of format release?
Um, well, the question was, why do we care about all your data at all? Why not just
call that release? You mean, you mean package source or the, uh, the, uh, the app?
Uh, the app? Well, you don't, you don't actually need releases, per se. It's just a way of, um,
it's just a way of announcing what's new every once in a while, uh, showing people that, you know,
it's, it gets updated. And basically, I, I just tag a release every time I deploy to the server.
There's also a little deploy pipeline, uh, that gets triggered by the release.
Are there any other questions?
How would you measure the, uh, the comparison? Complaints versus thank you for the, to the team.
So, the question was about complaints versus thank you. I think
I, uh, I think most people do kind of both in the same email. There's there.
You know, they're saying things like, I really enjoy using this thing, but it would be great if it also did
X.
Yeah, most of the, most of the people are very nice.
It's not a question, but I just want to, uh, make it, make it, what, when many says that
the package source is more to platform means that for example, for your mark.
Yes.
Yes. Yes. So, so one of the, one person works full-time on package source, uh, is, uh, employed a
giant and they, they produce this OS called smart OS, but they also provide, as you said,
Mac OS, uh, binary packages, they're very good. So, if you have a market set of going with home
rule, um, why not, you know, why not try package source very good.
All right. Thanks again.