Hello, everyone. My name is Michael. I'm a system software engineer at Apple. Hello.
My name is Evan. I'm a compiler engineer on the 15-met Apple. So, the process of porting a programming
language to a new system involves intimate understanding of the internal workings of the language
and the system you're porting it to. It also requires a healthy dose of patience for
when things inevitably go horribly wrong. You often have to deal with situations where
print statements aren't printing and the debugger isn't debugging, so you have to cobble together
whatever tools are available on the OS to figure out what's going wrong and fix it. So, this is the story
of how we dance with the demons and ported Swift to FreeBSD.
We'll start with a quick bit of history on previous porting efforts.
And then we'll break down the bootstrap that we used to bring up Swift. It's four steps.
Then we'll waltz our way through Build Errors, getting Swift built on FreeBSD. And finally, we'll
quick step through the code. Squashing bugs as we go.
So, this isn't the first time that Swift has been ported to FreeBSD.
The first porting effort landed in the Swift repository in December 15 or December
of 2015 shortly after Swift was opened sourced as part of Swift 2.2.
Roughly six months later, in May of 2016, the first ports in FreeBSD were landed.
And then things went quiet for about six years.
And without CI support, things kind of bit rotted.
So, then between 2022 and 2023, 18 commits were published to the Swift repository to revive the build.
Now, these efforts were driven by community members. They didn't have the backing CI support.
And they also didn't get the test suite fully running cleanly.
Now, in 2024, Swift, unfortunately, was no longer bootstrapable from an image that didn't have an existing Swift
installation. This broke some time in Swift 6.0.
So, in May of 2025, I brought the bootstrap back in Swift 6.2.
Then, in June of 2025, an unofficial Swift 510 was uploaded to the FreeBSD port street.
Which brings us to today?
So, given the efforts of the past, what are we doing differently to ensure that it doesn't
come out again? Well, we're working toward a package that tests cleanly across all of the tools in the tool chain and across the ecosystem.
And we also have CI support set up, so that we can ensure that things keep working going forward.
Let's talk about Swift. Swift is a general purpose programming language that makes a very nice balance between safety and performance without getting rid of rigability.
The package from Swift.org contains a tool chain, some developer tools, a Swift standard library, various platform libraries, and testing libraries.
Now, the tool chain is used to compile the libraries, but it also depends on the libraries, so we have a cycle that we need to break.
Now, we can't build a Swift standard library without a Swift compiler, so I had to aggressively start chopping pieces out of the compiler until we were left with something that didn't have any Swift left.
I did this initial work in Ubuntu 2404 because we already have support for Ubuntu 2404.
So, any test failures that we see out of the bootstrap are because the bootstrap failed to converge and not because we have missing platform support.
So, we take that partial compiler and use it to build the standard library and lib dispatch, lib dispatch is needed for concurrency.
This makes up our stage zero artifact.
We use that stage zero artifact to build a new Swift compiler that uses the stage zero runtimes.
The stage zero compiler is pretty brittle and it can't actually build a whole lot more than just what's here without crashing.
And anything that it does build is kind of an questionable state because the compiler is quite literally missing major portions of how it works.
So, it can very easily miscompile what it does build.
So, we now take our more complete compiler and build the standard library dispatch and foundation again.
This collection becomes our stage one artifact, which we then use to build a stage two.
Now, technically this compiler, the stage one compiler is able to build the entire tool chain without crashing.
But that tool chain doesn't pass the test suite, so we haven't actually converged yet.
And so, to save time, we only build a subset of the tools.
Now, we can take this stage two tool chain and build the complete tool chain.
That full tool chain passes the test suite, so our bootstrap has converged and we can stop.
Now that we have a path to bringing up a tool chain, it's time to start building on 3bst.
Over you Michael.
Thank you.
Once we get the tuition boost draft is mostly mechanical.
Most of the work is we need to address the compiler errors and warning as they come up.
And then we address some testing failures and we rebuild the tool chain again and we repeat it again and again.
But fortunately, because of the previous effort, we have some speculative support in Swift already, so it makes things a lot smoother.
That says, we do run into some really quirky, interesting issues along the way that you may not expect.
The first one is ellipsy pass pass, so LLVM has this concept called module map.
And not to be confused with the standard C pass pass modules.
Since 3bst uses LLVM and import it, it also inherits the module maps that comes with it.
Except the module maps in 3bst including 3bst 14 is a bit outdated, and it does not export the modules we need to come down Swift.
As we saw in the early stage when we were talking in 14.0, we actually need to go into the base system and manually patch those module maps, such that we can just build Swift.
Fortunately, this is all addressed in 14.3 when Swift, when 3bst import a new version of LLVM, and this is no longer required.
Another very interesting issue is, stand the pair API, API, you'd be like, what that just a stand that leaves C pass pass feature?
Well, except turns out 3bst has stuck with older C pass pass, API for a very long time, when the standard pair does not have a triple copy constructor.
Now, this is a rather easy face, which is like we move all the bits in the Swift compiler that uses the triple copy constructor, and we place with something more explicit.
Now, with that figure, let's look into debugging.
Porting a new tool, porting an entire tool to a new system is hard. There can be incorrect assumptions.
There could be like black king debugging tools at all, and generally speaking, many things can go wrong.
However, while also on 3bst and 3bst by default include a lot of debug utilities.
So, in the next couple of slides, we're going through a couple of very common issues, like crashes, hands, and just other issues, and see how we can fix them.
Let's use a Swift PM, which is an example. This is a step that we build the two shrink with Swift package manager.
We know something is wrong here, obviously, but it tells us a little about why.
And also, notice how the Python script is just like swallowed all the standard error and standard out and backfaces.
So, what should we do?
Well, first, let's tackle what really goes wrong. The build process was Swift project.
It involves passing artifacts and output between multiple components, like Swift driver, Swift package, Swift one, and so on.
So, first, I will be figuring out how these commands are invoked, and then are they invoked correctly?
3bst includes D trays in the base system, so we can easily use that to achieve this goal.
And to make it even better, 3bst also includes DWatch, which is less known, but it's a really nice replica for D trays, and by default includes color output, just make things whole lot easier.
So, here we see what you think, DWatch, to trace every single process invocations when we're building Swift, and we can verify that if we have passing wrong command line, argument, and if a process has been exit way too early, that is supposed to be.
So, one of our first goals is really, we covered the standard out and standard error, as it gives us information about what could really go wrong, and we can eyeball it.
So, to do that, we write a really simple D trays one line there, essentially, which is just tapping into the right system code, and just dump the standard out and standard error out.
In this particular example, we actually see the compiles actually crashing, and it means a code dump, and we can now see the code dump and see the context by the crashing and go on and fix them.
But, what if is the opposite? What if the process just never want to exit and just hands there? We can also do DWatch this with D trays. So, it's another one line there here. What we're doing here is, essentially, is to profile the user space stack, every 100 milliseconds, and this would give us a sample of different stack trays, and we can compare, and then try to figure out, is it really hanging, is it making thorough progress, or is it something else?
So, we can talk all the about D trays and how we use D trays to solve bugs, but because time, let's look into something that's a bit more interesting.
So, one, everyone's a wow, we can see this a lot, it's like a polluting our standard out, and we're like, okay, what is this? And it's definitely not coming from Swift, so it's definitely a previous D thing.
So, turns out, for VSC, it's a nice feature that will warn you if a threat has exit with left over a threat specific data. So, what does that really mean? It just means, like, you're setting up some threat local storage, you attach it, destructed to it, that destructed get runs multiple times, but the TLS is still not now. So, something must go wrong. So, this is Swift telling us that something that some bugs going on here.
So, we can just go on to all the dependency and the Swift codebase and try to figure out wherever we use threat local storage if there's something wrong, but we're on 3BSD, so we can do something better.
So, instead of going through our codebase, we just go to BSD, and then we just patch the base system, and just we build everything. This gives us a couple of bonus.
First of all, turns out, we're building the whole OS can actually faster than we're building the compiler, and then it's high to a tree.
Second, we can make it crash, so we can get a call done, and lastly, if we really worry about messing up the whole OS, because we're building it, we can just use a boot environment and to restore it later.
After applying the log, this is what we get instead, and now we can see way more data, and say, okay, hold on a second, that pointer get invoked multiple times, exactly four times, and something to do up. So, what is in that address?
Because we actually make it crash, we can now take this call done and take this address into LDB, actually from the base system, because we don't need a Swift bit to support this.
And we can tell now it's actually coming from a call foundation, okay, so let's fix that. Without taking too much details, we apply the fix instead of setting that pointer, we're setting to now, and now we fix it, and everything's well.
So, turns out the trace is very useful, and it's grateful to debugging, and also turns out patching and rebuilding the whole OS is a rather efficient debugging techniques if you can do it efficiently with that bombshell off to you.
Thank you, Michael. Let's talk about dispatch. So, lib dispatch is a library for managing asynchronous work. It's also the default backend for Swift concurrency.
It uses core accounts to help it determine how many threads to initialize thread pools with, and to help detect and avoid dead block states.
While bringing up dispatch on free bst, we had this little test value come up. Doesn't tell us a lot, so let's look at the test.
It's verifying that we will initialize a new thread if all of the existing threads have been consumed with existing work, and some new work items are added.
So, it saturates all of the cores with a timeout call and a bunch of spin calls.
Then, it schedules a call to this race flag function, which just increments a flag in the context object.
The initial timeout function is the one where the interesting stuff happens.
So, it sleeps for two seconds, which gives dispatch a chance to initialize the new threads and launch and run race call.
Once it returns from that sleep, it checks that the flag has been implemented to one. Yes, there's a race condition here. We know.
But, it just verifies that that is one and then goes on its merry way.
Now, the bug turned out to be in how the test found the number of cores to initialize everything with.
FreeBSD doesn't have the hardware active CPUs, this call.
This cuddle by name here was failing and returning a failure code that wasn't being checked, and so active CPU wasn't being set to a reasonable value.
So, this is a good reminder to check your return codes even in your test cases.
Now, the fix for this test is fairly simple.
Use this cuff to get the number of online processors, not terribly exciting, so why am I showing it to you?
Well, it's a little bit more involved than that, just querying for the number of online cores gives you all of the cores across the entire system.
There are things like C groups on Linux and Jails on FreeBSD, where you can limit the number of cores that are available to an application.
The discussion that came from fixing this test actually resulted in changes being merged into FreeBSD.
With the most recent release, according for the number of online cores with CISComp will give you the number of cores that are available to your application,
respecting limitations set by Jails, not all of the cores across the whole system.
So, this is Swift already influencing how FreeBSD works.
Let's talk about concurrency.
So, we had a test that looked like this.
It's supposed to sleep for five seconds, and then return.
But it was hanging somewhere in that sleep, and being timed out after about an hour.
So, using ProxStat, we can see what the threads are up to.
There's one that's on 6th to spend.
There's another that's sitting on a K event call.
I happen to know from knowing dispatch that the timer mechanism is using K event and timer events under the hood.
So, I also know that this is the first time that we are using the KQ back end for dispatch on a non-apple platform.
So, there are probably some constants that are not correctly set in there.
Seems suspicious.
Popping to open in the debugger, we are definitely on a K event call.
And then LLGP crashed on me, so I couldn't get any of the information about what the event was.
And, relaunching LLDV, it refused to re-disembolicate this file, so I couldn't set breakpoints here anymore.
So, what do we do? Back to detrace.
So, I wrote this little script here.
It just tracks everywhere in our test program, where we are calling K event on a timer event.
And then it prints out what the flags were and information about the event.
When we run the script, we see three events get registered.
Cool. That last one there has a raw time that looks really similar to the time we're looking for.
But, that filter flag is set to zero, which on 3DSD means milliseconds.
So, our value is like six orders of magnitude to be here.
We didn't ask it to wait for five seconds. We asked it for 57 days.
Okay, so we could do some division or just set the nanosecond flag and say that it is in nanoseconds, not milliseconds.
Catch that and we're golden ways for five seconds returns.
We're happy that those other events seem to be part of the event loop, which is also getting service at a more reasonable cadence now.
So, this is detrace rescuing us once again.
Now, let's talk about metadata.
So, it has runtime metadata is used by runtime and debugger for things like reflection and runtime type information.
The LDB test suite was crashing in some places and also just saying that there was no metadata in others.
So, it was a bit tricky.
There's a tool called Swift Reflection Dump, which extracts this data for us.
It runs on a Mac and it can open things.
So, I just took the files from 3DSD, sung along to my Mac, and bam, it crashed.
Cool. That's great.
Crashes are wonderful.
Attaching a debugger, point this right out where we're looking for.
It's a call to string length, which is trying to get the section name length.
There are a few variables that contributed to this call.
So, let's start poking around.
So, the string table, that's the kind of the main thing we're looking at here, looks fine.
That looks like a string table.
P-C, let's keep going.
Start pointer, looks like a pointer, but it's kind of too big.
Something's up with that.
And if we ask LDB to jump the memory, it says no.
Can't do it.
Okay, that looks a little sketchy.
Let's print out the other variables.
The table size is 652 bytes.
Okay, seems reasonable.
Offset, that's way too big.
Way too big.
And if we look at it in hex, that's definitely poison of some sort.
All right, but the issue with this number is it's coming directly from the section header name offset field.
So, what's up with that?
Uh-oh.
Our header looks like garbage.
So, did we completely break the linker?
Did we completely break the compiler?
What is going on here?
So, it turned out that this was the header for the fourth section.
So, let's start looking at it and read out.
Looks okay to me.
Looks good.
Says the new version.
Looks fine.
Okay.
Maybe read out if it's falling back on some clever error handling.
So, we've got the header section header offset of 2b40 and 64 bytes.
Uh, the fourth section will be at 2c40.
Popping it open in our favorite hex editor.
Looks good.
Cool.
It's little andian.
It's x8664, so little andian, but looks good.
So, I don't think read out if it's falling back on anything.
Um, this suggests that there's something all going on between where we are parsing the data and reading the data out of the file.
Because the file was fine.
Um, speeding things up here.
File buffer wasn't set here in this lambda, but so we were falling back on this read bytes thing.
Read bytes?
Not too much.
Let's keep going.
There's get content set address.
That looks a little bit more interesting.
If image coming out of this decode image index and address isn't set,
we'll return an empty string graph.
Hmm.
Well, here's what's going on.
We're iterating through the images.
And if none are found, we'll return an old pointer.
Yes, what we were returning.
No pointer.
So, we are definitely returning an empty string graph.
So, where did these images come from?
And why don't they have our data?
Well, uh, they're being constructed here, but that doesn't tell us a whole lot.
So, following that image constructor, uh, there's this scan elf type thing,
which is going through and segmenting our binary and applying relocations,
kind of like a mini dynamic loader.
If we take a step back, the metadata and swift is structured data.
It contains references within itself.
And some of these reference required that the data has been relocated to be meaningful.
So that makes sense.
For those of you who know what's no elf, that might be triggering some warnings.
Let's talk about that.
So, this is an elf file, roughly.
Um, the dynamic loader goes and reads the program headers.
And that tells it how to chunk up the file and where to put each chunk.
These are called the segments.
Elf tries to balance, uh, load times with memory consumption.
So, these, if you make the segments too big, uh, there will be fewer of them.
So, it loads faster, but you'll be pulling in more data than you actually need.
Uh, now this might pull in some of the control structures, like parts of the section header table.
So, in elf section header tables are not guaranteed to be loaded in any segment.
But that's what we were trying to read.
Now, why is the runtime okay?
Well, the runtime wouldn't see the bug, because these sections are allocated.
They have start stop symbols, which surround each of the sections.
Uh, sections.
The runtime uses that to load the data.
That's fine.
Symbols totally work.
The data still exists.
It's just the section headers that are bad.
Now, we haven't resolved this bug yet, because we're still weighing some options.
But that was a pretty nasty bug to deal with.
So, we dance with the demons.
We resurrected a bootstrap.
We waltz through the build errors and, um, quick step with tools like detrace.
And got Swift working more or less on FreeBSD.
Thank you for listening to our talk.
If you are interested,
if you want to get involved in this, uh, the forums.swift.org,
because it's a good place to reach us.
And if you want to try this package, we have a nightly preview that you can try out.
Um, yeah.
And if you're on FreeBSD 15, you'll need to install the compact package.
We've been doing development on FreeBSD 14.
But, uh, thank you.
Thank you.