Hello, I'm Yuan, and I work on swift services and infrastructure
at Apple.
And what that means is I often find myself building swift programs and putting them
in containers.
And so now I maintain a little package called swift container plug-in.
It's funny how that kind of thing happens.
We're living in the cloud native era, and that's changed not only how we run our services,
but also how we build and deploy them.
So let's see, I'm ready to deploy my service in the cloud.
It used to be that I would compile my binary and then FTP it up onto a server, and that was it.
But that's no good anymore, because my cloud provider runs containers.
The binary alone isn't enough.
And what that means in practice is that I have to wrap it in a container image.
That's a standardized packaging format that holds my application, and then any supporting
files that's going to need at runtime.
Once I have my image, I can publish it to an image store in the cloud called a registry.
You've heard of Docker Hub, but your cloud provider probably runs its own registry, and
if not, you can host self-host.
But the beauty of publishing a container image in a registry is that once it's up there,
I can deploy it to any container-based cloud provider I like, whether it's in the public cloud
or in a private data center.
But the part that particularly interests me is how these container images get built in the first place,
because that's the part that I deal with on a daily basis.
Containers are now part of my development cycle, or let's face it's really more of a debug cycle.
Because whenever I want to build or test my service, I have to build a container image,
and I do it over, and over, and over, and over again.
And it kind of turns my workflow inside out, because conventional container tools want
to manage the whole process for me.
The way I see it in this cloud native era, my final product is the container image,
and I'd really like my familiar development tools to be able to produce it for me.
And so the question is, can sort of package manager do it?
Well, it turns out that it can, but we'll need to pull off a couple of tricks to make it happen.
But don't worry, I'm not a member of the Magic Circle, I'm not sworn to secrecy,
and I'll let you know all details.
So, OK, so here we have just a little hello world.
It's very simple, but it has everything that we need.
One thing it doesn't have that you might be expecting to see is a container file.
You might call it a Docker file or a podman file or something like that.
But whatever you do call it, it's a recipe that tells a container runtime
how to build a container image, where we are going, we won't need one.
Because we're not using a container runtime to build this image.
Swift package manager to do it all by itself, using the Swift container plugin.
So to get started, I need to install the plugin, like I said.
And now I'm ready to build my image.
The plugin is asking for network permission, because on MacOS,
plug-ins run in a really tightly constrained sandbox environment,
and it needs network permission so that it can push the image up to the registry.
I'm running a local host registry, but it's still a network call.
So what it's done is it's built my service, and it's packaged it up into a container image,
and printed this identifier, and now we can run it.
Service running, I can color it.
Great.
But thank you.
It could be more topical, don't you think?
So let's just make a quick change.
Oops.
World, right.
And we'll go around the dev cycle one more time.
And the great thing about letting Swift package manager drive the build process
for the container is that we should get incremental builds for free.
If you've ever struggled with a container build that just insisted on rebuilding the world
for no reason you could, you could possibly figure out, particularly in the middle of a demo,
then you know what I'm talking about.
So what we've done is we've rebuilt.
We've got another container image reference.
I run it, hit the endpoint again, and that's much better.
OK.
So did you spot the tricks?
They were all behind this single command, and I promise you it is a single command.
It's just been line wrapped.
Check number one is to build the server binary.
And that doesn't really sound so difficult.
Except, remember that I'm running on macOS, but we built a container image, which deployed
on Linux, except it was coming from Ubuntu.
And that must mean that we've built a Linux binary at some point in that container image.
And that's a bit more magical.
And the second trick is building the container image.
And we did it from first principles.
We're building completely organic free range container images, and so let's see how it was done.
So we'll start with the server binary.
And the conventional way to build one to build a container image is to use a container run
time on your laptop or desktop.
And it's not always obvious, but when you do that, when you build a container that way,
you're actually doing your build on Linux itself, even if like me, you're running macOS locally.
And that's because behind the scenes, your container run time is running a Linux virtual machine.
It reads the container build recipe.
And that's that file that we didn't need a moment ago with the container plugin.
It starts up a container build inside the VM.
It copies in your code.
And then in there, it runs the Linux version of the Swift compiler.
In a full Linux environment, and that's how it's able to build a Linux binary to put in the container.
But there's another way that we could build that Linux binary, using the native Swift compiler
that we already have.
And that's because Swift has always been great for cross compilation.
The Swift compiler is based on LLVM, which gives it the ability to compile for lots of other
platforms.
But a cross compiler isn't quite enough on its own, because to produce viable binaries,
we need to be able to link against the runtime libraries and other files for the target system.
And that's where Swift SDKs come in.
They provide supporting files that we need to cross compile to Linux.
And even to different processor architectures using the open source Swift tool chain.
And cross compilation is just as useful if you're running on Linux.
Maybe you have an X86 laptop and you want to deploy an ARM.
Well, a Swift SDK will help you to do that.
Earlier on, we built our binary using the static Linux SDK from Swift.org.
So let's take a closer look.
Okay, so I've already installed the static Linux SDK.
And because it depends on the open source Swift tool chain, I've set that up as well.
But that's everything I need to cross compile.
I'll just create a little test project and build it.
So I'm passing the Swift SDK argument to Swift build to ask it to cross compile using my SDK.
The output is kind of underwhelming.
It looks just like what you would normally expect.
But what's a bit more impressive is we ended up with Linux binary.
And it's a statically linked one as well.
Now, statically linked binary is a great because they have very few runtime dependencies.
They're self-contained.
But they may not always be what you want because they can be a little bit bigger than dynamic linked
binary, precisely because they're being all that dependencies with them.
And you also might want to use some libraries that are available on your Linux system that aren't in the
SDK, you can get it from Swift.org.
And if that's the case, don't worry, we have you covered.
You can build yourself custom SDK tailored to your own requirements using project called the Swift SDK Generator.
It's also available on GitHub and I have it here.
So let's get it to build us an SDK based on the official Swift 603 image.
The first time you run this, it'll have to download quite a lot of data,
but it'll cache it so it only happens once.
And what it's doing behind the scenes is it's extracting out of that container image just enough
of the Linux Swift environment to support the micro, the cross compiler running on my QS.
So it's run and it's produced us a bundle, but before we install a bundle, let's just take a
look and see what we're dealing with here. It's quite a long path, I think this is what they
mean when they talk about a deep dive. But if you keep digging down, you'll find some directory
start the existing names at loop familiar if you've done any development on Linux. And if we go
a bit further, here are the dynamically linkable or dynamically libraries that support various
familiar looking Swift libraries. Okay, so let's install this thing and try and build.
So I install my SDK, come back up here and I should have a new SDK in my list, which I do.
And this time, I will pass a slightly different identifier to that Swift SDK parameter.
I get another really, they could put more fanfare into this because it's doing something
quite interesting. Another fairly, fairly vanilla output, but this time, we have a dynamically
linked Linux binary using the SDK that we just built. Okay. So that's how to build a Linux binary,
using the Swift compiler that we already have on macOS. The second trick is to build our container
image. But what is a container image really? There are a bit of a mystery because they're usually
managed for you behind the scenes. They're a bit like the Yeti. You see their footprints as
they're pushed and pulled around. You type some container command and a progress bar
creeps across the screen, but you rarely see one close up. You don't have a container image
sitting in your project build directory, usually. Well, actually it turns out that image are
pretty straightforward inside, possibly also like Yeti's. And because they're an open standard,
which was originally defined by Docker and then donated to the Linux Foundation,
there are now lots of tools which work with them and we can equally build our own tools in Swift.
So inside the image, there's a tree of objects and to keep it simple, we'll look at a single
architecture image here and that's actually the thing called a manifest. That points to some
configuration which tells the container runtime how to run the image and it also points to a stack of
layers which combine together to make up the file system. The metadata objects are Jason
and the layers are turbals and so this is something a structure that we can easily build in Swift.
And since one of the, I think one of the best ways to understand things is to take it apart,
even better if I can boot it back together again, let's see if we can find one of these creatures
and take a closer look. So, over here, right, so the registry server is just an HTTP server,
so we can fetch a container image by hand using curl. We need to start with the manifest,
at the top of the tree and work out from there. So here's our configuration blob,
we'll fetch that and it's Jason. So as you can see, this is the only information the runtime will
use to run this image and the most important thing is probably this entry point. That's the binary
that you should run inside the container when it starts up. So remember that name. The rest of it is the
file system and this is our ball and so this is our, this first layer is our basic underlying
Linux image and as you can see it's based on Ubuntu. There's lots of Linux stuff going on here,
lots of files etc. We page down, we'll find eventually binaries that you'd expect to see on
the Linux machine and further down there are libraries that we don't need to go that far.
The next layer isn't terribly interesting, it just adds on the CA certificates that this container
image would trust at start up, but this layer here is really important for us because this is where
we put Swift into the mix. So you can see here's the Swift system, you've got Swift module,
you're seeing already and if we come down further, here are those Linux.so files that we saw
moment ago in the SDK. So this is the layer that takes our generic Linux image and specializes
it into a Swift image. The only thing we require now is our binary and that's in the last layer.
So there's our hello world and that's how you take a container image apart. What about
putting it back together? Well, like so many things in life, we just need to retrace our steps
in the opposite order and we could do it all using curl, but there's a program in the container
plugin repository that does it for us in one hit. So we'll do that. What this is doing is it's
taking our hello world binary, it's constructing a layer around it, stacking it on top of the Swift
slim image by default but you can choose your own base image and then uploading it to the registry
in this new address. We have yet again an image reference, we can run it, it starts up and there's
our server. Okay, so you've seen all the tricks. We built a Linux binary by cross compilation and then we
packaged it in a container image. We've just got time, I hope we've just got time, five minutes.
To mention the unsung hero of our little show, the magicians are system stitching everything behind
this together behind the scenes. That is Swift package manager plugins.
Plugins license extends with package manager to suit our own requirements by adding our own commands
and build steps which look just like built-in features. When we ran this Swift command earlier,
what I actually happened behind the scenes was that Swift package manager started up and it spawned
our plugin, passing it its own set of arguments. And the plugin then called back in to Swift
package manager to say, please would you build the hello world target using our SDK. Swift package
manager obliged and it gave us a Linux binary back, which the plugin then wrapped in a container
image using container tool just like we did a moment ago by hand. That uploaded it to the
image to the registry and then returned an image reference and Swift package manager printed it on
the terminal ready to pipe into the next command. So Swift package manager plugins are the last
little trick that lets the Swift tool chain build a brilliant explainery on macOS package in a container
image and upload it to a registry all in one command. Hopefully we've also de-missed to find a
couple of things along the way. We looked inside an SDK and found that it's really a bundle of
Linux resources for the Swift cross compiler to link against. And we've also tracked down a yeti,
I mean it's container image and we discover that it's not really all that mysterious. It's just
some turbos and fancy dress. You can find all of this in the Swift container plugin repository on
GitHub. But what I hope you've also seen is that this is just one of the tricks that you can
conjure up with the tools and the Swift tool kit. You can use it as is to build container images
for your projects. It's particularly well suited for iterative local development like we saw at
the beginning. Or for places where you can't easily run a full container on-time like in some
CI systems. It's been tested with several popular registries. Please try it with your own favorite
and let us know how it goes. But remember it's a tool kit and you can use it to build your own
custom container tools designed to suit the way you work with containers. Take it apart but please do
put it back together again and see what tricks you can come up with. I'll be available in the whole
way after this if you'd like to talk about some of your ideas. Thank you very much.