Our next speaker, just admitted to me, he is not a go developer.
Wow, exciting.
People like that, you know, the go developer for some reason?
So why did I invite Dimash for you today?
Well, you don't build Go, oh well, we don't write Go, but you build it.
And this is most interesting, because those go binaries, they get them everywhere.
And we want to know how this is called, we've written very badly, and it's a positive
compiler.
A positive entry.
And yeah, hello, welcome.
My name is Dimitri.
I am an OS distribution developer.
So I've built lots of operating systems.
I am the Indian developer.
I'm a bunch of core developer.
I started Intel Clear Linux project currently working at Chingar Building WulfioS.
And I'm mostly focused on building production grade container images, such that all the
go code that I compile over a thousand projects, we deploy it in Kubernetes clusters in
production for ourselves and our customers.
And this talk I'm going to dwell into how we build all of the go projects that we ship.
And hopefully, such that other people adopt our best practices.
It will be Linux focused, right, such that majority of this applies to Linux binaries on
x86 and ARM.
However, most of it is transferable to go on other operating systems and architectures as
well.
But you may need to do your own testing.
Yeah, I've been doing this for a while, so hopefully you'll find something useful here.
First we'll start with simple things.
So by default, when you run go build, it will produce binaries with debug symbol attached.
I do want to ask you a question, who has here run GTB or DELF on a go binary to debug
it?
OK, that's a lot.
Keep your hand up if you've done it, well, the binary you're attaching to is running
in a production cluster.
OK, still a few hands.
OK, I want to posit that the debug information is often unused in production, right?
Such that when you build your production container, you can strip that.
So there is LD flags, dash W flag, which will remove debug symbols from the binary
you produce.
And if you want to keep a copy of your debug information, there is another option where
you can use strip and you can strip debug and store it in your debug info server, which
you are running locally.
Because then GTB can still connect, and you can still get all of your debug information
for the five people who connect to their production binaries, right?
They're saving on the binary stage.
It can be like 35, 40% of the binary size, such that you can potentially, you know,
half of all of your container sizes.
Next up, trim path.
So when you build binary, it encodes the file path of your build system inside the
binaries for majority of files that you're compiling.
It leaks the username, it leaks the file system layout of your build infrastructure into
the binary.
And if you strip that, you can save between like 2% and 5% of your binary size, and
you can also make your binary script reproducible and also be able to compile it on multiple
places and get the same hash every time.
It's a useful property.
It removes any extended attributes that you've set with LD flags from the build info file.
So there is an upstream bug about it.
They're hopefully going to fix it in the compiler, and we'll see if we do patch that out
because we want to still see all of the x flags that you pass to the binary and build info.
Okay.
How many people here disable Cgo when they build binaries?
Why?
So, static linking, that's very common ounces.
So back in the day, you have to pass it to get a static binary.
That's no longer the case.
By default, on Linux, go use this, a Glypsy, and name switch service to do a resolution
of DNS and the user names.
However, there is build tags, netgo, and OS usergo, which you can pass, and you can keep
Cgo and a vote, and you can still get static binary.
And then in your build info, it will also save those tags.
And then your build becomes the same whether you're cross compiling or natively compiling
the binary, which helps all of the developers who use like a MacBook to deploy to Linux
containers.
And you get compatibility with all of the FIPs toolchains and forks of official go-tool-chay,
such that you can compile the same project with boring crypto, go-open SSL, go-mikers
of FIPs.
If somebody ever tells you that you need FIPs compliance for your binary, because all
of those forks, they require Cgo to be an able to implement TLS and cryptography.
So I want to keep your projects behaving the same, still have static linking for the regular
go-tool-chay, but be able to use Cgo for accelerated code, optimized code, if you need it.
And there are two options for that.
There is a caveat.
If you do need name switch service resolution, then you sort of keep Cgo enabled, and
there is no mirror tags for that to opt into that behavior.
Who has production hardware, which is older than 20 years?
Excellent.
This slide is not for you three people.
You should set environmental variables to the highest API level for your production
great hardware, such that most people V2 is, if your hardware is newer than 20 years
old, if your hardware is like newer than 10 years, or you use latest instance types,
you can go to V3.
If you use AI ML only, you can go to V4.
Your binaries will run faster, and they'll have less code in them.
That's as simple as that.
And most Linux distributions, they'll rate you race the C, C++, minimum, and the I2, V2,
or V3.
However, they haven't done it for Go binaries.
So Go is starting to be slower than C, and I don't like that.
So export those environmental variables, and you'll get better performance out of your
binaries.
To do.
Next up, position independent executables encode.
So most Linux distributions in their toolchains, they enable hardening flags to make sure
that all binaries are linked with bind now, and position independent executables encode.
And they do it consistently for Rust, C, C++.
Not many people do it for Go, but they should.
For static binaries, it might not gain you much security, but it will not set off your security
scanners and your audits, because when security companies come into the audit of your deployments,
they will look for that, because lots of security scanners look at your whole container,
and they're like, okay, everything has position independent executables apart from the
five things that you are your entry point and your go binaries.
So you may want to enable it.
It does gain you security features if your go binary calls into C code, or if it has any
C go enabled, or if it has any hard work acceleration instructions.
You may not know this, but some of your dependencies may have assembly inside the Go binaries,
which can then be secured by the compiler.
It's an each thing.
Version numbers, who has difficulty tracking what binary is deployed in production, or
know what its version number is, a few people.
So when you run Go build, it doesn't bet VCS information into the binary, but it does not.
It does, but it encodes only the commit, shaw, and times them, but it doesn't include
the tag name.
If you run Go install, which you can for private builds with like Go private exported,
it will encodes the tag name, but not the commit, or the timestamp.
For that, I recommend to set a custom argument to your build environment where you get
described your tag, and then it will end up in the build information of your binary, because
then it doesn't matter if you go build or go install, you will have the same field and
build info, which you can then statically analyze and discover in all of your production
clusters.
C go hardening.
Who has set C go under course C flags in their life?
One, two, ooh, seven people, okay, that's good.
When you build C++ binary using like RPM build with RPM spec files, or in David, if
you use S build, normally they have lots of tooling, which will export really long C flags,
and LD flags and things like that, or they'll generate spec files, and all of that works
with like Mason and whatnot, but Go tool chain doesn't honor C flags environmental variable,
it has its own environmental variable code C go C flags, and none of the tooling out there
currently sets that by default, such that even if you package your software properly and
use a given OS vendors hardening flags, they will actually not be enforced for any C code,
which happens to be called by Go, and at that point, you need to reset those environmental
variables again, so if you detect in your binaries, that C goes enabled, then you need those
hardening options to be on, because otherwise your binary will be less secure and less
optimized as well, and for people who are interested in that instead of sending environmental
variables, I normally do wrappers around my GCC such that I can specify spec file elsewhere,
and I know it will propagate throughout my build systems, irrespective of what you use to compile
your code, because they all call GCC, and then I redirect and say, please set all of these things
for me, which are recommended by OpenSSF, so who has to deal with vulnerabilities in your
dependencies in the codes that you deploy? Who wants to not deal with that?
So, this is probably the best slide that I can do, so whenever I get the hello golfer
CML, I just panic, because usually it means that some big dependency library has vulnerability,
normally if you get an email two days earlier saying we will announce this next week, then you know
it's going to be like higher critical, and that there is a lot of reverse dependencies, right?
And at that point, you realize that suddenly, next week, you're stopping development,
you're going to upgrade your dependencies, rebuild everything and redeploy, right, because that's
what you need to prepare for. And there is a tool called GoVonCheck that you can run yourself,
and it is able to detect all of the CVs and all of your dependencies if you point a binary at it.
However, normally if you strip symbols out of your binaries, it will look at module level CVs only.
And that's not good enough, because despite you importing accident or despite you importing
accident HTML or something like that, you might not be using every single symbol, every single
functionality inside your binary, or inside your dependencies which you've pulled in, right?
And when vulnerabilities are announced, they're usually for a specific function call or a specific
feature. And all of the vulnerabilities canters without symbol level information, they will say
everything is vulnerable in your cluster container everywhere. However, if you keep symbols
information inside your binary, the vulnerabilities canters will be able to go into tech that,
yes, you're using vulnerable XNet. Now, you're not using the symbol that has security,
feature a security vulnerability inside of it, and we can skip patching and you don't need to
redeploy anything. So, for example, the previous email, which announced XNet vulnerability,
in WolfOS, we found that there was over a thousand binaries which imported that module, that package,
and then we found that like 200 of them had the symbols that used the HTML subcomponent
thing of XNet, and then because we keep symbol tables in all of our binaries, despite the
binary size, we were able to eliminate majority of them and close all of those city vulnerabilities
as code not present, because only 13 of them were actually using the vulnerable functionality.
And I can show you an example of this. So, like, if you go install Core DNS, 111 tag,
and if you run the go vulnerability checker on it, it will find 15 vulnerabilities.
If I do exactly the same release, and I keep the symbols information inside the binary,
despite it costing me 15% of the binary size, when I run vulnerabilities can on it, it will
automatically detect that, yes, the module input has two vulnerabilities or five or whatever,
but you're not using that functionality, and you don't need to patch or upgrade for that stuff.
So, I want to encourage everybody to keep symbols tables inside your binaries,
because that will help you later on to analyze and inspect whether your binary has
discovered vulnerabilities inside of it. Confused looks in the room. So, how do you do this?
This is the not slide, right? Do not use LD flags dashes, do not strip all.
If you look at the binary published by all major Linux distributions, they currently strip all,
and they currently remove all symbol tables from all of the go binaries, such that if you're using
go binaries that came from Linux distributions, a nameable debug symbol servers,
reattached debug symbols, and then rerun the vulnerabilities scanner that will help you to say
actually know the half or two thirds of vulnerabilities that are being detected by secure
it's scanner's false positive. To do do. There is small caveat, like in the previous slide,
it says it doesn't appear. If you craft your code to use reflection and to do certain tricky
things, there is a small percentage of cases where you're do not appear to use the symbol,
but it's linked in your binary, and you're actually using it via tricks and hacks. But if you have
very complicated go code, then you shouldn't be a go developer, and maybe try to write some
blur code. I have this slide of encouraging people to do this often. Many times people in production,
they run codes that was released a while ago. They're not on the latest release, not on the latest
point release. However, it would help a lot if you're able to continuously bump the go tool chain
stanza in your go mod. If you could bump all of your dependencies that have CVs resolved,
and if you would retag your upstream projects again. You don't have to make a huge announcement about it.
You don't have to write meaningless posts, PR statements, and whatnot, but it would help a lot for people
like the case with Core DNS. If they would have pushed out a micropoint release of V1.11.1.1.1,
right? Just to push out an extra release which uses a neural tool chain and uses neural dependencies,
which have all of these CVs resolved without changing the Core DNS project itself. That would help
all of the downstream users who still want to keep using that release, but they want to meet
to get CVs without changing anything else. Today, if you followed the best practices, and for example,
use Renovator depend about, normally it mitigates stuff for your master or main branch only.
It doesn't mitigate your stable branches or any of the tags. I hope that maybe somebody writes actions
to say, hey, I'm not changing any of your code. I only updating your go mod go sums, and I'm going
to help you retag all the releases to just keep maintaining them because that would help people.
Well, we do this a changer, but maybe upstream's want to do it as well.
To do it. I hope to contribute these things with examples, and with benchmark 2,
open SSF working group because they have an amazing hardening guide for CC++,
but they do not have something similar for go, and there is a lot of things that people should
be thinking about when they're building go binaries. At the front, I have some glitter stickers,
and I have about 10 minutes for questions or answers, or even I can go over some of these things
again, because some people look confused. Questions? I'm quickly going to him.
Very close to the microphone. You have to kiss it. Thanks. So, just a question. I'm coming from
OpenSusa, and you tell us to up the project to up the version number of the two things.
Yes. That's problematic. I just yesterday came about the project. I wanted to package my project
on devion, and that has go 115. So, what the hell? If you upgrade, all tell all projects to upgrade,
then I cannot package it for devion again. So, what to do? Excellent questions. So,
there is this conflict where security scanners look at the go version of the binary,
and the tool chain version that the binary was compiled with is encoded in every binary,
and it is the upstream version of go. So, it's like 1.15.2, for example, right? The latest point
releases 0.7, right? And devion, for example, cherry-picked security vulnerabilities,
and they have multiple releases of go 115.2, but there is no information in the compiled binary,
whether or not you use the vulnerable devion 115.2, or they are re-release of it, which they call
dash 2, or dash 3, or dash 4, or dash 5. I believe that we should fix upstream go tool chain
to enable vendors of the tool chain to set their own vendor version of the go compiler, which they
maintain for longer than upstream. Such that security scanners don't have to rely on the upstream
version number, such that they can say, oh, yes, it's go 115, whatever, but I see in VCS or go vendor
metadata or something that it was a patched version, because then we would be able to solve this
in a way that exposes information to the scanners and lets up things correctly. So, yeah.
Until then, please upgrade your tool chain standard, because that's the only way to go and see
that all of your downstreams and users will build production rate binary.
Yeah. One of your early slides was about ABI versions for architectures. Do you have
a way like some advice on how I can determine what the maximum save value I can use for that is for
my environments? How can I find out what all of my machine support? Yeah. If you have SSA checks
to all of your machines, if you run LD so dash dash help, it will actually print in the output
all of the ABI versions that are supported by your LD so tool chain, and it will say that, oh,
I know about V4 V3 V2, and it will also say in brackets, and your hardware supports this one.
If you can run that command across all of your clusters, all of your hardware, you can gather
that information and you can say actually everything apart from that one staging server supports V4
or supports V3, and then I can set it there. I did it for Wolfie by analyzing all of the
instance types and all of the big public clouds that all of our customers use, because then I'm like,
oh, I can go to V2 and in a year or two, I'll be able to go to V3 once a few deprecated instance
types get eliminated from public clouds. I should publish that. Okay, I will publish that.
Yeah, question. Last question? So in case of GoVone check, it technically is possible to run
it not in mode binary, but also in mode on analyzing the source code itself, in which case
you probably run and before you can buy the binary. Would the same set of problems basically
be solved if you run it on the sources? So you would basically not be, you will be able to strip
the symbols in that case, right? Correct. Correct. You will, if you have source code locally,
you yourself will be able to run it. However, the people who run your binaries may not be the people
who have the source code. If you distribute your application to run on prem or if you offer it as a
software as a service where people deploy your code elsewhere, they may not necessarily have all
of the Go code nor want to check it out. And also the symbols that end up in your binary
are dependent on the toolchain you compiled it with. So source code will analyze it against your
current compiler, not the compiler that was used in the past to build the binary that's running
in production. So there is small sliver of things that will show discrepancies.
While I still have all your attention and your very silent, I want to remind you that you don't
have to be a go developer to give a talk here. And there is still a chance that you can now
decide, I want to give a talk here. And we have space for you the last hour of the day,
we will have eight minute lightning toss. You can submit them at gofirst.love slash lights.
There are QR codes around the room to scan and submit them. But he proved that you did,
don't have to be a go developer to give an amazing talk and that's why I asked an extra lot of the
post.