Guidelines for reproducible Go binaries (2026)
hello, I joined as co-maintainer for syft after the recent todo item[1] and modernized the PKGBUILD. Based on the repro-get package[2] (which did reproduce successfully) I've changed the syft build instructions to this: ``` export CGO_CPPFLAGS="${CPPFLAGS}" export CGO_CFLAGS="${CFLAGS}" export CGO_CXXFLAGS="${CXXFLAGS}" export CGO_LDFLAGS="${LDFLAGS}" export CGO_REQUIRED="1" mkdir -p build go build \ -o build/ \ -buildmode=pie \ -mod=readonly \ -ldflags "-compressdwarf=false -linkmode=external -X main.version=${pkgver}" \ ./... ``` This did not reproduce in our verification system however: ``` │ ├── usr/bin/syft │ │┄ File has been modified after NT_GNU_BUILD_ID has been applied. │ │ ├── readelf --wide --notes {} │ │ │ @@ -1,16 +1,16 @@ │ │ │ │ │ │ Displaying notes found in: .note.gnu.property │ │ │ Owner Data size Description │ │ │ GNU 0x00000010 NT_GNU_PROPERTY_TYPE_0 Properties: x86 ISA needed: x86-64-baseline │ │ │ │ │ │ Displaying notes found in: .note.gnu.build-id │ │ │ Owner Data size Description │ │ │ - GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: 5d4cf47367358962632d3bfd04530c513f897790 │ │ │ + GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: dd9283dbad8ad8c4b742f7ac88f6a0e1f3b9acc1 │ │ │ │ │ │ Displaying notes found in: .note.go.buildid │ │ │ Owner Data size Description │ │ │ - Go 0x00000053 GO BUILDID description data: 42 49 56 6f 41 61 39 67 43 50 4e 49 6e 37 70 49 55 2d 76 64 2f 5f 6b 4a 66 6e 31 37 68 6e 4b 42 65 42 66 62 6f 39 74 4d 4b 2f 59 39 66 41 33 4d 74 7a 69 66 56 71 4b 4a 42 42 68 78 7a 57 2f 42 50 73 44 75 79 76 55 49 4c 37 30 47 37 73 6e 67 6a 55 73 │ │ │ + Go 0x00000053 GO BUILDID description data: 50 44 71 65 63 37 36 71 30 4e 52 69 4b 36 69 57 63 31 73 52 2f 5f 6b 4a 66 6e 31 37 68 6e 4b 42 65 42 66 62 6f 39 74 4d 4b 2f 59 39 66 41 33 4d 74 7a 69 66 56 71 4b 4a 42 42 68 78 7a 57 2f 73 6a 72 50 70 62 38 39 32 41 4c 48 4e 62 6c 4d 72 75 4d 46 │ │ │ │ │ │ Displaying notes found in: .note.ABI-tag │ │ │ Owner Data size Description │ │ │ GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag) OS: Linux, ABI: 4.4.0 │ │ ├── strings --all --bytes=8 {} │ │ │ @@ -1,8 +1,8 @@ │ │ │ -BIVoAa9gCPNIn7pIU-vd/_kJfn17hnKBeBfbo9tMK/Y9fA3MtzifVqKJBBhxzW/BPsDuyvUIL70G7sngjUs │ │ │ +PDqec76q0NRiK6iWc1sR/_kJfn17hnKBeBfbo9tMK/Y9fA3MtzifVqKJBBhxzW/sjrPpb892ALHNblMruMF │ │ │ /lib64/ld-linux-x86-64.so.2 │ │ │ __libc_start_main │ │ │ __cxa_finalize │ │ │ __vfprintf_chk │ │ │ sigfillset │ │ │ pthread_sigmask │ │ │ pthread_attr_init │ │ ├── readelf --wide --decompress --hex-dump=.gnu_debuglink {} │ │ │ @@ -1,4 +1,4 @@ │ │ │ │ │ │ Hex dump of section '.gnu_debuglink': │ │ │ - 0x00000000 73796674 2e646562 75670000 ca77ec4e syft.debug...w.N │ │ │ + 0x00000000 73796674 2e646562 75670000 6b0247ba syft.debug..k.G. ``` From what I understand both build IDs are hashes over other parts of the binary. Since they are the only difference though, the problem is likely in the debug symbols that get detached after the compile (and also after this hashing is done), but before diffoscope is executed on the final release artifact. Our guidelines for Go[3] mention the -trimpath option (which I forgot to add, but also wasn't necessary for repro-get), but since we have normalized paths in our build containers I'm not sure if the claim about Reproducible Builds applies to non-AUR packages? Using the data from reproducible.archlinux.org, there's currently 371 packages that depend on our Go package and have some kind of status attached to it: % rg -c GOOD resolved-go.txt 240 % rg -c BAD resolved-go.txt 131 Which is about ~65% In comparison, although many packages still use the `rustc --print` approach, our Rust build instructions essentially boil down to: ``` cargo fetch --locked --target host-tuple cargo build --frozen --release ``` Those two commands with no further manually set environment variables gets us: % rg -c GOOD resolved-rust.txt 669 % rg -c BAD resolved-rust.txt 86 Which is ~89% Although 20 percent-points doesn't seem like that much, it's a difference of "every 1 out of 3" vs "every 1 out of 10". Is there anything we can do to improve this overall situation? Both the amount of complexity necessary to compile a simple binary, as well as the reproducible build success ration? Maybe we could have something like /etc/makepkg.conf.d/rust.conf but for Go? cheers, kpcyrd [1]: https://archlinux.org/todo/rebuild-packages-signed-by-anatolik-morganamilo-s... [2]: https://gitlab.archlinux.org/archlinux/packaging/packages/repro-get/-/blob/m... [3]: https://wiki.archlinux.org/title/Go_package_guidelines
On Wed, Feb 04, 2026 at 01:42:03PM +0100, kpcyrd wrote:
hello,
Yo,
I joined as co-maintainer for syft after the recent todo item[1] and modernized the PKGBUILD.
Based on the repro-get package[2] (which did reproduce successfully) I've changed the syft build instructions to this:
``` export CGO_CPPFLAGS="${CPPFLAGS}" export CGO_CFLAGS="${CFLAGS}" export CGO_CXXFLAGS="${CXXFLAGS}" export CGO_LDFLAGS="${LDFLAGS}" export CGO_REQUIRED="1"
mkdir -p build go build \ -o build/ \ -buildmode=pie \ -mod=readonly \ -ldflags "-compressdwarf=false -linkmode=external -X main.version=${pkgver}" \ ./... ```
This did not reproduce in our verification system however:
[SNIP]
From what I understand both build IDs are hashes over other parts of the binary. Since they are the only difference though, the problem is likely in the debug symbols that get detached after the compile (and also after this hashing is done), but before diffoscope is executed on the final release artifact.
You've already figured out the issue but I'll explain it in detail for the benefit of the list. The issue here is that you didn't include `option=(!lto)` in the package file. Go produces a BUILD_ID by hashing all the object files during building and stamping them in the final build. This is reproducible for the Go compiler and works just fine. However we are enabling the `cgo` compiler backend. Go has two compilers and two linkers you can use and mix-match. The `go` backend is a full Go compiler, and the `cgo` backend produces C code which calls `gcc`. You then have the internal linker which is a Go program, and the external linker which is the usual `ldd`. The fundamental problem we have here is that we want `checksec` to pass with all hardening options, which the Go compiler with the internal linker simply does not do for us. This means we are explicitly enabling `cgo` and the external linker to give all Go binaries the hardening features we want in our distro. This makes building Go complicated, because `cgo` is complicated. The reason why `lto` needs to be disabled is because there is a `gcc` bug in the LTO streamer backend where bare `#line` macros get prepended a temporary directory which does not get stripped by the `prefix-strip` flag. I tried to propose a patch to the go compiler, and to gcc, but all of this has stalled. https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647303.html https://github.com/golang/go/pull/53528 I've spent several weekends working through all the this and I'm generally unhappy about the state of things. If we decide that we just follow the Go compiler and ignore `checksec`, all of this would be a lot simpler for us. I'm also unsure how much all the binary hardening options actually matter for the Go compiler considering we have a GC-runtime in play here. Note that the upstream does not give any reproducible builds guarantees for the `cgo` compiler setup, so I assume something is going to regress in the future. I have several patches in the Go compiler to resolve `cgo` issues already.
Our guidelines for Go[3] mention the -trimpath option (which I forgot to add, but also wasn't necessary for repro-get), but since we have normalized paths in our build containers I'm not sure if the claim about Reproducible Builds applies to non-AUR packages?
Correct, I have been lazy and not updated the guidelines to remove this option. I have largely been waiting for the Developer handbook before doing a refresh of the package guidelines. Note that including `-trimpath` also breaks debug packages for Go packages.
[SNIP]
Maybe we could have something like /etc/makepkg.conf.d/rust.conf but for Go?
Yes, I have been waiting on drop-in support for makepkg. Jelle started working out something, and I think we need this for the ports RFC anyway. I have simply not followed up on this. I still don't think it would solve a lot as the interplay between `GOFLAGS` and `go build` CLI flags do not play nicely with each other. So the packager would need to understand the interplay here, sadly. For the sake of progress; I plan on writing up a new revision of the Go package guidelines into the new handbook. And then I'll get to the `go` makepkg configurations at some point. -- Morten Linderud PGP: 9C02FF419FECBE16
With Foxboron's help I've disabled LTO and the package now reproduces (thanks!) I also wrote a simple wrapper that: - passes our buildflags to CGO - sets all arguments necessary for reproducible builds - without giving up hardening or debug symbols - fails the build if an option was detected that would cause issues (LTO) Before: ``` makedepends=('go') build() { cd "${pkgname}-${pkgver}" export CGO_CPPFLAGS="${CPPFLAGS}" export CGO_CFLAGS="${CFLAGS}" export CGO_CXXFLAGS="${CXXFLAGS}" export CGO_LDFLAGS="${LDFLAGS}" export CGO_REQUIRED="1" go build \ -o build/ \ -trimpath \ -buildmode=pie \ -mod=readonly \ -modcacherw -ldflags "-compressdwarf=false -linkmode=external -X main.version=${pkgver}" \ ./... } ``` After: ``` makedepends=('repro-go') build() { cd "${pkgname}-${pkgver}" repro-go build \ -o build/ \ -modcacherw \ -X main.version="${pkgver}" \ ./... } ``` The shortest possible invocation (if the project setup allows), yet with all binary hardening, reproducible builds options and debug symbols enabled: ``` build() { cd "${pkgname}-${pkgver}" repro-go build } ``` It's implemented in ~120 lines of code and then 2x that amount in unit-tests. The -X option is not part of -ldflags for technical reasons, the tool already needs to set linker flags, and having to share this with an application configuration interface is very clunky. I also obviously consider this a stop-gap solution, not something that should exist in the long run. In the meantime it may be useful for one packager or another who struggles to get the flags right (even if just as a resource what to set). I'm still interested in setting these by default through e.g. /etc/pacman/makepkg.conf.d/go.conf. cheers, kpcyrd
Please stop dropping random tooling into our repos with 0 consensus and encourage it's use. This is not the first time you've have done this. I'm not dealing with the consequences here. This is not helping anyone and just giving me a bunch of future problems. -- Morten Linderud PGP: 9C02FF419FECBE16
On 2/4/26 8:32 PM, Morten Linderud wrote:
Please stop dropping random tooling into our repos with 0 consensus and encourage it's use. This is not the first time you've have done this.
I'm not dealing with the consequences here.
This is not helping anyone and just giving me a bunch of future problems. I appreciate your effort on gcc (I really do!) but please don't problem-squat and put down other people's work if you don't have an alternative solution ready (e.g. rolling out /etc/makepkg.conf.d/go.conf in the imminent future).
There is similar tooling in the openSUSE project that we also carry in our repository since exactly 1 year: https://archlinux.org/packages/extra/any/reproducible-faketools/ We have a similar wrapper for meson at /usr/bin/arch-meson, they are both very explicitly opt-in, and I'm absolutely happy to drop mine once it has been obsoleted, but not before. In the past 2 years I have been the only person pushing this topic[1] (again, I do acknowledge and appreciate the work you do/did on gcc+go around the same time, but it only solves the LTO problem, not the build flag situation). [1]: https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.or... cheers, kpcyrd
On Wed, Feb 04, 2026 at 09:18:01PM +0100, kpcyrd wrote:
On 2/4/26 8:32 PM, Morten Linderud wrote:
Please stop dropping random tooling into our repos with 0 consensus and encourage it's use. This is not the first time you've have done this.
I'm not dealing with the consequences here.
This is not helping anyone and just giving me a bunch of future problems.
I appreciate your effort on gcc (I really do!) but please don't problem-squat and put down other people's work if you don't have an alternative solution ready (e.g. rolling out /etc/makepkg.conf.d/go.conf in the imminent future).
Introducing new tooling, written in Rust, for an ecosystem I've spent years on without as much of ping is not how this works. I'm not going to spend my time helping people doing Go packaging work if I have to figure out wrappers others have written. This could litterally have been a 10 line shell script if we actually needed it.
There is similar tooling in the openSUSE project that we also carry in our repository since exactly 1 year:
https://archlinux.org/packages/extra/any/reproducible-faketools/
Not a single package is using this, and you pushed this.
We have a similar wrapper for meson at /usr/bin/arch-meson, they are both very explicitly opt-in, and I'm absolutely happy to drop mine once it has been obsoleted, but not before.
`arch-meson` is a 1 line bash script maintained by the same person that maintains the meson package. https://gitlab.archlinux.org/archlinux/packaging/packages/meson/-/blob/main/... This script itself is already controversial and people are not sure if this is what we want.
In the past 2 years I have been the only person pushing this topic[1] (again, I do acknowledge and appreciate the work you do/did on gcc+go around the same time, but it only solves the LTO problem, not the build flag situation).
[1]: https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.or...
I don't see any edits from you here: https://wiki.archlinux.org/index.php?title=Go_package_guidelines&action=hist... Nor here: https://gitlab.archlinux.org/archlinux/packaging/packages/go/-/commits/main -- Morten Linderud PGP: 9C02FF419FECBE16
participants (2)
-
kpcyrd
-
Morten Linderud