[pacman-dev] [RFC PATCH] makepkg: extend the .BUILDINFO for enhanced reproducible support
Encode information about the compiler and strip flags used, as these will impact the resulting generated binaries. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- See https://kanboard.archlinux.org/public/task/40/7359bddc6311e0c07c31e421b6b496... for details. I've left out MAKEFLAGS as I'm unsure how this would actually affect the build. Also the various COMPRESS* options I simply don't know what to do with since we periodically add new ones and this results in updating them all over the place. Plus this isn't typically modified anyway and e.g. multithreaded compression tends to not be reproducible between two identical runs regardless, so I'm not sure we should be solving it... scripts/makepkg.sh.in | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 548feb91..3d6277d1 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -680,7 +680,7 @@ write_pkginfo() { write_buildinfo() { msg2 "$(gettext "Generating %s file...")" ".BUILDINFO" - write_kv_pair "format" "1" + write_kv_pair "format" "2" write_kv_pair "pkgname" "$pkgname" write_kv_pair "pkgbase" "$pkgbase" @@ -699,6 +699,13 @@ write_buildinfo() { write_kv_pair "builddir" "${BUILDDIR}" write_kv_pair "buildenv" "${BUILDENV[@]}" write_kv_pair "options" "${OPTIONS[@]}" + write_kv_pair "cppflags" "$CPPFLAGS" + write_kv_pair "cflags" "$CFLAGS" + write_kv_pair "cxxflags" "$CXXFLAGS" + write_kv_pair "ldflags" "$LDFLAGS" + write_kv_pair "strip_binaries" "$STRIP_BINARIES" + write_kv_pair "strip_shared" "$STRIP_SHARED" + write_kv_pair "strip_static" "$STRIP_STATIC" local pkglist=($(run_pacman -Qq)) local pkginfos="$(LC_ALL=C run_pacman -Qi ${pkglist[@]})" -- 2.18.0
On 19/07/18 02:12, Eli Schwartz wrote:
Encode information about the compiler and strip flags used, as these will impact the resulting generated binaries.
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> ---
Fairly sure we already rejected this... The assumption is that a distribution is using the default makepkg.conf options unless they are explicitly changed in the PKGBUILD.
See https://kanboard.archlinux.org/public/task/40/7359bddc6311e0c07c31e421b6b496... for details.
I've left out MAKEFLAGS as I'm unsure how this would actually affect the build.
Also the various COMPRESS* options I simply don't know what to do with since we periodically add new ones and this results in updating them all over the place. Plus this isn't typically modified anyway and e.g. multithreaded compression tends to not be reproducible between two identical runs regardless, so I'm not sure we should be solving it...
scripts/makepkg.sh.in | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 548feb91..3d6277d1 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -680,7 +680,7 @@ write_pkginfo() { write_buildinfo() { msg2 "$(gettext "Generating %s file...")" ".BUILDINFO"
- write_kv_pair "format" "1" + write_kv_pair "format" "2"
write_kv_pair "pkgname" "$pkgname" write_kv_pair "pkgbase" "$pkgbase" @@ -699,6 +699,13 @@ write_buildinfo() { write_kv_pair "builddir" "${BUILDDIR}" write_kv_pair "buildenv" "${BUILDENV[@]}" write_kv_pair "options" "${OPTIONS[@]}" + write_kv_pair "cppflags" "$CPPFLAGS" + write_kv_pair "cflags" "$CFLAGS" + write_kv_pair "cxxflags" "$CXXFLAGS" + write_kv_pair "ldflags" "$LDFLAGS" + write_kv_pair "strip_binaries" "$STRIP_BINARIES" + write_kv_pair "strip_shared" "$STRIP_SHARED" + write_kv_pair "strip_static" "$STRIP_STATIC"
local pkglist=($(run_pacman -Qq)) local pkginfos="$(LC_ALL=C run_pacman -Qi ${pkglist[@]})" -- 2.18.0 .
On 07/18/2018 06:07 PM, Allan McRae wrote:
On 19/07/18 02:12, Eli Schwartz wrote:
Encode information about the compiler and strip flags used, as these will impact the resulting generated binaries.
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> ---
Fairly sure we already rejected this... The assumption is that a distribution is using the default makepkg.conf options unless they are explicitly changed in the PKGBUILD.
I don't recall where we rejected this? Anyway, reproducibility is not just about reproducing packages created by devtools. It would be nice to be able to reproduce any package. :) It's also been noted on IRC that we could use it to detect packages built with an *old* makepkg.conf in devtools chroots. -- Eli Schwartz Bug Wrangler and Trusted User
On 19/07/18 08:15, Eli Schwartz wrote:
On 07/18/2018 06:07 PM, Allan McRae wrote:
On 19/07/18 02:12, Eli Schwartz wrote:
Encode information about the compiler and strip flags used, as these will impact the resulting generated binaries.
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> ---
Fairly sure we already rejected this... The assumption is that a distribution is using the default makepkg.conf options unless they are explicitly changed in the PKGBUILD.
I don't recall where we rejected this?
Anyway, reproducibility is not just about reproducing packages created by devtools. It would be nice to be able to reproduce any package. :)
It's also been noted on IRC that we could use it to detect packages built with an *old* makepkg.conf in devtools chroots.
Then you need to include all relevant environmental variables too. And given we don't know which are relevant, we need to include all. Which had privacy implications. Assumptions need to be made for reproducibilty. I'm happy with the package being built in a clean chroot as that assumption. A
Then you need to include all relevant environmental variables too. And given we don't know which are relevant, we need to include all.
I'd assume that the variables defined on the makepkg.conf should be the only relevant ones, otherwise the package is not reproducible ;) On the other hand, I do see a point in claiming that /etc/makepkg.conf is part of the toolchain, and as such it should be used to bootstrap a reproducible env. However, I do think that the intent of the buildinfo are well documented in their docs[1](emphasis mine): Absolutely necessary “human intent” embedded certificates if needed (rpm + tor windows) source pkg. version source pkg hash (contents) source package name architecture (target) (GNU host) build instruction (deb-implicit) __USE flags (gentoo). debian: build profile. build time configuration**__ build-depends. Abstract description of some tool that (fully) defines Build-Depends. Source Name/version Build-depends. Source packages’ HASH!!! I can see how Eli's patch can help beyond achieving the devtools environment and simplify any overlay tooling around it.
Which had privacy implications.
If you mean recording *all* the variables then I agree, but I don't think anyone is proposing this.
Assumptions need to be made for reproducibilty.
Likewise, but I believe that assumptions can be discussed to reach a consensus on what these assumptions should be. Thanks, -Santiago. [1] https://reproducible-builds.org/events/athens2015/buildinfo-content/
On 07/18/2018 06:26 PM, Allan McRae wrote:
Then you need to include all relevant environmental variables too. And given we don't know which are relevant, we need to include all. Which had privacy implications.
Assumptions need to be made for reproducibilty. I'm happy with the package being built in a clean chroot as that assumption.
I'm okay with makepkg only recording the information it is personally responsible for setting in the first place. :) That's what my patch does. If people are creating packages in such a way that the environment outside of makepkg affects the result, then there's really nothing that can handle that -- a thousand different tools have a thousand different boutique configuration files, for example. As long as both packages are built in, and record, some sort of environment where any input to the build process comes exclusively from: - the list of packages installed on the system - the PKGBUILD - makepkg.conf - the public API documented in makepkg(1) -- essentially, BUILDDIR and PKGEXT I will be happy, since makepkg has "done its duty" as far as reproducibility goes. The best way to ensure this is to build packages using a clean user account, but I don't think devtools should be implied, nor should makepkg itself consider reproducible build support to be conditional on devtools/makechrootpkg. -- Eli Schwartz Bug Wrangler and Trusted User
participants (3)
-
Allan McRae
-
Eli Schwartz
-
Santiago Torres