[pacman-dev] [PATCH] makepkg: add tool details to buildinfo to aid determining flags
From: Levente Polyak <anthraxx@archlinux.org> If a makepkg consumer uses a build wrapper to override compiler flags this may lead to unreproducible packages as there is no way to know which exact files were used for tooling that tries to reproduce said package. Instead of vendoring the whole used makepkg.conf file into buildinfo, this patch adds two new properties to the .BUILDINFO file named BUILDTOOL and BUILDTOOLVER which by default are simply makepkg's own values. Downstream consumers may override those values: For example in Arch Linux the devtools package can set those values and allow reproducible builds tooling to fetch the appropriate makepkg.conf. Signed-off-by: Levente Polyak <anthraxx@archlinux.org> --- scripts/makepkg.sh.in | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 92cb6398..e58edfa1 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -651,6 +651,8 @@ write_buildinfo() { write_kv_pair "builddate" "${SOURCE_DATE_EPOCH}" write_kv_pair "builddir" "${BUILDDIR}" write_kv_pair "startdir" "${startdir}" + write_kv_pair "buildtool" "${BUILDTOOL:-makepkg}" + write_kv_pair "buildtoolver" "${BUILDTOOLVER:-$makepkg_version}" write_kv_pair "buildenv" "${BUILDENV[@]}" write_kv_pair "options" "${OPTIONS[@]}" -- 2.31.1
On Thu, Apr 22, 2021 at 09:46:56PM +0200, anthraxx@archlinux.org wrote:
From: Levente Polyak <anthraxx@archlinux.org>
If a makepkg consumer uses a build wrapper to override compiler flags this may lead to unreproducible packages as there is no way to know which exact files were used for tooling that tries to reproduce said package.
Instead of vendoring the whole used makepkg.conf file into buildinfo, this patch adds two new properties to the .BUILDINFO file named BUILDTOOL and BUILDTOOLVER which by default are simply makepkg's own values. Downstream consumers may override those values: For example in Arch Linux the devtools package can set those values and allow reproducible builds tooling to fetch the appropriate makepkg.conf.
<3<3
Signed-off-by: Levente Polyak <anthraxx@archlinux.org> --- scripts/makepkg.sh.in | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 92cb6398..e58edfa1 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -651,6 +651,8 @@ write_buildinfo() { write_kv_pair "builddate" "${SOURCE_DATE_EPOCH}" write_kv_pair "builddir" "${BUILDDIR}" write_kv_pair "startdir" "${startdir}" + write_kv_pair "buildtool" "${BUILDTOOL:-makepkg}" + write_kv_pair "buildtoolver" "${BUILDTOOLVER:-$makepkg_version}" write_kv_pair "buildenv" "${BUILDENV[@]}" write_kv_pair "options" "${OPTIONS[@]}"
I think we should bump the .BUILDINFO version with this change? -- Morten Linderud PGP: 9C02FF419FECBE16
On 4/22/21 9:59 PM, Morten Linderud wrote:
diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 92cb6398..e58edfa1 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -651,6 +651,8 @@ write_buildinfo() { write_kv_pair "builddate" "${SOURCE_DATE_EPOCH}" write_kv_pair "builddir" "${BUILDDIR}" write_kv_pair "startdir" "${startdir}" + write_kv_pair "buildtool" "${BUILDTOOL:-makepkg}" + write_kv_pair "buildtoolver" "${BUILDTOOLVER:-$makepkg_version}" write_kv_pair "buildenv" "${BUILDENV[@]}" write_kv_pair "options" "${OPTIONS[@]}"
I think we should bump the .BUILDINFO version with this change?
The format version has already been bumped and there was no release yet with the increased version. I consider such change equal to ABI soname bumps so I do not believe we need additional bumps since pre release. cheers, Levente
On 4/22/21 3:46 PM, anthraxx@archlinux.org wrote:
From: Levente Polyak <anthraxx@archlinux.org>
If a makepkg consumer uses a build wrapper to override compiler flags this may lead to unreproducible packages as there is no way to know which exact files were used for tooling that tries to reproduce said package.
Instead of vendoring the whole used makepkg.conf file into buildinfo, this patch adds two new properties to the .BUILDINFO file named BUILDTOOL and BUILDTOOLVER which by default are simply makepkg's own values. Downstream consumers may override those values: For example in Arch Linux the devtools package can set those values and allow reproducible builds tooling to fetch the appropriate makepkg.conf.
I still believe we should be adding the specific parts of the makepkg.conf which are relevant -- the *FLAGS variables -- and previously submitted a patch to that effect. If we go the route you're suggesting, then does that mean devtools will install e.g. /usr/share/devtools/oldconfigs/makepkg-{date}.conf or are rebuilder tools supposed to clone devtools and check out the right version? If we're delegating to a buildtool version, should we simplify .BUILDINFO by removing buildenv/options?
Signed-off-by: Levente Polyak <anthraxx@archlinux.org> --- scripts/makepkg.sh.in | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 92cb6398..e58edfa1 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -651,6 +651,8 @@ write_buildinfo() { write_kv_pair "builddate" "${SOURCE_DATE_EPOCH}" write_kv_pair "builddir" "${BUILDDIR}" write_kv_pair "startdir" "${startdir}" + write_kv_pair "buildtool" "${BUILDTOOL:-makepkg}" + write_kv_pair "buildtoolver" "${BUILDTOOLVER:-$makepkg_version}" write_kv_pair "buildenv" "${BUILDENV[@]}" write_kv_pair "options" "${OPTIONS[@]}"
-- Eli Schwartz Bug Wrangler and Trusted User
On April 22, 2021 10:50:34 PM GMT+02:00, Eli Schwartz <eschwartz@archlinux.org> wrote:
On 4/22/21 3:46 PM, anthraxx@archlinux.org wrote:
From: Levente Polyak <anthraxx@archlinux.org>
If a makepkg consumer uses a build wrapper to override compiler flags this may lead to unreproducible packages as there is no way to know which exact files were used for tooling that tries to reproduce said package.
Instead of vendoring the whole used makepkg.conf file into buildinfo, this patch adds two new properties to the .BUILDINFO file named BUILDTOOL and BUILDTOOLVER which by default are simply makepkg's own values. Downstream consumers may override those values: For example in Arch Linux the devtools package can set those values and allow reproducible builds tooling to fetch the appropriate makepkg.conf.
I still believe we should be adding the specific parts of the makepkg.conf which are relevant -- the *FLAGS variables -- and previously submitted a patch to that effect.
If we go the route you're suggesting, then does that mean devtools will install e.g. /usr/share/devtools/oldconfigs/makepkg-{date}.conf or are rebuilder tools supposed to clone devtools and check out the right version?
I personally believe this is a more universal approach with less future maintenance instead of trying to replicate whatever flags get added into every single artifact that gets built. Reproducer tools need to have distro aspects and specific domain knowledge anyway, I believe grabbing a makepkg config file from the source tree based on an identifiable tag is within what's reasonably trivial.
If we're delegating to a buildtool version, should we simplify .BUILDINFO by removing buildenv/options?
I assume this would make sense if we keep this route, In case the approach won't be NACKed I will send a second patch removing the duplicate properties.
On Thu, 22 Apr 2021 at 21:50, Eli Schwartz <eschwartz@archlinux.org> wrote:
On 4/22/21 3:46 PM, anthraxx@archlinux.org wrote:
From: Levente Polyak <anthraxx@archlinux.org>
If a makepkg consumer uses a build wrapper to override compiler flags this may lead to unreproducible packages as there is no way to know which exact files were used for tooling that tries to reproduce said package.
Instead of vendoring the whole used makepkg.conf file into buildinfo, this patch adds two new properties to the .BUILDINFO file named BUILDTOOL and BUILDTOOLVER which by default are simply makepkg's own values. Downstream consumers may override those values: For example in Arch Linux the devtools package can set those values and allow reproducible builds tooling to fetch the appropriate makepkg.conf.
I still believe we should be adding the specific parts of the makepkg.conf which are relevant -- the *FLAGS variables -- and previously submitted a patch to that effect.
If we go the route you're suggesting, then does that mean devtools will install e.g. /usr/share/devtools/oldconfigs/makepkg-{date}.conf or are rebuilder tools supposed to clone devtools and check out the right version?
Before I f*ked it up and removed it, repro was using an equivalent to: pacman -S devtools cp /usr/share/devtools/makepkg-x86_64.conf /etc/makepkg.conf
If we're delegating to a buildtool version, should we simplify .BUILDINFO by removing buildenv/options?
It's a possibility. AFAICT Levente's solution is more scalable and since it won't require pacman release, easier/quicker to turnaround. While I can see value in both solutions, the pragmatist in me would opt for Levente's patch, while the perfectionist your proposal. -Emil
On 4/22/21 6:42 PM, Emil Velikov wrote:
On Thu, 22 Apr 2021 at 21:50, Eli Schwartz <eschwartz@archlinux.org> wrote:
On 4/22/21 3:46 PM, anthraxx@archlinux.org wrote:
From: Levente Polyak <anthraxx@archlinux.org>
If a makepkg consumer uses a build wrapper to override compiler flags this may lead to unreproducible packages as there is no way to know which exact files were used for tooling that tries to reproduce said package.
Instead of vendoring the whole used makepkg.conf file into buildinfo, this patch adds two new properties to the .BUILDINFO file named BUILDTOOL and BUILDTOOLVER which by default are simply makepkg's own values. Downstream consumers may override those values: For example in Arch Linux the devtools package can set those values and allow reproducible builds tooling to fetch the appropriate makepkg.conf.
I still believe we should be adding the specific parts of the makepkg.conf which are relevant -- the *FLAGS variables -- and previously submitted a patch to that effect.
If we go the route you're suggesting, then does that mean devtools will install e.g. /usr/share/devtools/oldconfigs/makepkg-{date}.conf or are rebuilder tools supposed to clone devtools and check out the right version?
Before I f*ked it up and removed it, repro was using an equivalent to:
pacman -S devtools cp /usr/share/devtools/makepkg-x86_64.conf /etc/makepkg.conf
This isn't inherently wrong though. That is what makerepropkg does too (it ships with devtools and inits a chroot using devtools). Though it does then add printf 'OPTIONS=(%s)\n' "${buildopts[*]@Q}" printf 'BUILDENV=(%s)\n' "${buildenv[*]@Q}" to the makepkg.conf (extracted from .BUILDINFO)
If we're delegating to a buildtool version, should we simplify .BUILDINFO by removing buildenv/options?
It's a possibility.
AFAICT Levente's solution is more scalable and since it won't require pacman release, easier/quicker to turnaround. While I can see value in both solutions, the pragmatist in me would opt for Levente's patch, while the perfectionist your proposal.
I simply don't understand what you're trying to say, since both do require pacman release. How else will buildtool get set? -- Eli Schwartz Bug Wrangler and Trusted User
On April 23, 2021 12:53:04 AM GMT+02:00, Eli
AFAICT Levente's solution is more scalable and since it won't require pacman release, easier/quicker to turnaround. While I can see value in both solutions, the pragmatist in me would opt for Levente's patch, while the perfectionist your proposal.
I simply don't understand what you're trying to say, since both do require pacman release. How else will buildtool get set?
I believe it's not about this particular release but any future release with changes in makepkg.conf that turn out to be required for reproducibility. Not using a universal solution required makepkg adjustments to the buildinfo file to be integrated and released before a reproduce could make use of it.
On 23/4/21 8:04 am, Levente Polyak wrote:
If we're delegating to a buildtool version, should we simplify .BUILDINFO by removing buildenv/options?
I assume this would make sense if we keep this route, In case the approach won't be NACKed I will send a second patch removing the duplicate properties.
I will nack this... it is handy to be able to extract this information from packages in a repo without going back to the original build tool. Reproducible builds is not the only possible use of this information. Allan
On 4/22/21 6:04 PM, Levente Polyak wrote:
On April 22, 2021 10:50:34 PM GMT+02:00, Eli Schwartz <eschwartz@archlinux.org> wrote:
I still believe we should be adding the specific parts of the makepkg.conf which are relevant -- the *FLAGS variables -- and previously submitted a patch to that effect.
If we go the route you're suggesting, then does that mean devtools will install e.g. /usr/share/devtools/oldconfigs/makepkg-{date}.conf or are rebuilder tools supposed to clone devtools and check out the right version?
I personally believe this is a more universal approach with less future maintenance instead of trying to replicate whatever flags get added into every single artifact that gets built.
It's a specific solution, instead of a generic solution. And I don't believe it is a maintenance burden to maintain a list of flag types which makepkg.conf and the prepare_buildenv() function don't see much churn on anyway. (The number of flag types has remained essentially constant for C/C++ until I added rust support.) It means that in order to create a package that can be reproduced, you need/are expected to add additional stuff to the environment -- if BUILDTOOL is unset, we might as well stop setting SOURCE_DATE_EPOCH, touch'ing files to a reproducible timestamp in create_package or after extract_sources, etc. Because if the build tool is just "makepkg", we're saying "no, this package is not expected to be reproducible". Right? We could even be daring and stop creating the .BUILDINFO file either.
Reproducer tools need to have distro aspects and specific domain knowledge anyway, I believe grabbing a makepkg config file from the source tree based on an identifiable tag is within what's reasonably trivial.
Then makerepropkg which is a release version of devtools, needs to know how to retrieve the development version of itself in order to operate. Which seems backwards. I know that repro needs to bootstrap itself from any OS, so it does more complicated steps including cloning sources, syncing makepkg.conf from devtools, etc. Once you're already doing that, "clone devtools and find the configuration file from a specific release" sounds less complicated and more like something you're already doing. And we will likely start seeing automated build tools (CI/CD release pipelines?) that are not devtools, but use the devtools makepkg.conf, call themselves "devtools" for compatibility purposes. Because the buildtool is not actually relevant, only the makepkg.conf it uses is relevant, but... people wanna message which makepkg.conf they're using so they can demonstrate their supply chain security. And docker builds shouldn't be measurably different from systemd-nspawn ones! If this ends up happening, we'll know we have reinvented the browser user agent...
If we're delegating to a buildtool version, should we simplify .BUILDINFO by removing buildenv/options?
I assume this would make sense if we keep this route, In case the approach won't be NACKed I will send a second patch removing the duplicate properties.
-- Eli Schwartz Bug Wrangler and Trusted User
On 4/22/21 6:59 PM, Levente Polyak wrote:
I believe it's not about this particular release but any future release with changes in makepkg.conf that turn out to be required for reproducibility. Not using a universal solution required makepkg adjustments to the buildinfo file to be integrated and released before a reproduce could make use of it.
If there are changes for newly added variables which makepkg.conf respects, then the required makepkg adjustments to the buildinfo file would be integrated in the same code checkin that adds it to the makepkg.conf file -- and released at the same time too. In general -- changes to makepkg.conf don't mean anything unless they're respected in /usr/bin/makepkg, and we know which changes are relevant to reproducible builds (MAKEFLAGS is not, SRCEXT isn't, *DEST isn't, DLAGENTS isn't. What is? Codegen flags, OPTIONS, some parts of BUILDENV.) Whether you look up the makepkg.conf of a well-known buildtool, or tell write_buildinfo to export the info directly, it's always getting released at the same time. -- Eli Schwartz Bug Wrangler and Trusted User
On 4/22/21 7:07 PM, Allan McRae wrote:
On 23/4/21 8:04 am, Levente Polyak wrote:
If we're delegating to a buildtool version, should we simplify .BUILDINFO by removing buildenv/options?
I assume this would make sense if we keep this route, In case the approach won't be NACKed I will send a second patch removing the duplicate properties.
I will nack this... it is handy to be able to extract this information from packages in a repo without going back to the original build tool. Reproducible builds is not the only possible use of this information.
I could say the same thing about CFLAGS. :) -- Eli Schwartz Bug Wrangler and Trusted User
On 4/22/21 3:46 PM, anthraxx@archlinux.org wrote:
From: Levente Polyak <anthraxx@archlinux.org>
If a makepkg consumer uses a build wrapper to override compiler flags this may lead to unreproducible packages as there is no way to know which exact files were used for tooling that tries to reproduce said package.
Instead of vendoring the whole used makepkg.conf file into buildinfo, this patch adds two new properties to the .BUILDINFO file named BUILDTOOL and BUILDTOOLVER which by default are simply makepkg's own values. Downstream consumers may override those values: For example in Arch Linux the devtools package can set those values and allow reproducible builds tooling to fetch the appropriate makepkg.conf.
Signed-off-by: Levente Polyak <anthraxx@archlinux.org> --- scripts/makepkg.sh.in | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 92cb6398..e58edfa1 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -651,6 +651,8 @@ write_buildinfo() { write_kv_pair "builddate" "${SOURCE_DATE_EPOCH}" write_kv_pair "builddir" "${BUILDDIR}" write_kv_pair "startdir" "${startdir}" + write_kv_pair "buildtool" "${BUILDTOOL:-makepkg}" + write_kv_pair "buildtoolver" "${BUILDTOOLVER:-$makepkg_version}"
This is not documented in makepkg(8), so there is no way for "Downstream consumers may override those values" without having the pacman patch submitter bring it to their attention. Using the variable in devtools would be a special case of this... The boat has sailed on adding documentation in the same commit that adds the feature...
write_kv_pair "buildenv" "${BUILDENV[@]}" write_kv_pair "options" "${OPTIONS[@]}"
-- Eli Schwartz Bug Wrangler and Trusted User
participants (6)
-
Allan McRae
-
anthraxx＠archlinux.org
-
Eli Schwartz
-
Emil Velikov
-
Levente Polyak
-
Morten Linderud