[pacman-dev] [PATCH] makepkg: respect $SOURCE_DATE_EPOCH to activate reproducible builds
If SOURCE_DATE_EPOCH is set, `touch` all source files before running build() to fix the modification times. This works around build systems and compilers that embed the file modification times into the file contents of release artifacts. Signed-off-by: Eli Schwartz <eschwartz93@gmail.com> --- Guarded by checking for the variable, because this stomps all over incremental builds. I prefer looking for SOURCE_DATE_EPOCH in the environment because less bloat IMHO, but perhaps this would be preferable as a makepkg flag or makepkg.conf option? Is there anything else that needs to be done to force reproducible builds, other than merely setting SOURCE_DATE_EPOCH which should be harmless as a global thing? scripts/makepkg.sh.in | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 20e9dd7e..fb1d40a9 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -79,15 +79,22 @@ PKGFUNC=0 PKGVERFUNC=0 PREPAREFUNC=0 REPKG=0 +REPRODUCIBLE=0 RMDEPS=0 SKIPCHECKSUMS=0 SKIPPGPCHECK=0 SIGNPKG='' SPLITPKG=0 SOURCEONLY=0 + VERIFYSOURCE=0 -export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-$(date +%s)} +if [[ -n $SOURCE_DATE_EPOCH ]]; then + REPRODUCIBLE=1 +else + SOURCE_DATE_EPOCH=$(date +%s) +fi +export SOURCE_DATE_EPOCH PACMAN_OPTS=() @@ -475,6 +482,12 @@ run_prepare() { } run_build() { + if (( REPRODUCIBLE )); then + # We have activated reproducible builds, so unify source times before + # building + find "$srcdir" -exec touch -h -d @$SOURCE_DATE_EPOCH {} + + fi + run_function_safe "build" } -- 2.13.2
On 07/07/17 03:26, Eli Schwartz wrote:
If SOURCE_DATE_EPOCH is set, `touch` all source files before running build() to fix the modification times. This works around build systems and compilers that embed the file modification times into the file contents of release artifacts.
I think this is a reasonable compromise. @Levente: are you happy with this approach?
On 07/28/2017 04:36 AM, Allan McRae wrote:
On 07/07/17 03:26, Eli Schwartz wrote:
If SOURCE_DATE_EPOCH is set, `touch` all source files before running build() to fix the modification times. This works around build systems and compilers that embed the file modification times into the file contents of release artifacts.
I think this is a reasonable compromise.
@Levente: are you happy with this approach?
The conditional REPRODUCIBLE touch block should be outside of run_build as former would imply that a build() function is mandatory inside every PKGBUILD to make this work. Its quite common (f.e. for python) to not have a build() function and I believe it should generally work without enforcing to have such function. Therefor I would recommend we move that block between run_build and run_prepare, f.e. a place where this works is line 1696 in commit bcc9c417. A tested/verified adjustment doing the above would be: https://github.com/anthraxx/pacman/commit/520acf93d83774c4b74ec8c19c1ba31fdd... Technically this approach will work, but personally i believe this is a bit too much hidden requirement to make it work. So for clarity I would either recommend we add a section describing this behaviour inside the manpage (as manually defining SOURCE_DATE_EPOCH will be mandatory to make it work) or we use makepkg flags plus config (which will also make it more transparent). I don't have any strong favorites, Eli implementation will do its job, the only difference will be that manually setting SOURCE_DATE_EPOCH is mandatory for the first initial invocation as well. cheers, Levente PS: thanks :P
On 07/28/2017 07:59 AM, Levente Polyak wrote:
On 07/28/2017 04:36 AM, Allan McRae wrote:
On 07/07/17 03:26, Eli Schwartz wrote:
If SOURCE_DATE_EPOCH is set, `touch` all source files before running build() to fix the modification times. This works around build systems and compilers that embed the file modification times into the file contents of release artifacts.
I think this is a reasonable compromise.
@Levente: are you happy with this approach?
The conditional REPRODUCIBLE touch block should be outside of run_build as former would imply that a build() function is mandatory inside every PKGBUILD to make this work. Its quite common (f.e. for python) to not have a build() function and I believe it should generally work without enforcing to have such function. Therefor I would recommend we move that block between run_build and run_prepare, f.e. a place where this works is line 1696 in commit bcc9c417. A tested/verified adjustment doing the above would be: https://github.com/anthraxx/pacman/commit/520acf93d83774c4b74ec8c19c1ba31fdd...
I kind of feel like this is a sign that people are doing the wrong thing, since you shouldn't be modifying $srcdir during package(). Personally, I am very careful about making sure `python setup.py build` is run during build(), the same way I am careful about making sure `make` is run during build() even when `make install` would cause `make all` to run anyway. I don't feel like we should make a split between GNU Make and other build systems, or between a build system that executes a C compiler vs. one that merely runs pod2man and copies files around. I won't even mention npm except to say, look at aur/rapydscript-ng-git.
Technically this approach will work, but personally i believe this is a bit too much hidden requirement to make it work. So for clarity I would either recommend we add a section describing this behaviour inside the manpage (as manually defining SOURCE_DATE_EPOCH will be mandatory to make it work) or we use makepkg flags plus config (which will also make it more transparent). I don't have any strong favorites, Eli implementation will do its job, the only difference will be that manually setting SOURCE_DATE_EPOCH is mandatory for the first initial invocation as well.
Okay. I personally prefer the implementation that doesn't add more flags or makepkg.conf options, because at a minimum anyone who wants to make use of this will need to have tooling to read from a previous invocation regardless or on top of those flags for the second invocation, so we might as well follow the same approach the whole time. Perhaps a REPRODUCIBILITY section in makepkg(1) right after ADDITIONAL FEATURES, then, to describe what efforts makepkg will go to to ensure a reproducible build? And then add SOURCE_DATE_EPOCH for environment variables makepkg will respect. -- Eli Schwartz
If SOURCE_DATE_EPOCH is set, `touch` all source files before running build() to fix the modification times. This works around build systems and compilers that embed the file modification times into the file contents of release artifacts. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- v2: Fix accidental extra blank line that sneaked into v1 scripts/makepkg.sh.in | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 20e9dd7e..77d39ca5 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -79,6 +79,7 @@ PKGFUNC=0 PKGVERFUNC=0 PREPAREFUNC=0 REPKG=0 +REPRODUCIBLE=0 RMDEPS=0 SKIPCHECKSUMS=0 SKIPPGPCHECK=0 @@ -87,7 +88,12 @@ SPLITPKG=0 SOURCEONLY=0 VERIFYSOURCE=0 -export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-$(date +%s)} +if [[ -n $SOURCE_DATE_EPOCH ]]; then + REPRODUCIBLE=1 +else + SOURCE_DATE_EPOCH=$(date +%s) +fi +export SOURCE_DATE_EPOCH PACMAN_OPTS=() @@ -475,6 +481,12 @@ run_prepare() { } run_build() { + if (( REPRODUCIBLE )); then + # We have activated reproducible builds, so unify source times before + # building + find "$srcdir" -exec touch -h -d @$SOURCE_DATE_EPOCH {} + + fi + run_function_safe "build" } -- 2.13.3
If SOURCE_DATE_EPOCH is set, `touch` all source files before running build() to fix the modification times. This works around build systems and compilers that embed the file modification times into the file contents of release artifacts. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- v3: add makepkg(8) documentation doc/makepkg.8.txt | 16 ++++++++++++++++ scripts/makepkg.sh.in | 14 +++++++++++++- 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/doc/makepkg.8.txt b/doc/makepkg.8.txt index 2dff1b19..4258e6bd 100644 --- a/doc/makepkg.8.txt +++ b/doc/makepkg.8.txt @@ -206,6 +206,7 @@ Options *\--printsrcinfo*:: Generate and print the SRCINFO file to stdout. + Additional Features ------------------- makepkg supports building development versions of packages without having to @@ -214,6 +215,19 @@ separate utility 'versionpkg'. See linkman:PKGBUILD[5] for details on how to set up a development PKGBUILD. +Reproducibility +--------------- +makepkg is designed to be compatible with +link:https://reproducible-builds.org/docs/[Reproducible Builds]. If the +**SOURCE_DATE_EPOCH** environment variable is set, it will be exported to +subprocesses, and source and package file modification times and package +metadata will be unified based on the timestamp specified. + +If the **SOURCE_DATE_EPOCH** environment variable is not set, makepkg will use +its own start date for internal use, but is not responsible for ensuring the +package files themselves are built reproducibly. + + Environment Variables --------------------- **PACMAN**:: @@ -265,6 +279,8 @@ Environment Variables Specify a key to use when signing packages, overriding the GPGKEY setting in linkman:makepkg.conf[5] +**SOURCE_DATE_EPOCH=**"<date>":: + Used for link:https://reproducible-builds.org/docs/[Reproducible Builds]. Configuration ------------- diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 20e9dd7e..77d39ca5 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -79,6 +79,7 @@ PKGFUNC=0 PKGVERFUNC=0 PREPAREFUNC=0 REPKG=0 +REPRODUCIBLE=0 RMDEPS=0 SKIPCHECKSUMS=0 SKIPPGPCHECK=0 @@ -87,7 +88,12 @@ SPLITPKG=0 SOURCEONLY=0 VERIFYSOURCE=0 -export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-$(date +%s)} +if [[ -n $SOURCE_DATE_EPOCH ]]; then + REPRODUCIBLE=1 +else + SOURCE_DATE_EPOCH=$(date +%s) +fi +export SOURCE_DATE_EPOCH PACMAN_OPTS=() @@ -475,6 +481,12 @@ run_prepare() { } run_build() { + if (( REPRODUCIBLE )); then + # We have activated reproducible builds, so unify source times before + # building + find "$srcdir" -exec touch -h -d @$SOURCE_DATE_EPOCH {} + + fi + run_function_safe "build" } -- 2.14.1
On 24/08/17 07:12, Eli Schwartz wrote:
If SOURCE_DATE_EPOCH is set, `touch` all source files before running build() to fix the modification times. This works around build systems and compilers that embed the file modification times into the file contents of release artifacts.
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> ---
v3: add makepkg(8) documentation
doc/makepkg.8.txt | 16 ++++++++++++++++ scripts/makepkg.sh.in | 14 +++++++++++++- 2 files changed, 29 insertions(+), 1 deletion(-)
diff --git a/doc/makepkg.8.txt b/doc/makepkg.8.txt index 2dff1b19..4258e6bd 100644 --- a/doc/makepkg.8.txt +++ b/doc/makepkg.8.txt @@ -206,6 +206,7 @@ Options *\--printsrcinfo*:: Generate and print the SRCINFO file to stdout.
+ Additional Features ------------------- makepkg supports building development versions of packages without having to @@ -214,6 +215,19 @@ separate utility 'versionpkg'. See linkman:PKGBUILD[5] for details on how to set up a development PKGBUILD.
+Reproducibility +--------------- +makepkg is designed to be compatible with +link:https://reproducible-builds.org/docs/[Reproducible Builds]. If the +**SOURCE_DATE_EPOCH** environment variable is set, it will be exported to +subprocesses, and source and package file modification times and package +metadata will be unified based on the timestamp specified. + +If the **SOURCE_DATE_EPOCH** environment variable is not set, makepkg will use +its own start date for internal use, but is not responsible for ensuring the +package files themselves are built reproducibly. +
I don't like the phrasing there. How about : If the **SOURCE_DATE_EPOCH** environment variable is not set, makepkg will use its own start date for internal use, but will not unify source file timestamps before building.
+ Environment Variables --------------------- **PACMAN**:: @@ -265,6 +279,8 @@ Environment Variables Specify a key to use when signing packages, overriding the GPGKEY setting in linkman:makepkg.conf[5]
+**SOURCE_DATE_EPOCH=**"<date>":: + Used for link:https://reproducible-builds.org/docs/[Reproducible Builds].
Configuration ------------- diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 20e9dd7e..77d39ca5 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -79,6 +79,7 @@ PKGFUNC=0 PKGVERFUNC=0 PREPAREFUNC=0 REPKG=0 +REPRODUCIBLE=0 RMDEPS=0 SKIPCHECKSUMS=0 SKIPPGPCHECK=0 @@ -87,7 +88,12 @@ SPLITPKG=0 SOURCEONLY=0 VERIFYSOURCE=0
-export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-$(date +%s)} +if [[ -n $SOURCE_DATE_EPOCH ]]; then + REPRODUCIBLE=1 +else + SOURCE_DATE_EPOCH=$(date +%s) +fi +export SOURCE_DATE_EPOCH
PACMAN_OPTS=()
@@ -475,6 +481,12 @@ run_prepare() { }
run_build() { + if (( REPRODUCIBLE )); then + # We have activated reproducible builds, so unify source times before + # building + find "$srcdir" -exec touch -h -d @$SOURCE_DATE_EPOCH {} + + fi +
I don't like this in run_build(). That will introduce an undocumented requirement that a PKGBUILD has a build() function to have its source file timestamps unified. I am happy with the location suggested here: https://github.com/anthraxx/pacman/commit/520acf93
run_function_safe "build" }
If SOURCE_DATE_EPOCH is set, `touch` all source files as part of prepare() to fix the modification times. This works around build systems and compilers that embed the file modification times into the file contents of release artifacts. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- doc/makepkg.8.txt | 16 ++++++++++++++++ scripts/makepkg.sh.in | 13 ++++++++++++- 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/doc/makepkg.8.txt b/doc/makepkg.8.txt index 2dff1b19..e1d50775 100644 --- a/doc/makepkg.8.txt +++ b/doc/makepkg.8.txt @@ -206,6 +206,7 @@ Options *\--printsrcinfo*:: Generate and print the SRCINFO file to stdout. + Additional Features ------------------- makepkg supports building development versions of packages without having to @@ -214,6 +215,19 @@ separate utility 'versionpkg'. See linkman:PKGBUILD[5] for details on how to set up a development PKGBUILD. +Reproducibility +--------------- +makepkg is designed to be compatible with +link:https://reproducible-builds.org/docs/[Reproducible Builds]. If the +**SOURCE_DATE_EPOCH** environment variable is set, it will be exported to +subprocesses, and source and package file modification times and package +metadata will be unified based on the timestamp specified. + +If the **SOURCE_DATE_EPOCH** environment variable is not set, makepkg will use +its own start date for internal use, but will not unify source file timestamps +before building. + + Environment Variables --------------------- **PACMAN**:: @@ -265,6 +279,8 @@ Environment Variables Specify a key to use when signing packages, overriding the GPGKEY setting in linkman:makepkg.conf[5] +**SOURCE_DATE_EPOCH=**"<date>":: + Used for link:https://reproducible-builds.org/docs/[Reproducible Builds]. Configuration ------------- diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 20e9dd7e..a466e4f1 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -79,6 +79,7 @@ PKGFUNC=0 PKGVERFUNC=0 PREPAREFUNC=0 REPKG=0 +REPRODUCIBLE=0 RMDEPS=0 SKIPCHECKSUMS=0 SKIPPGPCHECK=0 @@ -87,7 +88,12 @@ SPLITPKG=0 SOURCEONLY=0 VERIFYSOURCE=0 -export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-$(date +%s)} +if [[ -n $SOURCE_DATE_EPOCH ]]; then + REPRODUCIBLE=1 +else + SOURCE_DATE_EPOCH=$(date +%s) +fi +export SOURCE_DATE_EPOCH PACMAN_OPTS=() @@ -1686,6 +1692,11 @@ if (( !REPKG )); then if (( PREPAREFUNC )); then run_prepare fi + if (( REPRODUCIBLE )); then + # We have activated reproducible builds, so unify source times before + # building + find "$srcdir" -exec touch -h -d @$SOURCE_DATE_EPOCH {} + + fi fi if (( PKGVERFUNC )); then -- 2.14.1
participants (4)
-
Allan McRae
-
Eli Schwartz
-
Eli Schwartz
-
Levente Polyak