[pacman-dev] [PATCH] makepkg: Log commands to logfile
For debugging PKGBUILDs, and for reporting build failures upstream, it is very important to know the exact commands that are executed leading up to a problem. By using `set -x` and an appropriate value for PS4, commands are logged before they are executed. Signed-off-by: Alain Kalker <a.c.kalker@gmail.com> --- scripts/makepkg.sh.in | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 2a39157..859159f 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -1468,8 +1468,15 @@ run_function() { tee "$BUILDLOG" < "$logpipe" & local teepid=$! + local restoreps4=$PS4 + PS4='$ ' + set -x + $pkgfunc &>"$logpipe" + set +x + PS4=$restoreps4 + wait $teepid rm "$logpipe" else -- 1.8.3.2
On Tue, Jul 02, 2013 at 09:40:13PM +0200, Alain Kalker wrote:
For debugging PKGBUILDs, and for reporting build failures upstream, it is very important to know the exact commands that are executed leading up to a problem. By using `set -x` and an appropriate value for PS4, commands are logged before they are executed.
If you want to report build failures upstream, then building via makepkg is already wrong. -1 on this.
Signed-off-by: Alain Kalker <a.c.kalker@gmail.com> --- scripts/makepkg.sh.in | 7 +++++++ 1 file changed, 7 insertions(+)
diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 2a39157..859159f 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -1468,8 +1468,15 @@ run_function() { tee "$BUILDLOG" < "$logpipe" & local teepid=$!
+ local restoreps4=$PS4 + PS4='$ ' + set -x + $pkgfunc &>"$logpipe"
+ set +x + PS4=$restoreps4 + wait $teepid rm "$logpipe" else -- 1.8.3.2
On 07/02/2013 09:55 PM, Dave Reisner wrote: > On Tue, Jul 02, 2013 at 09:40:13PM +0200, Alain Kalker wrote: >> For debugging PKGBUILDs, and for reporting build failures upstream, >> it is very important to know the exact commands that are executed >> leading up to a problem. >> By using `set -x` and an appropriate value for PS4, commands are >> logged before they are executed. > If you want to report build failures upstream, then building via makepkg > is already wrong. Not if: - the build failure is caused by an Autoconf bug (see FS#35983 - [libxcb] Build failure: .../install-sh: No such file or directory) - the package is built as vanilla, as possible, as is supposedly the Arch Way. - the logged commands fully reflect the build process, i.e. no interference from outside environment variables, etc. > > -1 on this. Please don't NAK just because you disagree on one possible use of this feature! As I've stated clearly, the patch has other obvious benefits: debugging PKGBUILDs. Thank you.
On Tue, Jul 02, 2013 at 10:05:00PM +0200, Alain Kalker wrote: > On 07/02/2013 09:55 PM, Dave Reisner wrote: > >On Tue, Jul 02, 2013 at 09:40:13PM +0200, Alain Kalker wrote: > >>For debugging PKGBUILDs, and for reporting build failures upstream, > >>it is very important to know the exact commands that are executed > >>leading up to a problem. > >>By using `set -x` and an appropriate value for PS4, commands are > >>logged before they are executed. > >If you want to report build failures upstream, then building via makepkg > >is already wrong. > Not if: > - the build failure is caused by an Autoconf bug (see FS#35983 - > [libxcb] Build failure: .../install-sh: No such file or directory) Your patch does nothing to expose the problem you're reporting here. > - the package is built as vanilla, as possible, as is supposedly the > Arch Way. I can guarantee you that any sane upstream will tell you to build as per *their* instructions (i.e. not using makepkg) if you try and report a build failure to them. Regardless of how these failures were discovered, they absolutely require manual verification outside of makepkg. > - the logged commands fully reflect the build process, i.e. no > interference from outside environment variables, etc. Except that by design, specific environment variables are pulled in by makepkg. > > > >-1 on this. > Please don't NAK just because you disagree on one possible use of > this feature! > As I've stated clearly, the patch has other obvious benefits: > debugging PKGBUILDs. > Thank you. > You already have the list of commands to be executed right in front of you. You're running code that you yourself wrote into the PKGBUILD and nothing more.
On 07/02/2013 10:17 PM, Dave Reisner wrote: > On Tue, Jul 02, 2013 at 10:05:00PM +0200, Alain Kalker wrote: >> On 07/02/2013 09:55 PM, Dave Reisner wrote: >>> On Tue, Jul 02, 2013 at 09:40:13PM +0200, Alain Kalker wrote: >>>> For debugging PKGBUILDs, and for reporting build failures upstream, >>>> it is very important to know the exact commands that are executed >>>> leading up to a problem. >>>> By using `set -x` and an appropriate value for PS4, commands are >>>> logged before they are executed. >>> If you want to report build failures upstream, then building via makepkg >>> is already wrong. >> Not if: >> - the build failure is caused by an Autoconf bug (see FS#35983 - >> [libxcb] Build failure: .../install-sh: No such file or directory) > Your patch does nothing to expose the problem you're reporting here. > >> - the package is built as vanilla, as possible, as is supposedly the >> Arch Way. > I can guarantee you that any sane upstream will tell you to build as per > *their* instructions (i.e. not using makepkg) if you try and report a > build failure to them. Regardless of how these failures were discovered, > they absolutely require manual verification outside of makepkg. Then your guarantee doesn't mean squat, sorry.. See https://bugs.freedesktop.org/show_bug.cgi?id=66413 PKGBUILD build() and package() functions contain build recipes, do you honestly suggest that users should copy/paste $ export VAR1=val1 VAR2=val2 VAR3=val3 [and a bazillion more of those] $ ./configure [a bazillion configure options] $ make [a bazillion make options] into a terminal just to reproduce a failure exposed by running 'makepkg'? Perhaps later in the triaging process, if it can be proven that the makepkg environment interfered with the build and caused the failure, but that would be a Bad Thing(TM) in itself, _then_ executing all these command by hand would have merit >> - the logged commands fully reflect the build process, i.e. no >> interference from outside environment variables, etc. > Except that by design, specific environment variables are pulled in by > makepkg. See my remark above. Since the log file would contain the commands (with $variables expanded), it would be very easy to copy/paste these commands and run them in a 'clean' build tree. That's the whole point! > >>> -1 on this. >> Please don't NAK just because you disagree on one possible use of >> this feature! >> As I've stated clearly, the patch has other obvious benefits: >> debugging PKGBUILDs. >> Thank you. >> > You already have the list of commands to be executed right in front of > you. You're running code that you yourself wrote into the PKGBUILD and > nothing more. With the way current logging works, users have to guess what commands were executed when. The log only shows the output, not the command that caused it. Especially in more complicated PKGBUILDs, that amounts to a lot of guessing. Yes, you could litter your PKGBUILD with (hopefully matched!) pairs of 'set -x/set +x' commands to selectively log groups of commands, but what if you have a whole slew of packages to rebuild in a chroot? What if one of them fails and you find yourself having to search back and forth among this list to find the dependency that does something wrong, silently? I'm not convinced by your downvote, please explain. Kind regards, Alain > > >
On Tue, Jul 02, 2013 at 10:38:47PM +0200, Alain Kalker wrote: > On 07/02/2013 10:17 PM, Dave Reisner wrote: > >On Tue, Jul 02, 2013 at 10:05:00PM +0200, Alain Kalker wrote: > >>On 07/02/2013 09:55 PM, Dave Reisner wrote: > >>>On Tue, Jul 02, 2013 at 09:40:13PM +0200, Alain Kalker wrote: > >>>>For debugging PKGBUILDs, and for reporting build failures upstream, > >>>>it is very important to know the exact commands that are executed > >>>>leading up to a problem. > >>>>By using `set -x` and an appropriate value for PS4, commands are > >>>>logged before they are executed. > >>>If you want to report build failures upstream, then building via makepkg > >>>is already wrong. > >>Not if: > >>- the build failure is caused by an Autoconf bug (see FS#35983 - > >>[libxcb] Build failure: .../install-sh: No such file or directory) > >Your patch does nothing to expose the problem you're reporting here. > > > >>- the package is built as vanilla, as possible, as is supposedly the > >>Arch Way. > >I can guarantee you that any sane upstream will tell you to build as per > >*their* instructions (i.e. not using makepkg) if you try and report a > >build failure to them. Regardless of how these failures were discovered, > >they absolutely require manual verification outside of makepkg. > Then your guarantee doesn't mean squat, sorry.. See > https://bugs.freedesktop.org/show_bug.cgi?id=66413 You make zero mention of makepkg or any external build tool in that environment. I'm not sure how you expect anyone to pick up your veil and dispute your method of compilation. Quite frankly, this is rude of you, regardless of whether or not makepkg is at fault. > PKGBUILD build() and package() functions contain build recipes, do > you honestly suggest that users should copy/paste > > $ export VAR1=val1 VAR2=val2 VAR3=val3 [and a bazillion more of those] > $ ./configure [a bazillion configure options] > $ make [a bazillion make options] > into a terminal just to reproduce a failure exposed by running 'makepkg'? Yeah, I do. They aren't the same thing. I'd be happy to provide a test case for you where makepkg mangles the pacman build. It even affects the package in [core] right now. It's something that I absolutely can't reproduce with a manual build. > Perhaps later in the triaging process, if it can be proven that the > makepkg environment interfered with the build and caused the > failure, but that would be a Bad Thing(TM) in itself, _then_ > executing all these command by hand would have merit > >>- the logged commands fully reflect the build process, i.e. no > >>interference from outside environment variables, etc. > >Except that by design, specific environment variables are pulled in by > >makepkg. > See my remark above. Since the log file would contain the commands > (with $variables expanded), it would be very easy to copy/paste > these commands and run them in a 'clean' build tree. > That's the whole point! The PKGBUILD contains the exact same commands as the ones in your log. It's pretty easy to figure out what correlates to what. If your PKGBUILD is too complex for you to dissect/debug it, maybe you should reconsider how you've written your PKGBUILD. > > > >>>-1 on this. > >>Please don't NAK just because you disagree on one possible use of > >>this feature! > >>As I've stated clearly, the patch has other obvious benefits: > >>debugging PKGBUILDs. > >>Thank you. > >> > >You already have the list of commands to be executed right in front of > >you. You're running code that you yourself wrote into the PKGBUILD and > >nothing more. > With the way current logging works, users have to guess what > commands were executed when. The log only shows the output, not the > command that caused it. Especially in more complicated PKGBUILDs, > that amounts to a lot of guessing. > Yes, you could litter your PKGBUILD with (hopefully matched!) pairs > of 'set -x/set +x' commands to selectively log groups of commands, > but what if you have a whole slew of packages to rebuild in a > chroot? What if one of them fails and you find yourself having to > search back and forth among this list to find the dependency that > does something wrong, silently? > > I'm not convinced by your downvote, please explain. And I'm not convinced by your reasoning. Fortunately, the burden of proof is on you, not me. I'll let others weigh in, but I have no interest in this, especially if you're going to ignore the better features of PS4. d
On 07/02/2013 11:07 PM, Dave Reisner wrote:
You make zero mention of makepkg or any external build tool in that environment. I'm not sure how you expect anyone to pick up your veil and dispute your method of compilation. Quite frankly, this is rude of you, regardless of whether or not makepkg is at fault. Please take this kind of musdlinging somewhere else, IRC, 4chan, a muddy beach, or whatever.
My final try at proving my point is the following script: it takes a logfile from makepkg -L (with my patch applied), and an optional makepkg.conf file (defaulting to /etc/makepkg), then generates a script containing selected environment variables from the configfile, followed by a replay of all commands executed during the build or package phase (determined by the name of the logfile. Script is self-documenting, but here is the gist of how to use it for TL;DR: - Build package using patched `makepkg -L` - Run makereplay on the generated logfile - Clean out src and pkg directories - Run `makepkg -o` - Change to the top of the extracted source tree - Run `clean-build` and `clean-package` as per the docs inside. <<Here is where the fun begins>> - Now you can do a diff between the log generated by `makepkg -L` and from running the clean-* scripts. - If the build failure occurs with the clean build also, simply submit the logs and clean-* scripts upstream along with your bug report and any needed patches used in the build. - ??? - Profit!! And I mean it, for great justice or whatever. Feel free to use this stuff to track down your core package breakage, feel free even to include it in some toolbox, as long as this pissing contest stops here, i'm allergic to flamethrowers. Kind regards, Alain ---[file: makereplay]--- #!/bin/bash # makereplay - Generate a script to replay commands from a makepkg logfile # using environment variables from a makepkg.conf file PROGNAME=$(basename "$0") usage() { echo "usage: $PROGNAME [option] logfile [makepkg.conf file]" } help() { usage echo echo " Options:" echo " -d Use debug variables from makepkg.conf" } [[ $# -eq 0 ]] && usage && exit 1 [[ $1 == "-h" ]] && help && exit 0 [[ $1 == "-d" ]] && { debugbuild=yes; shift; } || debugbuild=no [[ $# -eq 0 ]] && usage && exit 1 logfile=$1; shift [[ ! -f $logfile ]] && echo "Error: logfile does not exist" >&2 && exit 1 [[ $logfile =~ "build.log" ]] && scriptname=replay-build.sh [[ $logfile =~ "package.log" ]] && scriptname=replay-package.sh makepkgconf=$1; shift [[ -z $makepkgconf ]] && makepkgconf=/etc/makepkg.conf [[ ! -f $makepkgconf ]] && echo "Error: makepkg.conf file does not exist" >&2 && exit 1 if [[ $debug = "yes" ]]; then debugfilter="s/^DEBUG_//g" else debugfilter="/^DEBUG_/d" fi badvars=('CARCH' 'CHOST' 'BUILDENV' 'OPTIONS' 'INTEGRITY_CHECK' 'STRIP_.*' '.*_DIRS' '.*_TARGETS' 'COMPRESS.*' '.*EXT') badvarsfilter=$(IFS='|'; echo "/""${badvars[*]}""/d") { cat <<-EOF #!/bin/bash # $scriptname - Replay commands from a makepkg logfile # This script was created using the following configuration: # Log file: $logfile # Build settings: $makepkgconf # Debug build: $debugbuild # Example usage: # $ cd <directory containing PKGBUILD> # $ rm -rf pkg src # $ mkdir pkg src # $ makepkg -o # # Do a clean build or install # $ cd src/<top of source tree> # $ ../../$scriptname 2>&1 | tee ../../$logfile.clean # # Then compare the logs # $ cd ../.. # $ diff -u $logfile{,.clean} EOF echo "# Set up environment" grep "^[^#].*=[^(]" "$makepkgconf" | \ sed -re "$badvarsfilter" | \ sed -re "$debugfilter" | \ sed -re "s/^/export /g" echo echo "# Set up command logging" echo 'restoreps4=$PS4' echo "PS4='$ '" echo "set -x" echo echo "# Replay commands from logfile" sed -n "s/^\$ \(.*\)$/\1/gp" < $logfile echo echo "# Restore command logging" echo "set +x" echo "PS4=$restoreps4" } > $scriptname chmod +x "$scriptname" echo "Script written to: $scriptname" ---[end: makereplay]---
On 07/03/2013 01:45 AM, Alain Kalker wrote:
On 07/02/2013 11:07 PM, Dave Reisner wrote:
You make zero mention of makepkg or any external build tool in that environment. I'm not sure how you expect anyone to pick up your veil and dispute your method of compilation. Quite frankly, this is rude of you, regardless of whether or not makepkg is at fault. Please take this kind of musdlinging somewhere else, IRC, 4chan, a muddy beach, or whatever.
My final try at proving my point is the following script: it takes a logfile from makepkg -L (with my patch applied), and an optional makepkg.conf file (defaulting to /etc/makepkg), then generates a script containing selected environment variables from the configfile, followed by a replay of all commands executed during the build or package phase (determined by the name of the logfile. Script is self-documenting, but here is the gist of how to use it for TL;DR: Here is an example script generated for the example logfile (for RRTL;DR:) ---[file: clean-build.sh]--- #!/bin/bash
# replay-build.sh - Replay commands from a makepkg logfile # This script was created using the following configuration: # Log file: libxcb-1.9.1-1-x86_64-build.log # Build settings: /etc/makepkg.conf # Debug build: no # Example usage: # $ cd <directory containing PKGBUILD> # $ rm -rf pkg src # $ mkdir pkg src # $ makepkg -o # # Do a clean build or install # $ cd src/<top of source tree> # $ ../../replay-build.sh 2>&1 | tee ../../libxcb-1.9.1-1-x86_64-build.log.clean # # Then compare the logs # $ cd ../.. # $ diff -u libxcb-1.9.1-1-x86_64-build.log{,.clean} # Set up environment export CPPFLAGS="-D_FORTIFY_SOURCE=2" export CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4" export CXXFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4" export LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro" # Set up command logging restoreps4=$PS4 PS4='$ ' set -x # Replay commands from logfile cd /home/miki/pkg/abs/extra/libxcb/src/libxcb-1.9.1 patch -Np1 -i /home/miki/pkg/abs/extra/libxcb/src/libxcb-1.1-no-pthread-stubs.patch patch -Np1 -i /home/miki/pkg/abs/extra/libxcb/src/libxcb-1.9.1-configure-ac-reorder.patch ./autogen.sh --prefix=/usr --enable-xinput --enable-xkb --disable-static make # Restore command logging set +x PS4= ---[end: clean-build.sh]--- Clean, concise, easily adaptable for submission upstream, methinks. Kind regards, Alain
On 03/07/13 07:07, Dave Reisner wrote:
Yeah, I do. They aren't the same thing. I'd be happy to provide a test case for you where makepkg mangles the pacman build. It even affects the package in [core] right now. It's something that I absolutely can't reproduce with a manual build.
Please do! I am quite interested... <snip>
I'll let others weigh in, but I have no interest in this, especially if you're going to ignore the better features of PS4.
I'm 90% sure we have rejected this before... I think this is only of use when there is an issue that can not be replicated outside of makepkg. I agree with Dave that (most) upstreams will request the output from a build failure with a manual build anyway. Allan
For debugging PKGBUILDs, and for reporting build failures upstream, it is very important to know the exact commands that are executed leading up to a problem. By using `set -x` and an appropriate value for PS4, commands are logged before they are executed. An example logfile snippet which hopefully shows the usefulness of this
On 07/02/2013 09:40 PM, Alain Kalker wrote: patch: ---[file: libxcb-1.9.1-1-x86_64-build.log]--- $ cd /home/miki/pkg/abs/extra/libxcb/src/libxcb-1.9.1 $ patch -Np1 -i /home/miki/pkg/abs/extra/libxcb/src/libxcb-1.1-no-pthread-stubs.patch patching file configure.ac Hunk #1 succeeded at 42 (offset 7 lines). $ patch -Np1 -i /home/miki/pkg/abs/extra/libxcb/src/libxcb-1.9.1-configure-ac-reorder.patch patching file configure.ac $ ./autogen.sh --prefix=/usr --enable-xinput --enable-xkb --disable-static autoreconf: Entering directory `.' [...] ---[end: libxcb-1.9.1-1-x86_64-build.log]--- Kind regards, Alain Kalker
participants (3)
-
Alain Kalker
-
Allan McRae
-
Dave Reisner