On 07/02/2013 09:55 PM, Dave Reisner wrote:
> On Tue, Jul 02, 2013 at 09:40:13PM +0200, Alain Kalker wrote:
>> For debugging PKGBUILDs, and for reporting build failures upstream,
>> it is very important to know the exact commands that are executed
>> leading up to a problem.
>> By using `set -x` and an appropriate value for PS4, commands are
>> logged before they are executed.
> If you want to report build failures upstream, then building via makepkg
> is already wrong.
Not if:
- the build failure is caused by an Autoconf bug (see FS#35983 - 
[libxcb] Build failure: .../install-sh: No such file or directory)
- the package is built as vanilla, as possible, as is supposedly the 
Arch Way.
- the logged commands fully reflect the build process, i.e. no 
interference from outside environment variables, etc.
>
> -1 on this.
Please don't NAK just because you disagree on one possible use of this 
feature!
As I've stated clearly, the patch has other obvious benefits: debugging 
PKGBUILDs.
Thank you.

On 07/02/2013 10:17 PM, Dave Reisner wrote:
> On Tue, Jul 02, 2013 at 10:05:00PM +0200, Alain Kalker wrote:
>> On 07/02/2013 09:55 PM, Dave Reisner wrote:
>>> On Tue, Jul 02, 2013 at 09:40:13PM +0200, Alain Kalker wrote:
>>>> For debugging PKGBUILDs, and for reporting build failures upstream,
>>>> it is very important to know the exact commands that are executed
>>>> leading up to a problem.
>>>> By using `set -x` and an appropriate value for PS4, commands are
>>>> logged before they are executed.
>>> If you want to report build failures upstream, then building via makepkg
>>> is already wrong.
>> Not if:
>> - the build failure is caused by an Autoconf bug (see FS#35983 -
>> [libxcb] Build failure: .../install-sh: No such file or directory)
> Your patch does nothing to expose the problem you're reporting here.
>
>> - the package is built as vanilla, as possible, as is supposedly the
>> Arch Way.
> I can guarantee you that any sane upstream will tell you to build as per
> *their* instructions (i.e. not using makepkg) if you try and report a
> build failure to them. Regardless of how these failures were discovered,
> they absolutely require manual verification outside of makepkg.
Then your guarantee doesn't mean squat, sorry.. See 
https://bugs.freedesktop.org/show_bug.cgi?id=66413

PKGBUILD build() and package() functions contain build recipes, do you 
honestly suggest that users should copy/paste

$ export VAR1=val1 VAR2=val2 VAR3=val3 [and a bazillion more of those]
$ ./configure [a bazillion configure options]
$ make [a bazillion make options]

into a terminal just to reproduce a failure exposed by running 'makepkg'?
Perhaps later in the triaging process, if it can be proven that the 
makepkg environment interfered with the build and caused the failure, 
but that would be a Bad Thing(TM) in itself, _then_ executing all these 
command by hand would have merit
>> - the logged commands fully reflect the build process, i.e. no
>> interference from outside environment variables, etc.
> Except that by design, specific environment variables are pulled in by
> makepkg.
See my remark above. Since the log file would contain the commands (with 
$variables expanded), it would be very easy to copy/paste these commands 
and run them in a 'clean' build tree.
That's the whole point!

>
>>> -1 on this.
>> Please don't NAK just because you disagree on one possible use of
>> this feature!
>> As I've stated clearly, the patch has other obvious benefits:
>> debugging PKGBUILDs.
>> Thank you.
>>
> You already have the list of commands to be executed right in front of
> you. You're running code that you yourself wrote into the PKGBUILD and
> nothing more.
With the way current logging works, users have to guess what commands 
were executed when. The log only shows the output, not the command that 
caused it. Especially in more complicated PKGBUILDs, that amounts to a 
lot of guessing.
Yes, you could litter your PKGBUILD with (hopefully matched!) pairs of 
'set -x/set +x' commands to selectively log groups of commands, but what 
if you have a whole slew of packages to rebuild in a chroot? What if one 
of them fails and you find yourself having to search back and forth 
among this list to find the dependency that does something wrong, silently?

I'm not convinced by your downvote, please explain.

Kind regards, Alain
>
>
>

On 07/02/2013 11:07 PM, Dave Reisner wrote:

You make zero mention of makepkg or any external build tool in that environment. I'm not sure how you expect anyone to pick up your veil and dispute your method of compilation. Quite frankly, this is rude of you, regardless of whether or not makepkg is at fault. Please take this kind of musdlinging somewhere else, IRC, 4chan, a muddy beach, or whatever.

My final try at proving my point is the following script: it takes a logfile from makepkg -L (with my patch applied), and an optional makepkg.conf file (defaulting to /etc/makepkg), then generates a script containing selected environment variables from the configfile, followed by a replay of all commands executed during the build or package phase (determined by the name of the logfile. Script is self-documenting, but here is the gist of how to use it for TL;DR: - Build package using patched `makepkg -L` - Run makereplay on the generated logfile - Clean out src and pkg directories - Run `makepkg -o` - Change to the top of the extracted source tree - Run `clean-build` and `clean-package` as per the docs inside. <<Here is where the fun begins>> - Now you can do a diff between the log generated by `makepkg -L` and from running the clean-* scripts. - If the build failure occurs with the clean build also, simply submit the logs and clean-* scripts upstream along with your bug report and any needed patches used in the build. - ??? - Profit!! And I mean it, for great justice or whatever. Feel free to use this stuff to track down your core package breakage, feel free even to include it in some toolbox, as long as this pissing contest stops here, i'm allergic to flamethrowers. Kind regards, Alain ---[file: makereplay]--- #!/bin/bash # makereplay - Generate a script to replay commands from a makepkg logfile # using environment variables from a makepkg.conf file PROGNAME=$(basename "$0") usage() { echo "usage: $PROGNAME [option] logfile [makepkg.conf file]" } help() { usage echo echo " Options:" echo " -d Use debug variables from makepkg.conf" } [[ $# -eq 0 ]] && usage && exit 1 [[ $1 == "-h" ]] && help && exit 0 [[ $1 == "-d" ]] && { debugbuild=yes; shift; } || debugbuild=no [[ $# -eq 0 ]] && usage && exit 1 logfile=$1; shift [[ ! -f $logfile ]] && echo "Error: logfile does not exist" >&2 && exit 1 [[ $logfile =~ "build.log" ]] && scriptname=replay-build.sh [[ $logfile =~ "package.log" ]] && scriptname=replay-package.sh makepkgconf=$1; shift [[ -z $makepkgconf ]] && makepkgconf=/etc/makepkg.conf [[ ! -f $makepkgconf ]] && echo "Error: makepkg.conf file does not exist" >&2 && exit 1 if [[ $debug = "yes" ]]; then debugfilter="s/^DEBUG_//g" else debugfilter="/^DEBUG_/d" fi badvars=('CARCH' 'CHOST' 'BUILDENV' 'OPTIONS' 'INTEGRITY_CHECK' 'STRIP_.*' '.*_DIRS' '.*_TARGETS' 'COMPRESS.*' '.*EXT') badvarsfilter=$(IFS='|'; echo "/""${badvars[*]}""/d") { cat <<-EOF #!/bin/bash # $scriptname - Replay commands from a makepkg logfile # This script was created using the following configuration: # Log file: $logfile # Build settings: $makepkgconf # Debug build: $debugbuild # Example usage: # $ cd <directory containing PKGBUILD> # $ rm -rf pkg src # $ mkdir pkg src # $ makepkg -o # # Do a clean build or install # $ cd src/<top of source tree> # $ ../../$scriptname 2>&1 | tee ../../$logfile.clean # # Then compare the logs # $ cd ../.. # $ diff -u $logfile{,.clean} EOF echo "# Set up environment" grep "^[^#].*=[^(]" "$makepkgconf" | \ sed -re "$badvarsfilter" | \ sed -re "$debugfilter" | \ sed -re "s/^/export /g" echo echo "# Set up command logging" echo 'restoreps4=$PS4' echo "PS4='$ '" echo "set -x" echo echo "# Replay commands from logfile" sed -n "s/^\$ \(.*\)$/\1/gp" < $logfile echo echo "# Restore command logging" echo "set +x" echo "PS4=$restoreps4" } > $scriptname chmod +x "$scriptname" echo "Script written to: $scriptname" ---[end: makereplay]---

On 07/03/2013 01:45 AM, Alain Kalker wrote:

On 07/02/2013 11:07 PM, Dave Reisner wrote:

...

You make zero mention of makepkg or any external build tool in that environment. I'm not sure how you expect anyone to pick up your veil and dispute your method of compilation. Quite frankly, this is rude of you, regardless of whether or not makepkg is at fault. Please take this kind of musdlinging somewhere else, IRC, 4chan, a muddy beach, or whatever.

My final try at proving my point is the following script: it takes a logfile from makepkg -L (with my patch applied), and an optional makepkg.conf file (defaulting to /etc/makepkg), then generates a script containing selected environment variables from the configfile, followed by a replay of all commands executed during the build or package phase (determined by the name of the logfile. Script is self-documenting, but here is the gist of how to use it for TL;DR: Here is an example script generated for the example logfile (for RRTL;DR:) ---[file: clean-build.sh]--- #!/bin/bash

# replay-build.sh - Replay commands from a makepkg logfile # This script was created using the following configuration: # Log file: libxcb-1.9.1-1-x86_64-build.log # Build settings: /etc/makepkg.conf # Debug build: no # Example usage: # $ cd <directory containing PKGBUILD> # $ rm -rf pkg src # $ mkdir pkg src # $ makepkg -o # # Do a clean build or install # $ cd src/<top of source tree> # $ ../../replay-build.sh 2>&1 | tee ../../libxcb-1.9.1-1-x86_64-build.log.clean # # Then compare the logs # $ cd ../.. # $ diff -u libxcb-1.9.1-1-x86_64-build.log{,.clean} # Set up environment export CPPFLAGS="-D_FORTIFY_SOURCE=2" export CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4" export CXXFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4" export LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro" # Set up command logging restoreps4=$PS4 PS4='$ ' set -x # Replay commands from logfile cd /home/miki/pkg/abs/extra/libxcb/src/libxcb-1.9.1 patch -Np1 -i /home/miki/pkg/abs/extra/libxcb/src/libxcb-1.1-no-pthread-stubs.patch patch -Np1 -i /home/miki/pkg/abs/extra/libxcb/src/libxcb-1.9.1-configure-ac-reorder.patch ./autogen.sh --prefix=/usr --enable-xinput --enable-xkb --disable-static make # Restore command logging set +x PS4= ---[end: clean-build.sh]--- Clean, concise, easily adaptable for submission upstream, methinks. Kind regards, Alain