[arch-projects] [DEVTOOLS][PATCH 1/6] makechrootpkg: -I to handle multiple packages
Since commit cb3a6ce, running makechroot 2 times to insert a package in a build directory require to find a directory without PKGBUILD cd /var/empty makechrootpkg -cu -I virtualbox-host-dkms-*-i686.pkg.tar.xz -r <dir> makechrootpkg -I virtualbox-host-dkms-*-i686.pkg.tar.xz -r <dir> cd - makechrootpkg -n -r <dir> This patch allow makechrootpkg to handle more than one package to be installed before the build is run and simplify the previous case in makechrootpkg -ncu -I virtualbox-host-dkms-*-i686.pkg.tar.xz -I virtualbox-guest-dkms-*-i686.pkg.tar.xz -r <dir> Signed-off-by: Sébastien Luttringer <seblu@seblu.net> --- makechrootpkg.in | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index fb91100..9021b3e 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -21,6 +21,7 @@ add_to_db=false run_namcap=false chrootdir= passeddir= +declare -a install_pkgs default_copy=$USER [[ -n $SUDO_USER ]] && default_copy=$SUDO_USER @@ -66,7 +67,7 @@ while getopts 'hcudr:I:l:n' arg; do u) update_first=true ;; d) add_to_db=true ;; r) passeddir="$OPTARG" ;; - I) install_pkg="$OPTARG" ;; + I) install_pkgs+=("$OPTARG") ;; l) copy="$OPTARG" ;; n) run_namcap=true; makepkg_args="$makepkg_args -i" ;; *) makepkg_args="$makepkg_args -$arg $OPTARG" ;; @@ -98,7 +99,7 @@ if (( EUID )); then die 'This script must be run as root.' fi -if [[ ! -f PKGBUILD && -z $install_pkg ]]; then +if [[ ! -f PKGBUILD && -z "${install_pkgs[*]}" ]]; then die 'This must be run in a directory containing a PKGBUILD.' fi @@ -152,14 +153,17 @@ if [[ ! -d $copydir ]] || $clean_first; then exec 8>&- fi -if [[ -n $install_pkg ]]; then - pkgname="${install_pkg##*/}" - cp "$install_pkg" "$copydir/$pkgname" +if [[ -n "${install_pkgs[*]}" ]]; then + declare -i ret=0 + for install_pkg in "${install_pkgs[@]}"; do + pkgname="${install_pkg##*/}" + cp "$install_pkg" "$copydir/$pkgname" - mkarchroot -r "pacman -U /$pkgname --noconfirm" "$copydir" - ret=$? + mkarchroot -r "pacman -U /$pkgname --noconfirm" "$copydir" + (( ret += !! $? )) - rm "$copydir/$pkgname" + rm "$copydir/$pkgname" + done # If there is no PKGBUILD we have done [[ -f PKGBUILD ]] || exit $ret -- Sébastien "Seblu" Luttringer
Signed-off-by: Sébastien Luttringer <seblu@seblu.net> --- makechrootpkg.in | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index 9021b3e..ed8ab88 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -117,7 +117,7 @@ umask 0022 # Note this is the same FD number as in mkarchroot exec 9>"$copydir.lock" if ! flock -n 9; then - stat_busy "Locking chroot copy '$copy'" + stat_busy "Locking chroot copy [$copy]" flock 9 stat_done fi @@ -133,7 +133,7 @@ if [[ ! -d $copydir ]] || $clean_first; then stat_done fi - stat_busy 'Creating clean working copy' + stat_busy "Creating clean working copy [$copy]" use_rsync=false if type -P btrfs >/dev/null; then [[ -d $copydir ]] && btrfs subvolume delete "$copydir" &>/dev/null -- Sébastien "Seblu" Luttringer
Enable btrfs features only if the underlying filesystem is btrfs and not rely on the presence of the btrfs tools. Signed-off-by: Sébastien Luttringer <seblu@seblu.net> --- makechrootpkg.in | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index ed8ab88..c8b5f87 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -77,6 +77,9 @@ done # Canonicalize chrootdir, getting rid of trailing / chrootdir=$(readlink -e "$passeddir") +# Detect chrootdir filesystem type +chroottype=$(stat -f -c %T "$chrootdir") + if [[ ${copy:0:1} = / ]]; then copydir=$copy else @@ -134,16 +137,14 @@ if [[ ! -d $copydir ]] || $clean_first; then fi stat_busy "Creating clean working copy [$copy]" - use_rsync=false - if type -P btrfs >/dev/null; then - [[ -d $copydir ]] && btrfs subvolume delete "$copydir" &>/dev/null - btrfs subvolume snapshot "$chrootdir/root" "$copydir" &>/dev/null || - use_rsync=true + if [[ "$chroottype" == btrfs ]]; then + if [[ -d $copydir ]]; then + btrfs subvolume delete "$copydir" >/dev/null || + die "Unable to delete subvolume $copydir" + fi + btrfs subvolume snapshot "$chrootdir/root" "$copydir" >/dev/null || + die "Unable to create subvolume $copydir" else - use_rsync=true - fi - - if $use_rsync; then mkdir -p "$copydir" rsync -a --delete -q -W -x "$chrootdir/root/" "$copydir" fi -- Sébastien "Seblu" Luttringer
btrfs COW allow almost gratuitous create/destroy chroot copy. To allow automatic parallel builds the default copy value is $RANDOM when the underlying filesytem is btrfs. Signed-off-by: Sébastien Luttringer <seblu@seblu.net> --- makechrootpkg.in | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index c8b5f87..7a780df 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -55,7 +55,7 @@ usage() { echo '-I <pkg> Install a package into the working copy of the chroot' echo '-l <copy> The directory to use as the working copy of the chroot' echo ' Useful for maintaining multiple copies.' - echo " Default: $default_copy" + echo " Default: $default_copy or random for btrfs" echo '-n Run namcap on the package' exit 1 } @@ -80,12 +80,17 @@ chrootdir=$(readlink -e "$passeddir") # Detect chrootdir filesystem type chroottype=$(stat -f -c %T "$chrootdir") +# Define copy directory if [[ ${copy:0:1} = / ]]; then copydir=$copy -else - [[ -z $copy ]] && copy=$default_copy +elif [[ -n $copy ]]; then copydir="$chrootdir/$copy" +elif [[ "$chroottype" == btrfs ]]; then + copydir="$chrootdir/$RANDOM" +else + copydir="$chrootdir/$default_copy" fi +copy=${copydir##*/} # Pass all arguments after -- right to makepkg makepkg_args="$makepkg_args ${*:$OPTIND}" -- Sébastien "Seblu" Luttringer
On Sat, Mar 2, 2013 at 5:21 AM, Sébastien Luttringer <seblu@seblu.net> wrote:
btrfs COW allow almost gratuitous create/destroy chroot copy.
To allow automatic parallel builds the default copy value is $RANDOM when the underlying filesytem is btrfs.
Signed-off-by: Sébastien Luttringer <seblu@seblu.net> --- makechrootpkg.in | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
diff --git a/makechrootpkg.in b/makechrootpkg.in index c8b5f87..7a780df 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -55,7 +55,7 @@ usage() { echo '-I <pkg> Install a package into the working copy of the chroot' echo '-l <copy> The directory to use as the working copy of the chroot' echo ' Useful for maintaining multiple copies.' - echo " Default: $default_copy" + echo " Default: $default_copy or random for btrfs" echo '-n Run namcap on the package' exit 1 } @@ -80,12 +80,17 @@ chrootdir=$(readlink -e "$passeddir") # Detect chrootdir filesystem type chroottype=$(stat -f -c %T "$chrootdir")
+# Define copy directory if [[ ${copy:0:1} = / ]]; then copydir=$copy -else - [[ -z $copy ]] && copy=$default_copy +elif [[ -n $copy ]]; then copydir="$chrootdir/$copy" +elif [[ "$chroottype" == btrfs ]]; then + copydir="$chrootdir/$RANDOM" +else + copydir="$chrootdir/$default_copy" fi +copy=${copydir##*/}
# Pass all arguments after -- right to makepkg makepkg_args="$makepkg_args ${*:$OPTIND}" -- Sébastien "Seblu" Luttringer
NACK. See comment to patch 6/6.
Signed-off-by: Sébastien Luttringer <seblu@seblu.net> --- makechrootpkg.in | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index 7a780df..0e2d5c0 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -22,6 +22,7 @@ run_namcap=false chrootdir= passeddir= declare -a install_pkgs +declare -i ret=0 default_copy=$USER [[ -n $SUDO_USER ]] && default_copy=$SUDO_USER @@ -160,7 +161,6 @@ if [[ ! -d $copydir ]] || $clean_first; then fi if [[ -n "${install_pkgs[*]}" ]]; then - declare -i ret=0 for install_pkg in "${install_pkgs[@]}"; do pkgname="${install_pkg##*/}" cp "$install_pkg" "$copydir/$pkgname" @@ -277,9 +277,7 @@ cat >"$copydir/chrootbuild" <<EOF export HOME=/build cd /build -sudo -u nobody makepkg $makepkg_args || touch BUILD_FAILED - -[[ -f BUILD_FAILED ]] && exit 1 +sudo -u nobody makepkg $makepkg_args || exit 1 if $run_namcap; then pacman -S --needed --noconfirm namcap @@ -313,7 +311,7 @@ if mkarchroot -r "/chrootbuild" "$copydir"; then done else # Just in case. We returned 1, make sure we fail - touch "$copydir/build/BUILD_FAILED" + ret=1 fi for f in "$copydir"/srcdest/*; do @@ -321,7 +319,6 @@ for f in "$copydir"/srcdest/*; do mv "$f" "$SRCDEST" done -if [[ -e $copydir/build/BUILD_FAILED ]]; then - rm "$copydir/build/BUILD_FAILED" +if (( ret != 0 )); then die "Build failed, check $copydir/build" fi -- Sébastien "Seblu" Luttringer
Avoid to manually cleanup of subvolumes when a build was successful. Really useful with random copy directory. Signed-off-by: Sébastien Luttringer <seblu@seblu.net> --- makechrootpkg.in | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index 0e2d5c0..3f160bc 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -23,6 +23,7 @@ chrootdir= passeddir= declare -a install_pkgs declare -i ret=0 +no_purge=false default_copy=$USER [[ -n $SUDO_USER ]] && default_copy=$SUDO_USER @@ -58,10 +59,11 @@ usage() { echo ' Useful for maintaining multiple copies.' echo " Default: $default_copy or random for btrfs" echo '-n Run namcap on the package' + echo "-p Don't purge the chroot copy after succesful build (btrfs only)" exit 1 } -while getopts 'hcudr:I:l:n' arg; do +while getopts 'hcudr:I:l:np' arg; do case "$arg" in h) usage ;; c) clean_first=true ;; @@ -71,6 +73,7 @@ while getopts 'hcudr:I:l:n' arg; do I) install_pkgs+=("$OPTARG") ;; l) copy="$OPTARG" ;; n) run_namcap=true; makepkg_args="$makepkg_args -i" ;; + p) no_purge=true ;; *) makepkg_args="$makepkg_args -$arg $OPTARG" ;; esac done @@ -321,4 +324,8 @@ done if (( ret != 0 )); then die "Build failed, check $copydir/build" +elif [[ "$chroottype" == btrfs ]] && ! $no_purge; then + btrfs subvolume delete "$copydir" >/dev/null || + error "Unable to delete subvolume $copydir" + rm "$copydir.lock" fi -- Sébastien "Seblu" Luttringer
On Sat, Mar 2, 2013 at 5:21 AM, Sébastien Luttringer <seblu@seblu.net> wrote:
Avoid to manually cleanup of subvolumes when a build was successful. Really useful with random copy directory.
Signed-off-by: Sébastien Luttringer <seblu@seblu.net> --- makechrootpkg.in | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/makechrootpkg.in b/makechrootpkg.in index 0e2d5c0..3f160bc 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -23,6 +23,7 @@ chrootdir= passeddir= declare -a install_pkgs declare -i ret=0 +no_purge=false
default_copy=$USER [[ -n $SUDO_USER ]] && default_copy=$SUDO_USER @@ -58,10 +59,11 @@ usage() { echo ' Useful for maintaining multiple copies.' echo " Default: $default_copy or random for btrfs" echo '-n Run namcap on the package' + echo "-p Don't purge the chroot copy after succesful build (btrfs only)" exit 1 }
-while getopts 'hcudr:I:l:n' arg; do +while getopts 'hcudr:I:l:np' arg; do case "$arg" in h) usage ;; c) clean_first=true ;; @@ -71,6 +73,7 @@ while getopts 'hcudr:I:l:n' arg; do I) install_pkgs+=("$OPTARG") ;; l) copy="$OPTARG" ;; n) run_namcap=true; makepkg_args="$makepkg_args -i" ;; + p) no_purge=true ;; *) makepkg_args="$makepkg_args -$arg $OPTARG" ;; esac done @@ -321,4 +324,8 @@ done
if (( ret != 0 )); then die "Build failed, check $copydir/build" +elif [[ "$chroottype" == btrfs ]] && ! $no_purge; then + btrfs subvolume delete "$copydir" >/dev/null || + error "Unable to delete subvolume $copydir" + rm "$copydir.lock" fi -- Sébastien "Seblu" Luttringer
NACK. I disagree with making this the default. Copydirs named just with numbers are unhelpful on multiuser build servers. Chroots for failed builds are not removed, leading to clutter when builds fail. Copies aren't cheap enough to just leave them around, especially when additional dependencies are installed. Maybe this could be done if we had some kind of old copy garbage collection. Failing that, I'd rather have a single option (e.g. named -T for "temporary") combining both a random copydir, named using: [[ -z $copy ]] && copy=$default_copy copydir="$(mktemp -u "$chrootdir/$copy-XXXXXX")" and unconditionally removing the copydir after the build, whether it failed or not. The default behavior of makechrootpkg (without the option) remains.
Am 02.03.2013 07:32, schrieb Jan Steffens:
I disagree with making this the default. Copydirs named just with numbers are unhelpful on multiuser build servers. Chroots for failed builds are not removed, leading to clutter when builds fail. Copies aren't cheap enough to just leave them around, especially when additional dependencies are installed. Maybe this could be done if we had some kind of old copy garbage collection.
Failing that, I'd rather have a single option (e.g. named -T for "temporary") combining both a random copydir, named using: [[ -z $copy ]] && copy=$default_copy copydir="$(mktemp -u "$chrootdir/$copy-XXXXXX")" and unconditionally removing the copydir after the build, whether it failed or not. The default behavior of makechrootpkg (without the option) remains.
I would agree on that. -- Pierre Schmitz, https://pierre-schmitz.com
participants (3)
-
Jan Steffens
-
Pierre Schmitz
-
Sébastien Luttringer