[arch-projects] [DEVTOOLS][PATCH 1/6] makechrootpkg: -I to handle multiple packages

newer
[arch-projects] [devtools] [GIT]...

older
[arch-projects] [devtools][PATCH]...

Sébastien Luttringer

2 Mar 2013 2 Mar '13

4:21 a.m.

Since commit cb3a6ce, running makechroot 2 times to insert a package in a build directory require to find a directory without PKGBUILD

cd /var/empty makechrootpkg -cu -I virtualbox-host-dkms-*-i686.pkg.tar.xz -r <dir> makechrootpkg -I virtualbox-host-dkms-*-i686.pkg.tar.xz -r <dir> cd - makechrootpkg -n -r <dir>

This patch allow makechrootpkg to handle more than one package to be installed before the build is run and simplify the previous case in

makechrootpkg -ncu -I virtualbox-host-dkms-*-i686.pkg.tar.xz -I virtualbox-guest-dkms-*-i686.pkg.tar.xz -r <dir>

Signed-off-by: Sébastien Luttringer seblu@seblu.net --- makechrootpkg.in | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/makechrootpkg.in b/makechrootpkg.in index fb91100..9021b3e 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -21,6 +21,7 @@ add_to_db=false run_namcap=false chrootdir= passeddir= +declare -a install_pkgs

default_copy=$USER [[ -n $SUDO_USER ]] && default_copy=$SUDO_USER @@ -66,7 +67,7 @@ while getopts 'hcudr:I:l:n' arg; do u) update_first=true ;; d) add_to_db=true ;; r) passeddir="$OPTARG" ;; - I) install_pkg="$OPTARG" ;; + I) install_pkgs+=("$OPTARG") ;; l) copy="$OPTARG" ;; n) run_namcap=true; makepkg_args="$makepkg_args -i" ;; *) makepkg_args="$makepkg_args -$arg $OPTARG" ;; @@ -98,7 +99,7 @@ if (( EUID )); then die 'This script must be run as root.' fi

-if [[ ! -f PKGBUILD && -z $install_pkg ]]; then +if [[ ! -f PKGBUILD && -z "${install_pkgs[*]}" ]]; then die 'This must be run in a directory containing a PKGBUILD.' fi

@@ -152,14 +153,17 @@ if [[ ! -d $copydir ]] || $clean_first; then exec 8>&- fi

-if [[ -n $install_pkg ]]; then - pkgname="${install_pkg##*/}" - cp "$install_pkg" "$copydir/$pkgname" +if [[ -n "${install_pkgs[*]}" ]]; then + declare -i ret=0 + for install_pkg in "${install_pkgs[@]}"; do + pkgname="${install_pkg##*/}" + cp "$install_pkg" "$copydir/$pkgname"

- mkarchroot -r "pacman -U /$pkgname --noconfirm" "$copydir" - ret=$? + mkarchroot -r "pacman -U /$pkgname --noconfirm" "$copydir" + (( ret += !! $? ))

- rm "$copydir/$pkgname" + rm "$copydir/$pkgname" + done

# If there is no PKGBUILD we have done [[ -f PKGBUILD ]] || exit $ret

-- Sébastien "Seblu" Luttringer

Show replies by date

Sébastien Luttringer

2 Mar 2 Mar

4:21 a.m.

New subject: [arch-projects] [DEVTOOLS][PATCH 2/6] makechrootpkg: Display uniform $copy in messages

Signed-off-by: Sébastien Luttringer seblu@seblu.net --- makechrootpkg.in | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/makechrootpkg.in b/makechrootpkg.in index 9021b3e..ed8ab88 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -117,7 +117,7 @@ umask 0022 # Note this is the same FD number as in mkarchroot exec 9>"$copydir.lock" if ! flock -n 9; then - stat_busy "Locking chroot copy '$copy'" + stat_busy "Locking chroot copy [$copy]" flock 9 stat_done fi @@ -133,7 +133,7 @@ if [[ ! -d $copydir ]] || $clean_first; then stat_done fi

- stat_busy 'Creating clean working copy' + stat_busy "Creating clean working copy [$copy]" use_rsync=false if type -P btrfs >/dev/null; then [[ -d $copydir ]] && btrfs subvolume delete "$copydir" &>/dev/null

-- Sébastien "Seblu" Luttringer

Sébastien Luttringer

4:21 a.m.

New subject: [arch-projects] [DEVTOOLS][PATCH 3/6] makechrootpkg: Properly detect filesystem type

Enable btrfs features only if the underlying filesystem is btrfs and not rely on the presence of the btrfs tools.

Signed-off-by: Sébastien Luttringer seblu@seblu.net --- makechrootpkg.in | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/makechrootpkg.in b/makechrootpkg.in index ed8ab88..c8b5f87 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -77,6 +77,9 @@ done # Canonicalize chrootdir, getting rid of trailing / chrootdir=$(readlink -e "$passeddir")

+# Detect chrootdir filesystem type +chroottype=$(stat -f -c %T "$chrootdir") + if [[ ${copy:0:1} = / ]]; then copydir=$copy else @@ -134,16 +137,14 @@ if [[ ! -d $copydir ]] || $clean_first; then fi

stat_busy "Creating clean working copy [$copy]" - use_rsync=false - if type -P btrfs >/dev/null; then - [[ -d $copydir ]] && btrfs subvolume delete "$copydir" &>/dev/null - btrfs subvolume snapshot "$chrootdir/root" "$copydir" &>/dev/null || - use_rsync=true + if [[ "$chroottype" == btrfs ]]; then + if [[ -d $copydir ]]; then + btrfs subvolume delete "$copydir" >/dev/null || + die "Unable to delete subvolume $copydir" + fi + btrfs subvolume snapshot "$chrootdir/root" "$copydir" >/dev/null || + die "Unable to create subvolume $copydir" else - use_rsync=true - fi - - if $use_rsync; then mkdir -p "$copydir" rsync -a --delete -q -W -x "$chrootdir/root/" "$copydir" fi

-- Sébastien "Seblu" Luttringer

Sébastien Luttringer

4:21 a.m.

New subject: [arch-projects] [DEVTOOLS][PATCH 4/6] makechrootpkg: Use random copy directory with btrfs

btrfs COW allow almost gratuitous create/destroy chroot copy.

To allow automatic parallel builds the default copy value is $RANDOM when the underlying filesytem is btrfs.

Signed-off-by: Sébastien Luttringer seblu@seblu.net --- makechrootpkg.in | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/makechrootpkg.in b/makechrootpkg.in index c8b5f87..7a780df 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -55,7 +55,7 @@ usage() { echo '-I <pkg> Install a package into the working copy of the chroot' echo '-l <copy> The directory to use as the working copy of the chroot' echo ' Useful for maintaining multiple copies.' - echo " Default: $default_copy" + echo " Default: $default_copy or random for btrfs" echo '-n Run namcap on the package' exit 1 } @@ -80,12 +80,17 @@ chrootdir=$(readlink -e "$passeddir") # Detect chrootdir filesystem type chroottype=$(stat -f -c %T "$chrootdir")

+# Define copy directory if [[ ${copy:0:1} = / ]]; then copydir=$copy -else - [[ -z $copy ]] && copy=$default_copy +elif [[ -n $copy ]]; then copydir="$chrootdir/$copy" +elif [[ "$chroottype" == btrfs ]]; then + copydir="$chrootdir/$RANDOM" +else + copydir="$chrootdir/$default_copy" fi +copy=${copydir##*/}

# Pass all arguments after -- right to makepkg makepkg_args="$makepkg_args ${*:$OPTIND}"

-- Sébastien "Seblu" Luttringer

Jan Steffens

6:15 a.m.

New subject: [arch-projects] [DEVTOOLS][PATCH 4/6] makechrootpkg: Use random copy directory with btrfs

On Sat, Mar 2, 2013 at 5:21 AM, Sébastien Luttringer seblu@seblu.net wrote:

...

btrfs COW allow almost gratuitous create/destroy chroot copy.

To allow automatic parallel builds the default copy value is $RANDOM when the underlying filesytem is btrfs.

Signed-off-by: Sébastien Luttringer seblu@seblu.net

makechrootpkg.in | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/makechrootpkg.in b/makechrootpkg.in index c8b5f87..7a780df 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -55,7 +55,7 @@ usage() { echo '-I <pkg> Install a package into the working copy of the chroot' echo '-l <copy> The directory to use as the working copy of the chroot' echo ' Useful for maintaining multiple copies.'
  echo "           Default: $default_copy"
  echo "           Default: $default_copy or random for btrfs"
  echo '-n         Run namcap on the package'
  exit 1
} @@ -80,12 +80,17 @@ chrootdir=$(readlink -e "$passeddir") # Detect chrootdir filesystem type chroottype=$(stat -f -c %T "$chrootdir")

+# Define copy directory if [[ ${copy:0:1} = / ]]; then copydir=$copy -else
  [[ -z $copy ]] && copy=$default_copy
+elif [[ -n $copy ]]; then copydir="$chrootdir/$copy" +elif [[ "$chroottype" == btrfs ]]; then
  copydir="$chrootdir/$RANDOM"
+else
  copydir="$chrootdir/$default_copy"
fi +copy=${copydir##*/}

# Pass all arguments after -- right to makepkg makepkg_args="$makepkg_args ${*:$OPTIND}" -- Sébastien "Seblu" Luttringer

NACK. See comment to patch 6/6.

Sébastien Luttringer

4:21 a.m.

New subject: [arch-projects] [DEVTOOLS][PATCH 5/6] makechrootpkg: use var instead of file in $copydir

Signed-off-by: Sébastien Luttringer seblu@seblu.net --- makechrootpkg.in | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/makechrootpkg.in b/makechrootpkg.in index 7a780df..0e2d5c0 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -22,6 +22,7 @@ run_namcap=false chrootdir= passeddir= declare -a install_pkgs +declare -i ret=0

default_copy=$USER [[ -n $SUDO_USER ]] && default_copy=$SUDO_USER @@ -160,7 +161,6 @@ if [[ ! -d $copydir ]] || $clean_first; then fi

if [[ -n "${install_pkgs[*]}" ]]; then - declare -i ret=0 for install_pkg in "${install_pkgs[@]}"; do pkgname="${install_pkg##*/}" cp "$install_pkg" "$copydir/$pkgname" @@ -277,9 +277,7 @@ cat >"$copydir/chrootbuild" <<EOF export HOME=/build

cd /build -sudo -u nobody makepkg $makepkg_args || touch BUILD_FAILED - -[[ -f BUILD_FAILED ]] && exit 1 +sudo -u nobody makepkg $makepkg_args || exit 1

if $run_namcap; then pacman -S --needed --noconfirm namcap @@ -313,7 +311,7 @@ if mkarchroot -r "/chrootbuild" "$copydir"; then done else # Just in case. We returned 1, make sure we fail - touch "$copydir/build/BUILD_FAILED" + ret=1 fi

for f in "$copydir"/srcdest/*; do @@ -321,7 +319,6 @@ for f in "$copydir"/srcdest/*; do mv "$f" "$SRCDEST" done

-if [[ -e $copydir/build/BUILD_FAILED ]]; then - rm "$copydir/build/BUILD_FAILED" +if (( ret != 0 )); then die "Build failed, check $copydir/build" fi

-- Sébastien "Seblu" Luttringer

Sébastien Luttringer

4:21 a.m.

New subject: [arch-projects] [DEVTOOLS][PATCH 6/6] makechrootpkg: Remove btrfs copy after successful build

Avoid to manually cleanup of subvolumes when a build was successful. Really useful with random copy directory.

Signed-off-by: Sébastien Luttringer seblu@seblu.net --- makechrootpkg.in | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/makechrootpkg.in b/makechrootpkg.in index 0e2d5c0..3f160bc 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -23,6 +23,7 @@ chrootdir= passeddir= declare -a install_pkgs declare -i ret=0 +no_purge=false

default_copy=$USER [[ -n $SUDO_USER ]] && default_copy=$SUDO_USER @@ -58,10 +59,11 @@ usage() { echo ' Useful for maintaining multiple copies.' echo " Default: $default_copy or random for btrfs" echo '-n Run namcap on the package' + echo "-p Don't purge the chroot copy after succesful build (btrfs only)" exit 1 }

-while getopts 'hcudr:I:l:n' arg; do +while getopts 'hcudr:I:l:np' arg; do case "$arg" in h) usage ;; c) clean_first=true ;; @@ -71,6 +73,7 @@ while getopts 'hcudr:I:l:n' arg; do I) install_pkgs+=("$OPTARG") ;; l) copy="$OPTARG" ;; n) run_namcap=true; makepkg_args="$makepkg_args -i" ;; + p) no_purge=true ;; *) makepkg_args="$makepkg_args -$arg $OPTARG" ;; esac done @@ -321,4 +324,8 @@ done

if (( ret != 0 )); then die "Build failed, check $copydir/build" +elif [[ "$chroottype" == btrfs ]] && ! $no_purge; then + btrfs subvolume delete "$copydir" >/dev/null || + error "Unable to delete subvolume $copydir" + rm "$copydir.lock" fi

-- Sébastien "Seblu" Luttringer

Jan Steffens

6:32 a.m.

New subject: [arch-projects] [DEVTOOLS][PATCH 6/6] makechrootpkg: Remove btrfs copy after successful build

On Sat, Mar 2, 2013 at 5:21 AM, Sébastien Luttringer seblu@seblu.net wrote:

...

Avoid to manually cleanup of subvolumes when a build was successful. Really useful with random copy directory.

Signed-off-by: Sébastien Luttringer seblu@seblu.net

makechrootpkg.in | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/makechrootpkg.in b/makechrootpkg.in index 0e2d5c0..3f160bc 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -23,6 +23,7 @@ chrootdir= passeddir= declare -a install_pkgs declare -i ret=0 +no_purge=false

default_copy=$USER [[ -n $SUDO_USER ]] && default_copy=$SUDO_USER @@ -58,10 +59,11 @@ usage() { echo ' Useful for maintaining multiple copies.' echo " Default: $default_copy or random for btrfs" echo '-n Run namcap on the package'
  echo "-p         Don't purge the chroot copy after succesful build (btrfs only)"
  exit 1
}

-while getopts 'hcudr:I:l:n' arg; do +while getopts 'hcudr:I:l:np' arg; do case "$arg" in h) usage ;; c) clean_first=true ;; @@ -71,6 +73,7 @@ while getopts 'hcudr:I:l:n' arg; do I) install_pkgs+=("$OPTARG") ;; l) copy="$OPTARG" ;; n) run_namcap=true; makepkg_args="$makepkg_args -i" ;;
          p) no_purge=true ;;
          *) makepkg_args="$makepkg_args -$arg $OPTARG" ;;
  esac
done @@ -321,4 +324,8 @@ done

if (( ret != 0 )); then die "Build failed, check $copydir/build" +elif [[ "$chroottype" == btrfs ]] && ! $no_purge; then
  btrfs subvolume delete "$copydir" >/dev/null ||
          error "Unable to delete subvolume $copydir"
  rm "$copydir.lock"
fi

Sébastien "Seblu" Luttringer

NACK.

I disagree with making this the default. Copydirs named just with numbers are unhelpful on multiuser build servers. Chroots for failed builds are not removed, leading to clutter when builds fail. Copies aren't cheap enough to just leave them around, especially when additional dependencies are installed. Maybe this could be done if we had some kind of old copy garbage collection.

Failing that, I'd rather have a single option (e.g. named -T for "temporary") combining both a random copydir, named using: [[ -z $copy ]] && copy=$default_copy copydir="$(mktemp -u "$chrootdir/$copy-XXXXXX")" and unconditionally removing the copydir after the build, whether it failed or not. The default behavior of makechrootpkg (without the option) remains.

Pierre Schmitz

10 Mar 10 Mar

9:44 a.m.

New subject: [arch-projects] [DEVTOOLS][PATCH 6/6] makechrootpkg: Remove btrfs copy after successful build

Am 02.03.2013 07:32, schrieb Jan Steffens:

...

I disagree with making this the default. Copydirs named just with numbers are unhelpful on multiuser build servers. Chroots for failed builds are not removed, leading to clutter when builds fail. Copies aren't cheap enough to just leave them around, especially when additional dependencies are installed. Maybe this could be done if we had some kind of old copy garbage collection.

Failing that, I'd rather have a single option (e.g. named -T for "temporary") combining both a random copydir, named using: [[ -z $copy ]] && copy=$default_copy copydir="$(mktemp -u "$chrootdir/$copy-XXXXXX")" and unconditionally removing the copydir after the build, whether it failed or not. The default behavior of makechrootpkg (without the option) remains.

I would agree on that.

-- Pierre Schmitz, https://pierre-schmitz.com

3928

Age (days ago)

3936

Last active (days ago)

List overview

Download

8 comments

3 participants

tags (0)

participants (3)

Jan Steffens
Pierre Schmitz
Sébastien Luttringer