[arch-projects] [devtools] [PATCH 1/4] commitpkg: check all files at once for version control

newer
[arch-projects] [mkinitcpio] [GIT]...

older
[arch-projects] [devtools] [GIT]...

Dave Reisner

20 Oct 2013 20 Oct '13

8:17 p.m.

Instead of dying at the first sight of an unversioned file, this lets commitpkg dump all known unversioned files at once. Signed-off-by: Dave Reisner --- commitpkg.in | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/commitpkg.in b/commitpkg.in index fe9348b..8dcbd7c 100644 --- a/commitpkg.in +++ b/commitpkg.in @@ -55,24 +55,28 @@ case "$cmd" in ;; esac -# check if all local source files are under version control +# find files which should be under source control +needsversioning=() for s in "${source[@]}"; do - if [[ $s != *://* ]] && ! svn status -v "$s@" | grep -q '^[ AMRX~]'; then - die "%s is not under version control" "$s" - fi + [[ $s != *://* ]] && needsversioning+=("$s") done - -# check if changelog and install files are under version control for i in 'changelog' 'install'; do while read -r file; do # evaluate any bash variables used eval file=\"$(sed 's/^$['\''"]$$.*$\1$/\2/' <<< "$file")\" - if ! svn status -v "${file}" | grep -q '^[ AMRX~]'; then - die "%s is not under version control" "$file" - fi + needsversioning+=("$file") done < <(sed -n "s/^[[:space:]]*$i=//p" PKGBUILD) done +# assert that they really are controlled by SVN +if (( ${#needsversioning[*]} )); then + # svn status's output is only two columns when the status is unknown + while read -r status filename; do + [[ $status = '?' ]] && unversioned+=("$filename") + done < <(svn status -v "${needsversioning[@]}") + (( ${#unversioned[*]} )) && die "%s is not under version control" "${unversioned[@]}" +fi + rsyncopts=(-e ssh -p --chmod=ug=rw,o=r -c -h -L --progress --partial -y) archreleaseopts=() while getopts ':l:a:s:f' flag; do -- 1.8.4.1

Show replies by date

Dave Reisner

20 Oct 20 Oct

8:17 p.m.

New subject: [arch-projects] [devtools] [PATCH 2/4] makechrootpkg: Look harder for -R argument

We shouldn't be in the business of reparsing makepkg's arguments, but since we have to treat the case of repackaging separately, do a better job of trying to find signs of it happening. This change lets you pass the longopt, --repackage, or multiple shortopts such as -RA, and still get the intended effect. Signed-off-by: Dave Reisner --- This still isn't really an exact science, but it improves the current situation. makechrootpkg.in | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index 8c64ae1..00e538a 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -104,11 +104,13 @@ fi makepkg_args="$makepkg_args ${*:$OPTIND}" # See if -R was passed to makepkg -for arg in ${*:$OPTIND}; do - if [[ $arg = -R ]]; then - repack=true - break - fi +for arg in "${@:OPTIND}"; do + case ${arg%%=*} in + -*R*|--repackage) + repack=true + break 2 + ;; + esac done if [[ -n $SUDO_USER ]]; then -- 1.8.4.1

Dave Reisner

8:17 p.m.

New subject: [arch-projects] [devtools] [PATCH 3/4] enforce tabs across scripts

Don't enforce anything else, though. Signed-off-by: Dave Reisner --- arch-nspawn.in | 2 ++ archbuild.in | 2 ++ archco.in | 2 ++ archrelease.in | 2 ++ archrm.in | 2 ++ bash_completion.in | 2 ++ checkpkg.in | 2 ++ commitpkg.in | 2 ++ crossrepomove.in | 2 ++ find-libdeps.in | 2 ++ finddeps.in | 2 ++ lddd.in | 2 ++ makechrootpkg.in | 2 ++ mkarchroot.in | 2 ++ rebuildpkgs.in | 2 ++ zsh_completion.in | 2 ++ 16 files changed, 32 insertions(+) diff --git a/arch-nspawn.in b/arch-nspawn.in index 6900382..30d315e 100644 --- a/arch-nspawn.in +++ b/arch-nspawn.in @@ -102,3 +102,5 @@ exec ${CARCH:+setarch "$CARCH"} systemd-nspawn 2>/dev/null \ --machine "$machine_name" \ "${mount_args[@]}" \ "$@" + +# vim: set noet: diff --git a/archbuild.in b/archbuild.in index b1c96f9..e7d2256 100644 --- a/archbuild.in +++ b/archbuild.in @@ -78,3 +78,5 @@ fi msg "Building in chroot for [${repo}] (${arch})..." exec makechrootpkg -r "${chroots}/${repo}-${arch}" "${makechrootpkg_args[@]}" + +# vim: set noet: diff --git a/archco.in b/archco.in index 5bdc4e6..10424fc 100644 --- a/archco.in +++ b/archco.in @@ -22,3 +22,5 @@ esac for i in "$@"; do svn co "$SVNURL/$i" done + +# vim: set noet: diff --git a/archrelease.in b/archrelease.in index 6f52dbc..0e009b8 100644 --- a/archrelease.in +++ b/archrelease.in @@ -84,3 +84,5 @@ svn commit -q -m "archrelease: copy ${trunk} to $tag_list" || abort stat_done popd >/dev/null + +# vim: set noet: diff --git a/archrm.in b/archrm.in index 7c7139b..3f81066 100644 --- a/archrm.in +++ b/archrm.in @@ -13,3 +13,5 @@ fi #popd rm -rf $1 + +# vim: set noet: diff --git a/bash_completion.in b/bash_completion.in index dd140fc..7afa274 100644 --- a/bash_completion.in +++ b/bash_completion.in @@ -84,3 +84,5 @@ _arch-nspawn() { } && complete -F _arch-nspawn arch-nspawn # ex:et ts=2 sw=2 ft=sh + +# vim: set noet: diff --git a/checkpkg.in b/checkpkg.in index ef46399..9a32d00 100644 --- a/checkpkg.in +++ b/checkpkg.in @@ -83,3 +83,5 @@ for _pkgname in "${pkgname[@]}"; do done msg "Files saved to $TEMPDIR" + +# vim: set noet: diff --git a/commitpkg.in b/commitpkg.in index 8dcbd7c..d584f92 100644 --- a/commitpkg.in +++ b/commitpkg.in @@ -218,3 +218,5 @@ else popd >/dev/null fi fi + +# vim: set noet: diff --git a/crossrepomove.in b/crossrepomove.in index 912504f..b761cd9 100644 --- a/crossrepomove.in +++ b/crossrepomove.in @@ -79,3 +79,5 @@ svn -q rm "source_checkout/${pkgbase}" svn -q commit -m"${scriptname}: Moving ${pkgbase} from ${source_repo} to ${target_repo}" source_checkout popd >/dev/null + +# vim: set noet: diff --git a/find-libdeps.in b/find-libdeps.in index c9b451e..f71a127 100644 --- a/find-libdeps.in +++ b/find-libdeps.in @@ -85,3 +85,5 @@ find . -type f $find_args | while read filename; do done popd >/dev/null + +# vim: set noet: diff --git a/finddeps.in b/finddeps.in index 7a2a3fb..c583a47 100644 --- a/finddeps.in +++ b/finddeps.in @@ -37,3 +37,5 @@ find . -type d | while read d; do done fi done + +# vim: set noet: diff --git a/lddd.in b/lddd.in index 43aa8c1..228b900 100644 --- a/lddd.in +++ b/lddd.in @@ -46,3 +46,5 @@ done sort -u $TEMPDIR/pacman.txt >> $TEMPDIR/possible-rebuilds.txt msg "Files saved to $TEMPDIR" + +# vim: set noet: diff --git a/makechrootpkg.in b/makechrootpkg.in index 00e538a..2e0cec4 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -381,3 +381,5 @@ if (( ret != 0 )); then else true fi + +# vim: set noet: diff --git a/mkarchroot.in b/mkarchroot.in index 7cdb274..88c278f 100644 --- a/mkarchroot.in +++ b/mkarchroot.in @@ -77,3 +77,5 @@ exec arch-nspawn \ ${makepkg_conf:+-M "$makepkg_conf"} \ ${cache_dir:+-c "$cache_dir"} \ "$working_dir" locale-gen + +# vim: set noet: diff --git a/rebuildpkgs.in b/rebuildpkgs.in index 2f71c40..cc5fb2b 100644 --- a/rebuildpkgs.in +++ b/rebuildpkgs.in @@ -95,3 +95,5 @@ if [[ -n $FAILED ]]; then fi msg 'SVN pkgbumps in svn-packages/ - commit when ready' + +# vim: set noet: diff --git a/zsh_completion.in b/zsh_completion.in index 4c6dd99..30c25bf 100644 --- a/zsh_completion.in +++ b/zsh_completion.in @@ -69,3 +69,5 @@ _devtools() { } _devtools + +# vim: set noet: -- 1.8.4.1

Dave Reisner

8:17 p.m.

New subject: [arch-projects] [devtools] [PATCH 4/4] common: implement find_cached_package

This function (currently) searches through $PWD and $PKGDEST looking for a tarball matching the requested package name, architecture, and pkgver. If found, it writes the full path to the located package to stdout and returns 0, else 1. If more than 1 match is found, it's treated as an error and the user will need to figure out what to do. Use this in checkpkg and commitpkg, which previously implemented their own less complete logic, to locate the build artifacts they rely on. Signed-off-by: Dave Reisner --- This fixes a regression I caused in 9c85d116f042 which didn't consider finding the package in PKGDEST. I took the heavy handed approach and wrote a generic function to find cached packages since we do this in both checkpkg and commitpkg (and maybe some day elsewhere). Neither of the existing solutions are particularly thorough. checkpkg.in | 14 ++++-------- commitpkg.in | 24 +++------------------ lib/common.sh | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 76 insertions(+), 31 deletions(-) diff --git a/checkpkg.in b/checkpkg.in index 9a32d00..3ad912f 100644 --- a/checkpkg.in +++ b/checkpkg.in @@ -29,18 +29,12 @@ STARTDIR=$(pwd) TEMPDIR=$(mktemp -d --tmpdir checkpkg-script.XXXX) for _pkgname in "${pkgname[@]}"; do - pkgfile=(${_pkgname}-$(get_full_version $_pkgname)-${CARCH}.pkg.tar?(.?z)) - if (( ${#pkgfile[*]} != 1 )); then - die 'Ambiguous package name: %s\n' "${pkgfile[*]}" + target_pkgver=$(get_full_version "$_pkgname") + if ! pkgfile=$(find_cached_package "$_pkgname" "$target_pkgver" "$CARCH"); then + die 'tarball not found for package: %s' "${_pkgname}-$target_pkgver" fi - if [[ -f "$STARTDIR/$pkgfile" ]]; then - ln -s "$STARTDIR/$pkgfile" "$TEMPDIR/$pkgfile" - elif [[ -f "$PKGDEST/$pkgfile" ]]; then - ln -s "$PKGDEST/$pkgfile" "$TEMPDIR/$pkgfile" - else - die "File \"$pkgfile\" doesn't exist" - fi + ln -s "$pkgfile" "$TEMPDIR" pkgurl=$(pacman -Spdd --print-format '%l' --noconfirm "$_pkgname") diff --git a/commitpkg.in b/commitpkg.in index d584f92..809ff81 100644 --- a/commitpkg.in +++ b/commitpkg.in @@ -2,22 +2,6 @@ m4_include(lib/common.sh) -getpkgfile() { - case $# in - 0) - error 'No canonical package found!' - return 1 - ;; - [!1]) - error 'Failed to canonicalize package name -- multiple packages found:' - msg2 '%s' "$@" - return 1 - ;; - esac - - echo "$1" -} - # Source makepkg.conf; fail if it is not found if [[ -r '/etc/makepkg.conf' ]]; then source '/etc/makepkg.conf' @@ -99,9 +83,8 @@ for _arch in ${arch[@]}; do for _pkgname in ${pkgname[@]}; do fullver=$(get_full_version $_pkgname) - if pkgfile=$(shopt -s nullglob; - getpkgfile "${PKGDEST+$PKGDEST/}$_pkgname-$fullver-${_arch}".pkg.tar.?z); then - if grep -q "packager = Unknown Packager" <(bsdtar -xOqf $pkgfile .PKGINFO); then + if pkgfile=$(find_cached_package "$_pkgname" "$_arch" "$fullver"); then + if grep -q "packager = Unknown Packager" <(bsdtar -xOqf "$pkgfile" .PKGINFO); then die "PACKAGER was not set when building package" fi fi @@ -151,8 +134,7 @@ for _arch in ${arch[@]}; do for _pkgname in ${pkgname[@]}; do fullver=$(get_full_version $_pkgname) - if ! pkgfile=$(shopt -s nullglob; - getpkgfile "${PKGDEST+$PKGDEST/}$_pkgname-$fullver-${_arch}".pkg.tar.?z); then + if ! pkgfile=$(find_cached_package "$_pkgname" "$fullver" "${_arch}"); then warning "Skipping $_pkgname-$fullver-$_arch: failed to locate package file" skip_arches+=($_arch) continue 2 diff --git a/lib/common.sh b/lib/common.sh index 3ec26ff..1812cb7 100644 --- a/lib/common.sh +++ b/lib/common.sh @@ -1,6 +1,8 @@ # Avoid any encoding problems export LANG=C +shopt -s extglob + # check if messages are to be printed using color unset ALL_OFF BOLD BLUE GREEN RED YELLOW if [[ -t 2 ]]; then @@ -154,3 +156,70 @@ slock() { stat_done fi } + +## +# usage: pkgver_equal( $pkgver1, $pkgver2 ) +## +pkgver_equal() { + local left right + + if [[ $1 = *-* && $2 = *-* ]]; then + # if both versions have a pkgrel, then they must be an exact match + [[ $1 = "$2" ]] + else + # otherwise, trim any pkgrel and compare the bare version. + [[ ${1%%-*} = "${2%%-*}" ]] + fi +} + +## +# usage: find_cached_package( $pkgname, $pkgver, $arch ) +# +# $pkgver can be supplied with or without a pkgrel appended. +# If not supplied, any pkgrel will be matched. +## +find_cached_package() { + local searchdirs=("$PWD" "$PKGDEST") results=() + local targetname=$1 targetver=$2 targetarch=$3 + local dir pkg pkgbasename pkgparts name ver rel arch size results + + for dir in "${searchdirs[@]}"; do + [[ -d $dir ]] || continue + + for pkg in "$dir"/*.pkg.tar?(.?z); do + [[ -f $pkg ]] || continue + + # split apart package filename into parts + pkgbasename=${pkg##*/} + pkgbasename=${pkgbasename%.pkg.tar?(.?z)} + + arch=${pkgbasename##*-} + pkgbasename=${pkgbasename%-"$arch"} + + rel=${pkgbasename##*-} + pkgbasename=${pkgbasename%-"$rel"} + + ver=${pkgbasename##*-} + name=${pkgbasename%-"$ver"} + + if [[ $targetname = "$name" && $targetarch = "$arch" ]] && + pkgver_equal "$targetver" "$ver-$rel"; then + results+=("$pkg") + fi + done + done + + case ${#results[*]} in + 0) + return 1 + ;; + 1) + printf '%s\n' "$results" + return 0 + ;; + *) + error 'Multiple packages found:' + printf '\t%s\n' "${results[@]}" + return 1 + esac +} -- 1.8.4.1

Dave Reisner

21 Oct 21 Oct

12:24 p.m.

New subject: [arch-projects] [devtools] [PATCH 4/4] common: implement find_cached_package

On Sun, Oct 20, 2013 at 04:17:45PM -0400, Dave Reisner wrote:

...

This function (currently) searches through $PWD and $PKGDEST looking for a tarball matching the requested package name, architecture, and pkgver. If found, it writes the full path to the located package to stdout and returns 0, else 1. If more than 1 match is found, it's treated as an error and the user will need to figure out what to do.

Use this in checkpkg and commitpkg, which previously implemented their own less complete logic, to locate the build artifacts they rely on.

Signed-off-by: Dave Reisner --- This fixes a regression I caused in 9c85d116f042 which didn't consider finding the package in PKGDEST. I took the heavy handed approach and wrote a generic function to find cached packages since we do this in both checkpkg and commitpkg (and maybe some day elsewhere). Neither of the existing solutions are particularly thorough.

Replying to myself...

...

checkpkg.in | 14 ++++-------- commitpkg.in | 24 +++------------------ lib/common.sh | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 76 insertions(+), 31 deletions(-)

diff --git a/checkpkg.in b/checkpkg.in index 9a32d00..3ad912f 100644 --- a/checkpkg.in +++ b/checkpkg.in @@ -29,18 +29,12 @@ STARTDIR=$(pwd) TEMPDIR=$(mktemp -d --tmpdir checkpkg-script.XXXX)

for _pkgname in "${pkgname[@]}"; do - pkgfile=(${_pkgname}-$(get_full_version $_pkgname)-${CARCH}.pkg.tar?(.?z)) - if (( ${#pkgfile[*]} != 1 )); then - die 'Ambiguous package name: %s\n' "${pkgfile[*]}" + target_pkgver=$(get_full_version "$_pkgname") + if ! pkgfile=$(find_cached_package "$_pkgname" "$target_pkgver" "$CARCH"); then + die 'tarball not found for package: %s' "${_pkgname}-$target_pkgver" fi

- if [[ -f "$STARTDIR/$pkgfile" ]]; then - ln -s "$STARTDIR/$pkgfile" "$TEMPDIR/$pkgfile" - elif [[ -f "$PKGDEST/$pkgfile" ]]; then - ln -s "$PKGDEST/$pkgfile" "$TEMPDIR/$pkgfile" - else - die "File \"$pkgfile\" doesn't exist" - fi + ln -s "$pkgfile" "$TEMPDIR"

pkgurl=$(pacman -Spdd --print-format '%l' --noconfirm "$_pkgname")

diff --git a/commitpkg.in b/commitpkg.in index d584f92..809ff81 100644 --- a/commitpkg.in +++ b/commitpkg.in @@ -2,22 +2,6 @@

m4_include(lib/common.sh)

-getpkgfile() { - case $# in - 0) - error 'No canonical package found!' - return 1 - ;; - [!1]) - error 'Failed to canonicalize package name -- multiple packages found:' - msg2 '%s' "$@" - return 1 - ;; - esac - - echo "$1" -} - # Source makepkg.conf; fail if it is not found if [[ -r '/etc/makepkg.conf' ]]; then source '/etc/makepkg.conf' @@ -99,9 +83,8 @@ for _arch in ${arch[@]}; do for _pkgname in ${pkgname[@]}; do fullver=$(get_full_version $_pkgname)

- if pkgfile=$(shopt -s nullglob; - getpkgfile "${PKGDEST+$PKGDEST/}$_pkgname-$fullver-${_arch}".pkg.tar.?z); then - if grep -q "packager = Unknown Packager" <(bsdtar -xOqf $pkgfile .PKGINFO); then + if pkgfile=$(find_cached_package "$_pkgname" "$_arch" "$fullver"); then + if grep -q "packager = Unknown Packager" <(bsdtar -xOqf "$pkgfile" .PKGINFO); then die "PACKAGER was not set when building package" fi fi @@ -151,8 +134,7 @@ for _arch in ${arch[@]}; do for _pkgname in ${pkgname[@]}; do fullver=$(get_full_version $_pkgname)

- if ! pkgfile=$(shopt -s nullglob; - getpkgfile "${PKGDEST+$PKGDEST/}$_pkgname-$fullver-${_arch}".pkg.tar.?z); then + if ! pkgfile=$(find_cached_package "$_pkgname" "$fullver" "${_arch}"); then warning "Skipping $_pkgname-$fullver-$_arch: failed to locate package file" skip_arches+=($_arch) continue 2 diff --git a/lib/common.sh b/lib/common.sh index 3ec26ff..1812cb7 100644 --- a/lib/common.sh +++ b/lib/common.sh @@ -1,6 +1,8 @@ # Avoid any encoding problems export LANG=C

+shopt -s extglob + # check if messages are to be printed using color unset ALL_OFF BOLD BLUE GREEN RED YELLOW if [[ -t 2 ]]; then @@ -154,3 +156,70 @@ slock() { stat_done fi } + +## +# usage: pkgver_equal( $pkgver1, $pkgver2 ) +## +pkgver_equal() { + local left right + + if [[ $1 = *-* && $2 = *-* ]]; then + # if both versions have a pkgrel, then they must be an exact match + [[ $1 = "$2" ]] + else + # otherwise, trim any pkgrel and compare the bare version. + [[ ${1%%-*} = "${2%%-*}" ]] + fi +} + +## +# usage: find_cached_package( $pkgname, $pkgver, $arch ) +# +# $pkgver can be supplied with or without a pkgrel appended. +# If not supplied, any pkgrel will be matched. +## +find_cached_package() { + local searchdirs=("$PWD" "$PKGDEST") results=() + local targetname=$1 targetver=$2 targetarch=$3 + local dir pkg pkgbasename pkgparts name ver rel arch size results + + for dir in "${searchdirs[@]}"; do + [[ -d $dir ]] || continue + + for pkg in "$dir"/*.pkg.tar?(.?z); do + [[ -f $pkg ]] || continue + + # split apart package filename into parts + pkgbasename=${pkg##*/} + pkgbasename=${pkgbasename%.pkg.tar?(.?z)} + + arch=${pkgbasename##*-} + pkgbasename=${pkgbasename%-"$arch"} + + rel=${pkgbasename##*-} + pkgbasename=${pkgbasename%-"$rel"} + + ver=${pkgbasename##*-} + name=${pkgbasename%-"$ver"} + + if [[ $targetname = "$name" && $targetarch = "$arch" ]] && + pkgver_equal "$targetver" "$ver-$rel"; then + results+=("$pkg") + fi + done + done + + case ${#results[*]} in + 0) + return 1

Need to print an error here because we don't return enough to the caller to let them differentiate between no results and multiple results.

...

+ ;; + 1) + printf '%s\n' "$results" + return 0 + ;; + *) + error 'Multiple packages found:' + printf '\t%s\n' "${results[@]}" + return 1 + esac +} -- 1.8.4.1

3849

Age (days ago)

3850

Last active (days ago)

List overview

Download

4 comments

2 participants

participants (2)

Dave Reisner
Dave Reisner