[arch-projects] [dbscripts] [PATCH v2 0/3] Misc touchup
From: Luke Shumaker <lukeshu@parabola.nu> These are a few minor unrelated touch-ups that I noticed when doing the PKGEXT work. There's no real theme to them. v2: - Use Bash 4.4 feature to avoid introducing temporary variables in the printf commit - Mention splitting declaring and setting the result variable in the __getCheckSum commit message. Luke Shumaker (3): test: common.bash:__getCheckSum: Don't rely on IFS test: Fixup glob matching Update messages to make fuller use of printf formatters cron-jobs/ftpdir-cleanup | 8 ++++---- cron-jobs/integrity-check | 2 +- cron-jobs/sourceballs | 8 ++++---- db-functions | 4 ++-- db-move | 4 ++-- db-remove | 2 +- db-repo-add | 2 +- db-repo-remove | 2 +- db-update | 2 +- test/cases/ftpdir-cleanup.bats | 4 ++-- test/cases/sourceballs.bats | 4 ++-- test/lib/common.bash | 17 ++++++++++++++--- testing2x | 2 +- 13 files changed, 36 insertions(+), 25 deletions(-) -- 2.16.1
From: Luke Shumaker <lukeshu@parabola.nu> I managed to stumble across a bug in BATS where the run() function screwed with the global IFS. The bug has been fixed in git, but isn't in a release yet. https://github.com/sstephenson/bats/issues/89 Anyway, this bug breaks __getCheckSum(). Fortunately, we have avoided tripping it so far because luck has it that we never call __getCheckSum() after run() in the same test. So, there's nothing actually broken here, but it makes me nervous. So go ahead and modify __getCheckSum to not rely on IFS. And, while we're at it: declare the result variable and set it as separate commands. Doing both in the same command masks the exit code of the subshell expansion. We don't explicitly check the exit code, but BATS runs the test suite with `set -e`, so splitting it does mean that BATS will now detect errors from sha1sum. We don't really expect that to happen, but if BATS will give us error checking on it for free, why not? v2: - commit message: Mention spliting declare/set into separate commands. --- test/lib/common.bash | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/test/lib/common.bash b/test/lib/common.bash index 568a541..5411641 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -9,8 +9,9 @@ __updatePKGBUILD() { } __getCheckSum() { - local result=($(sha1sum $1)) - echo ${result[0]} + local result + result="$(sha1sum "$1")" + echo "${result%% *}" } __buildPackage() { -- 2.16.1
From: Luke Shumaker <lukeshu@parabola.nu> - ftpdir-cleanup.bats: Glob expansion does not occur in [[ -f ]] tests. The [[ ! -f .../${pkgname}-*${PKGEXT} ]] checks were checking that there were no files containing a literal '*' for that part of their name. Obviously, this isn't what was intended. - sourceballs.bats: [ -r ] checks explode if the glob returns >1 file. Avoid using them if the path being checked contains a glob. - common.bash: Globbing happens on the RHS of a [[ = ]] test. This means that we must quote variables on the RHS that are to be taken verbatim. This is surprising, because we don't need to quote the LHS. --- test/cases/ftpdir-cleanup.bats | 4 ++-- test/cases/sourceballs.bats | 4 ++-- test/lib/common.bash | 12 +++++++++++- 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/test/cases/ftpdir-cleanup.bats b/test/cases/ftpdir-cleanup.bats index fd485f3..8c713c6 100644 --- a/test/cases/ftpdir-cleanup.bats +++ b/test/cases/ftpdir-cleanup.bats @@ -13,8 +13,8 @@ __checkRepoRemovedPackage() { local pkgname for pkgname in $(__getPackageNamesFromPackageBase ${pkgbase}); do - [[ ! -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-*${PKGEXT} ]] - [[ ! -f ${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}-*${PKGEXT} ]] + ! __isGlobfile "${FTP_BASE}/${PKGPOOL}/${pkgname}"-*"${PKGEXT}" + ! __isGlobfile "${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}"-*"${PKGEXT}" done } diff --git a/test/cases/sourceballs.bats b/test/cases/sourceballs.bats index a0a2999..df7ddd4 100644 --- a/test/cases/sourceballs.bats +++ b/test/cases/sourceballs.bats @@ -2,12 +2,12 @@ load ../lib/common __checkSourcePackage() { local pkgbase=$1 - [ -r ${FTP_BASE}/${SRCPOOL}/${pkgbase}-*${SRCEXT} ] + __isGlobfile "${FTP_BASE}/${SRCPOOL}/${pkgbase}"-*"${SRCEXT}" } __checkRemovedSourcePackage() { local pkgbase=$1 - [ ! -r ${FTP_BASE}/${SRCPOOL}/${pkgbase}-*${SRCEXT} ] + ! __isGlobfile "${FTP_BASE}/${SRCPOOL}/${pkgbase}"-*"${SRCEXT}" } @test "create simple package sourceballs" { diff --git a/test/lib/common.bash b/test/lib/common.bash index 5411641..6e2b3df 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -14,6 +14,16 @@ __getCheckSum() { echo "${result%% *}" } +# Proxy function to check if a file exists. Using [[ -f ... ]] directly is not +# always wanted because we might want to expand bash globs first. This way we +# can pass unquoted globs to __isGlobfile() and have them expanded as function +# arguments before being checked. +# +# This is a copy of db-functions is_globfile +__isGlobfile() { + [[ -f $1 ]] +} + __buildPackage() { local pkgdest=${1:-.} local p @@ -203,7 +213,7 @@ checkPackageDB() { for repoarch in ${repoarches[@]}; do # Only 'any' packages can be found in repos of both arches if [[ $pkgarch != any ]]; then - if [[ $pkgarch != ${repoarch} ]]; then + if [[ $pkgarch != "$repoarch" ]]; then continue fi fi -- 2.16.1
On 02/22/2018 09:15 PM, Luke Shumaker wrote:
- common.bash: Globbing happens on the RHS of a [[ = ]] test. This means that we must quote variables on the RHS that are to be taken verbatim. This is surprising, because we don't need to quote the LHS.
Unless we intend to do a general style cleanup, it is unnecessary to "fix" cases where variables with known static content aren't quoted. -- Eli Schwartz Bug Wrangler and Trusted User
From: Luke Shumaker <lukeshu@parabola.nu> These are things that were (IMO) missed in 5afac1e. I found them using: git grep -E '(plain|msg|msg2|warning|error|die) "[^"]*\$' I went a little above-and-beyond for escaping strings for the error messages in db-functions' arch_repo_add and arch_repo_remove. The code should explain itself, but I wanted to point it out, as it's more than the usual "slap %s in there, and move the ${...} to the right". v2: - arch_repo_add, arch_repo_remove: Use Bash 4.4 parameter transformation to avoid having to introduce temporary variables. --- cron-jobs/ftpdir-cleanup | 8 ++++---- cron-jobs/integrity-check | 2 +- cron-jobs/sourceballs | 8 ++++---- db-functions | 4 ++-- db-move | 4 ++-- db-remove | 2 +- db-repo-add | 2 +- db-repo-remove | 2 +- db-update | 2 +- testing2x | 2 +- 10 files changed, 18 insertions(+), 18 deletions(-) diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index ff65d46..e24e614 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -50,7 +50,7 @@ for repo in ${PKGREPOS[@]}; do if (( ${#missing_pkgs[@]} >= 1 )); then error "Missing packages in [%s] (%s)..." "$repo" "$arch" for missing_pkg in ${missing_pkgs[@]}; do - msg2 "${missing_pkg}" + msg2 '%s' "${missing_pkg}" done fi @@ -58,7 +58,7 @@ for repo in ${PKGREPOS[@]}; do if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old packages from [%s] (%s)..." "$repo" "$arch" for old_pkg in ${old_pkgs[@]}; do - msg2 "${old_pkg}" + msg2 '%s' "${old_pkg}" clean_pkg "${FTP_BASE}/${repo}/os/${arch}/${old_pkg}" done fi @@ -76,7 +76,7 @@ old_pkgs=($(comm -23 "${WORKDIR}/pool" "${WORKDIR}/db")) if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old packages from package pool..." for old_pkg in ${old_pkgs[@]}; do - msg2 "${old_pkg}" + msg2 '%s' "${old_pkg}" clean_pkg "$FTP_BASE/${PKGPOOL}/${old_pkg}" done fi @@ -91,7 +91,7 @@ done if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do - msg2 "${old_pkg}" + msg2 '%s' "${old_pkg}" if ! ${CLEANUP_DRYRUN}; then rm -f "${CLEANUP_DESTDIR}/${old_pkg}" rm -f "${CLEANUP_DESTDIR}/${old_pkg}.sig" diff --git a/cron-jobs/integrity-check b/cron-jobs/integrity-check index f1e75ab..47cf856 100755 --- a/cron-jobs/integrity-check +++ b/cron-jobs/integrity-check @@ -8,7 +8,7 @@ dirname="$(dirname $0)" script_lock if (( $# != 1 )); then - die "usage: ${0##*/} <mailto>" + die "usage: %s <mailto>" "${0##*/}" fi mailto=$1 diff --git a/cron-jobs/sourceballs b/cron-jobs/sourceballs index 8f089b3..7370594 100755 --- a/cron-jobs/sourceballs +++ b/cron-jobs/sourceballs @@ -109,13 +109,13 @@ for repo in ${PKGREPOS[@]}; do if [ ${#newpkgs[@]} -ge 1 ]; then msg "Adding source packages for [%s]..." "$repo" for new_pkg in ${newpkgs[@]}; do - msg2 "${new_pkg}" + msg2 '%s' "${new_pkg}" done fi if [ ${#failedpkgs[@]} -ge 1 ]; then msg "Failed to create source packages for [%s]..." "$repo" for failed_pkg in ${failedpkgs[@]}; do - msg2 "${failed_pkg}" + msg2 '%s' "${failed_pkg}" done fi done @@ -129,7 +129,7 @@ if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old source packages..." ${SOURCE_CLEANUP_DRYRUN} && warning 'dry run mode is active' for old_pkg in ${old_pkgs[@]}; do - msg2 "${old_pkg}" + msg2 '%s' "${old_pkg}" if ! ${SOURCE_CLEANUP_DRYRUN}; then mv_acl "$FTP_BASE/${SRCPOOL}/${old_pkg}" "${SOURCE_CLEANUP_DESTDIR}/${old_pkg}" touch "${SOURCE_CLEANUP_DESTDIR}/${old_pkg}" @@ -148,7 +148,7 @@ done if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old source packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do - msg2 "${old_pkg}" + msg2 '%s' "${old_pkg}" ${SOURCE_CLEANUP_DRYRUN} || rm -f "${SOURCE_CLEANUP_DESTDIR}/${old_pkg}" done fi diff --git a/db-functions b/db-functions index 8b71cae..fb45513 100644 --- a/db-functions +++ b/db-functions @@ -450,7 +450,7 @@ arch_repo_add() { # package files might be relative to repo dir pushd "${FTP_BASE}/${repo}/os/${arch}" >/dev/null /usr/bin/repo-add -q "${repo}${DBEXT}" ${pkgs[@]} \ - || error "repo-add ${repo}${DBEXT} ${pkgs[@]}" + || error 'repo-add %q %s' "${repo}${DBEXT}" "${pkgs[*]@Q}" popd >/dev/null set_repo_permission "${repo}" "${arch}" @@ -468,7 +468,7 @@ arch_repo_remove() { return 1 fi /usr/bin/repo-remove -q "${dbfile}" ${pkgs[@]} \ - || error "repo-remove ${dbfile} ${pkgs[@]}" + || error 'repo-remove %q %s' "$dbfile" "${pkgs[*]@Q}" set_repo_permission "${repo}" "${arch}" REPO_MODIFIED=1 diff --git a/db-move b/db-move index fb7ebac..806ad9a 100755 --- a/db-move +++ b/db-move @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions" if (( $# < 3 )); then - msg "usage: ${0##*/} <repo-from> <repo-to> <pkgname|pkgbase> ..." + msg "usage: %s <repo-from> <repo-to> <pkgname|pkgbase> ..." "${0##*/}" exit 1 fi @@ -74,7 +74,7 @@ for pkgbase in ${args[@]:2}; do else tarches=("${pkgarch}") fi - msg2 "${pkgbase} ($(echo ${tarches[@]}))" + msg2 "%s (%s)" "$pkgbase" "${tarches[*]}" pkgnames=($(. "${svnrepo_from}/PKGBUILD"; echo ${pkgname[@]})) pkgver=$(. "${svnrepo_from}/PKGBUILD"; get_full_version) diff --git a/db-remove b/db-remove index 70502bc..e7aed31 100755 --- a/db-remove +++ b/db-remove @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions" if (( $# < 3 )); then - msg "usage: ${0##*/} <repo> <arch> <pkgname|pkgbase> ..." + msg "usage: %s <repo> <arch> <pkgname|pkgbase> ..." "${0##*/}" exit 1 fi diff --git a/db-repo-add b/db-repo-add index d7be302..4ea758f 100755 --- a/db-repo-add +++ b/db-repo-add @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions" if (( $# < 3 )); then - msg "usage: ${0##*/} <repo> <arch> <pkgfile> ..." + msg "usage: %s <repo> <arch> <pkgfile> ..." "${0##*/}" exit 1 fi diff --git a/db-repo-remove b/db-repo-remove index 32d167e..2f24edd 100755 --- a/db-repo-remove +++ b/db-repo-remove @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions" if (( $# < 3 )); then - msg "usage: ${0##*/} <repo> <arch> <pkgname> ..." + msg "usage: %s <repo> <arch> <pkgname> ..." "${0##*/}" exit 1 fi diff --git a/db-update b/db-update index 4e17184..5077389 100755 --- a/db-update +++ b/db-update @@ -78,7 +78,7 @@ for repo in ${repos[@]}; do arch_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-${pkgarch}${PKGEXTS} 2>/dev/null)) for pkg in ${arch_pkgs[@]} ${any_pkgs[@]}; do pkgfile="${pkg##*/}" - msg2 "${pkgfile} (${pkgarch})" + msg2 '%s (%s)' "$pkgfile" "$pkgarch" # any packages might have been moved by the previous run if [[ -f ${pkg} ]]; then mv "${pkg}" "$FTP_BASE/${PKGPOOL}" diff --git a/testing2x b/testing2x index f0d77cb..d68e405 100755 --- a/testing2x +++ b/testing2x @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions" if (( $# < 1 )); then - msg "usage: ${0##*/} <pkgname|pkgbase> ..." + msg "usage: %s <pkgname|pkgbase> ..." "${0##*/}" exit 1 fi -- 2.16.1
On 02/22/2018 09:15 PM, Luke Shumaker wrote:
From: Luke Shumaker <lukeshu@parabola.nu>
These are things that were (IMO) missed in 5afac1e. I found them using:
git grep -E '(plain|msg|msg2|warning|error|die) "[^"]*\$'
I went a little above-and-beyond for escaping strings for the error messages in db-functions' arch_repo_add and arch_repo_remove. The code should explain itself, but I wanted to point it out, as it's more than the usual "slap %s in there, and move the ${...} to the right".
v2: - arch_repo_add, arch_repo_remove: Use Bash 4.4 parameter transformation to avoid having to introduce temporary variables.
--- a/db-functions +++ b/db-functions @@ -450,7 +450,7 @@ arch_repo_add() { # package files might be relative to repo dir pushd "${FTP_BASE}/${repo}/os/${arch}" >/dev/null /usr/bin/repo-add -q "${repo}${DBEXT}" ${pkgs[@]} \ - || error "repo-add ${repo}${DBEXT} ${pkgs[@]}" + || error 'repo-add %q %s' "${repo}${DBEXT}" "${pkgs[*]@Q}" popd >/dev/null set_repo_permission "${repo}" "${arch}"
@@ -468,7 +468,7 @@ arch_repo_remove() { return 1 fi /usr/bin/repo-remove -q "${dbfile}" ${pkgs[@]} \ - || error "repo-remove ${dbfile} ${pkgs[@]}" + || error 'repo-remove %q %s' "$dbfile" "${pkgs[*]@Q}" set_repo_permission "${repo}" "${arch}"
I think for consistency we should use the same style which means using "${dbfile@Q}" Currently we'll get the dbfile backslash-escaped, followed by the package names unequivocally quoted on the same line of error output. It makes more sense to ensure that they are all quoted. While in theory one could bikeshed over which is the better way to consistently print them, there is only one option for us since we're trying to interject all the ${pkgs[*]} as one argument...
REPO_MODIFIED=1 diff --git a/db-move b/db-move index fb7ebac..806ad9a 100755 --- a/db-move +++ b/db-move @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions"
if (( $# < 3 )); then - msg "usage: ${0##*/} <repo-from> <repo-to> <pkgname|pkgbase> ..." + msg "usage: %s <repo-from> <repo-to> <pkgname|pkgbase> ..." "${0##*/}" exit 1 fi
@@ -74,7 +74,7 @@ for pkgbase in ${args[@]:2}; do else tarches=("${pkgarch}") fi - msg2 "${pkgbase} ($(echo ${tarches[@]}))" + msg2 "%s (%s)" "$pkgbase" "${tarches[*]}" pkgnames=($(. "${svnrepo_from}/PKGBUILD"; echo ${pkgname[@]})) pkgver=$(. "${svnrepo_from}/PKGBUILD"; get_full_version)
diff --git a/db-remove b/db-remove index 70502bc..e7aed31 100755 --- a/db-remove +++ b/db-remove @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions"
if (( $# < 3 )); then - msg "usage: ${0##*/} <repo> <arch> <pkgname|pkgbase> ..." + msg "usage: %s <repo> <arch> <pkgname|pkgbase> ..." "${0##*/}" exit 1 fi
diff --git a/db-repo-add b/db-repo-add index d7be302..4ea758f 100755 --- a/db-repo-add +++ b/db-repo-add @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions"
if (( $# < 3 )); then - msg "usage: ${0##*/} <repo> <arch> <pkgfile> ..." + msg "usage: %s <repo> <arch> <pkgfile> ..." "${0##*/}" exit 1 fi
diff --git a/db-repo-remove b/db-repo-remove index 32d167e..2f24edd 100755 --- a/db-repo-remove +++ b/db-repo-remove @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions"
if (( $# < 3 )); then - msg "usage: ${0##*/} <repo> <arch> <pkgname> ..." + msg "usage: %s <repo> <arch> <pkgname> ..." "${0##*/}" exit 1 fi
diff --git a/db-update b/db-update index 4e17184..5077389 100755 --- a/db-update +++ b/db-update @@ -78,7 +78,7 @@ for repo in ${repos[@]}; do arch_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-${pkgarch}${PKGEXTS} 2>/dev/null)) for pkg in ${arch_pkgs[@]} ${any_pkgs[@]}; do pkgfile="${pkg##*/}" - msg2 "${pkgfile} (${pkgarch})" + msg2 '%s (%s)' "$pkgfile" "$pkgarch" # any packages might have been moved by the previous run if [[ -f ${pkg} ]]; then mv "${pkg}" "$FTP_BASE/${PKGPOOL}" diff --git a/testing2x b/testing2x index f0d77cb..d68e405 100755 --- a/testing2x +++ b/testing2x @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions"
if (( $# < 1 )); then - msg "usage: ${0##*/} <pkgname|pkgbase> ..." + msg "usage: %s <pkgname|pkgbase> ..." "${0##*/}" exit 1 fi
-- Eli Schwartz Bug Wrangler and Trusted User
On Mon, 26 Feb 2018 00:46:48 -0500, Eli Schwartz wrote:
+++ b/db-functions @@ -450,7 +450,7 @@ arch_repo_add() { # package files might be relative to repo dir pushd "${FTP_BASE}/${repo}/os/${arch}" >/dev/null /usr/bin/repo-add -q "${repo}${DBEXT}" ${pkgs[@]} \ - || error "repo-add ${repo}${DBEXT} ${pkgs[@]}" + || error 'repo-add %q %s' "${repo}${DBEXT}" "${pkgs[*]@Q}" popd >/dev/null set_repo_permission "${repo}" "${arch}"
@@ -468,7 +468,7 @@ arch_repo_remove() { return 1 fi /usr/bin/repo-remove -q "${dbfile}" ${pkgs[@]} \ - || error "repo-remove ${dbfile} ${pkgs[@]}" + || error 'repo-remove %q %s' "$dbfile" "${pkgs[*]@Q}" set_repo_permission "${repo}" "${arch}"
I think for consistency we should use the same style which means using "${dbfile@Q}"
I was going for consistency with the repo-add version, which doesn't have a single dbfile variable to @Q. Would you have me introduce a dbfile variable in arch_repo_add? -- Happy hacking, ~ Luke Shumaker
On 02/26/2018 05:14 PM, Luke Shumaker wrote:
On Mon, 26 Feb 2018 00:46:48 -0500, Eli Schwartz wrote:
+++ b/db-functions @@ -450,7 +450,7 @@ arch_repo_add() { # package files might be relative to repo dir pushd "${FTP_BASE}/${repo}/os/${arch}" >/dev/null /usr/bin/repo-add -q "${repo}${DBEXT}" ${pkgs[@]} \ - || error "repo-add ${repo}${DBEXT} ${pkgs[@]}" + || error 'repo-add %q %s' "${repo}${DBEXT}" "${pkgs[*]@Q}" popd >/dev/null set_repo_permission "${repo}" "${arch}"
@@ -468,7 +468,7 @@ arch_repo_remove() { return 1 fi /usr/bin/repo-remove -q "${dbfile}" ${pkgs[@]} \ - || error "repo-remove ${dbfile} ${pkgs[@]}" + || error 'repo-remove %q %s' "$dbfile" "${pkgs[*]@Q}" set_repo_permission "${repo}" "${arch}"
I think for consistency we should use the same style which means using "${dbfile@Q}"
I was going for consistency with the repo-add version, which doesn't have a single dbfile variable to @Q. Would you have me introduce a dbfile variable in arch_repo_add?
Well, we basically use it hardcoded three times, so why not. :p Just like the repo-add version, except with pushd "${dbfile%/*}" Actually, this decreases the difference between the two enough that we may want to just have one call the other... or do this: https://lists.archlinux.org/pipermail/arch-projects/2018-February/004832.htm... -- Eli Schwartz Bug Wrangler and Trusted User
participants (2)
-
Eli Schwartz
-
Luke Shumaker