[arch-projects] [dbscripts] [PATCH 0/3] Misc touchup
From: Luke Shumaker <lukeshu@parabola.nu> These are a few minor unrelated touch-ups that I noticed when doing the PKGEXT work. There's no real theme to them. Luke Shumaker (3): test: common.bash:__getCheckSum: Don't rely on IFS test: Fixup glob matching Update messages to make fuller use of printf formatters cron-jobs/ftpdir-cleanup | 8 ++++---- cron-jobs/integrity-check | 2 +- cron-jobs/sourceballs | 8 ++++---- db-functions | 8 ++++++-- db-move | 4 ++-- db-remove | 2 +- db-repo-add | 2 +- db-repo-remove | 2 +- db-update | 2 +- test/cases/ftpdir-cleanup.bats | 4 ++-- test/cases/sourceballs.bats | 4 ++-- test/lib/common.bash | 17 ++++++++++++++--- testing2x | 2 +- 13 files changed, 40 insertions(+), 25 deletions(-) -- Happy hacking, ~ Luke Shumaker
From: Luke Shumaker <lukeshu@parabola.nu> I managed to stumble across a bug in BATS where the run() function screwed with the global IFS. The bug has been fixed in git, but isn't in a release yet. https://github.com/sstephenson/bats/issues/89 Anyway, this bug breaks __getCheckSum(). Fortunately, we have avoided tripping it so far because luck has it that we never call __getCheckSum() after run() in the same test. So, there's nothing actually broken here, but it makes me nervous. So go ahead and modify __getCheckSum to not rely on IFS. --- test/lib/common.bash | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/test/lib/common.bash b/test/lib/common.bash index 568a541..5411641 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -9,8 +9,9 @@ __updatePKGBUILD() { } __getCheckSum() { - local result=($(sha1sum $1)) - echo ${result[0]} + local result + result="$(sha1sum "$1")" + echo "${result%% *}" } __buildPackage() { -- 2.16.1
On 02/22/2018 03:43 PM, Luke Shumaker wrote:
From: Luke Shumaker <lukeshu@parabola.nu>
I managed to stumble across a bug in BATS where the run() function screwed with the global IFS. The bug has been fixed in git, but isn't in a release yet.
https://github.com/sstephenson/bats/issues/89
Anyway, this bug breaks __getCheckSum(). Fortunately, we have avoided tripping it so far because luck has it that we never call __getCheckSum() after run() in the same test.
So, there's nothing actually broken here, but it makes me nervous. So go ahead and modify __getCheckSum to not rely on IFS. --- test/lib/common.bash | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/test/lib/common.bash b/test/lib/common.bash index 568a541..5411641 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -9,8 +9,9 @@ __updatePKGBUILD() { }
__getCheckSum() { - local result=($(sha1sum $1)) - echo ${result[0]} + local result + result="$(sha1sum "$1")" + echo "${result%% *}"
Why are you moving over to declaring the variable and assigning it on different lines? -- Eli Schwartz Bug Wrangler and Trusted User
On Thu, 22 Feb 2018 16:43:36 -0500, Eli Schwartz wrote:
__getCheckSum() { - local result=($(sha1sum $1)) - echo ${result[0]} + local result + result="$(sha1sum "$1")" + echo "${result%% *}"
Why are you moving over to declaring the variable and assigning it on different lines?
Because shellcheck complains about it, so it's a habit I've gotten in to :) Even in cases where it doesn't really make a difference. https://github.com/koalaman/shellcheck/wiki/SC2155 However, BATS does run the test suite with `set -e`, so splitting it does mean that BATS will now detect errors from sha1sum. We don't really expect that to happen, but if BATS will give us error checking on it for free, why not? -- Happy hacking, ~ Luke Shumaker
On 02/22/2018 06:43 PM, Luke Shumaker wrote:
On Thu, 22 Feb 2018 16:43:36 -0500, Eli Schwartz wrote:
__getCheckSum() { - local result=($(sha1sum $1)) - echo ${result[0]} + local result + result="$(sha1sum "$1")" + echo "${result%% *}"
Why are you moving over to declaring the variable and assigning it on different lines?
Because shellcheck complains about it, so it's a habit I've gotten in to :) Even in cases where it doesn't really make a difference.
https://github.com/koalaman/shellcheck/wiki/SC2155
However, BATS does run the test suite with `set -e`, so splitting it does mean that BATS will now detect errors from sha1sum. We don't really expect that to happen, but if BATS will give us error checking on it for free, why not?
Then the commit message should say so... -- Eli Schwartz Bug Wrangler and Trusted User
On Thu, 22 Feb 2018 19:27:04 -0500, Eli Schwartz wrote:
[1 Re: [arch-projects] [dbscripts] [PATCH 1/3] test: common.bash:__getCheckSum: Don't rely on IFS <multipart/mixed (7bit)>] [1.1 <text/plain; utf-8 (quoted-printable)>] On 02/22/2018 06:43 PM, Luke Shumaker wrote:
On Thu, 22 Feb 2018 16:43:36 -0500, Eli Schwartz wrote:
__getCheckSum() { - local result=($(sha1sum $1)) - echo ${result[0]} + local result + result="$(sha1sum "$1")" + echo "${result%% *}"
Why are you moving over to declaring the variable and assigning it on different lines?
Because shellcheck complains about it, so it's a habit I've gotten in to :) Even in cases where it doesn't really make a difference.
https://github.com/koalaman/shellcheck/wiki/SC2155
However, BATS does run the test suite with `set -e`, so splitting it does mean that BATS will now detect errors from sha1sum. We don't really expect that to happen, but if BATS will give us error checking on it for free, why not?
Then the commit message should say so...
Honestly, I didn't even think about it. Like I said, I've just made it a habit. -- Happy hacking, ~ Luke Shumaker
From: Luke Shumaker <lukeshu@parabola.nu> - ftpdir-cleanup.bats: Glob expansion does not occur in [[ -f ]] tests. The [[ ! -f .../${pkgname}-*${PKGEXT} ]] checks were checking that there were no files containing a literal '*' for that part of their name. Obviously, this isn't what was intended. - sourceballs.bats: [ -r ] checks explode if the glob returns >1 file. Avoid using them if the path being checked contains a glob. - common.bash: Globbing happens on the RHS of a [[ = ]] test. This means that we must quote variables on the RHS that are to be taken verbatim. This is surprising, because we don't need to quote the LHS. --- test/cases/ftpdir-cleanup.bats | 4 ++-- test/cases/sourceballs.bats | 4 ++-- test/lib/common.bash | 12 +++++++++++- 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/test/cases/ftpdir-cleanup.bats b/test/cases/ftpdir-cleanup.bats index fd485f3..8c713c6 100644 --- a/test/cases/ftpdir-cleanup.bats +++ b/test/cases/ftpdir-cleanup.bats @@ -13,8 +13,8 @@ __checkRepoRemovedPackage() { local pkgname for pkgname in $(__getPackageNamesFromPackageBase ${pkgbase}); do - [[ ! -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-*${PKGEXT} ]] - [[ ! -f ${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}-*${PKGEXT} ]] + ! __isGlobfile "${FTP_BASE}/${PKGPOOL}/${pkgname}"-*"${PKGEXT}" + ! __isGlobfile "${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}"-*"${PKGEXT}" done } diff --git a/test/cases/sourceballs.bats b/test/cases/sourceballs.bats index a0a2999..df7ddd4 100644 --- a/test/cases/sourceballs.bats +++ b/test/cases/sourceballs.bats @@ -2,12 +2,12 @@ load ../lib/common __checkSourcePackage() { local pkgbase=$1 - [ -r ${FTP_BASE}/${SRCPOOL}/${pkgbase}-*${SRCEXT} ] + __isGlobfile "${FTP_BASE}/${SRCPOOL}/${pkgbase}"-*"${SRCEXT}" } __checkRemovedSourcePackage() { local pkgbase=$1 - [ ! -r ${FTP_BASE}/${SRCPOOL}/${pkgbase}-*${SRCEXT} ] + ! __isGlobfile "${FTP_BASE}/${SRCPOOL}/${pkgbase}"-*"${SRCEXT}" } @test "create simple package sourceballs" { diff --git a/test/lib/common.bash b/test/lib/common.bash index 5411641..6e2b3df 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -14,6 +14,16 @@ __getCheckSum() { echo "${result%% *}" } +# Proxy function to check if a file exists. Using [[ -f ... ]] directly is not +# always wanted because we might want to expand bash globs first. This way we +# can pass unquoted globs to __isGlobfile() and have them expanded as function +# arguments before being checked. +# +# This is a copy of db-functions is_globfile +__isGlobfile() { + [[ -f $1 ]] +} + __buildPackage() { local pkgdest=${1:-.} local p @@ -203,7 +213,7 @@ checkPackageDB() { for repoarch in ${repoarches[@]}; do # Only 'any' packages can be found in repos of both arches if [[ $pkgarch != any ]]; then - if [[ $pkgarch != ${repoarch} ]]; then + if [[ $pkgarch != "$repoarch" ]]; then continue fi fi -- 2.16.1
On 02/22/2018 03:43 PM, Luke Shumaker wrote:
From: Luke Shumaker <lukeshu@parabola.nu>
- ftpdir-cleanup.bats: Glob expansion does not occur in [[ -f ]] tests. The [[ ! -f .../${pkgname}-*${PKGEXT} ]] checks were checking that there were no files containing a literal '*' for that part of their name. Obviously, this isn't what was intended.
Good catch, thanks!
- sourceballs.bats: [ -r ] checks explode if the glob returns >1 file. Avoid using them if the path being checked contains a glob.
Arguably I'd like to upgrade the whole testsuite to use [[ ... ]] Thanks.
- common.bash: Globbing happens on the RHS of a [[ = ]] test. This means that we must quote variables on the RHS that are to be taken verbatim. This is surprising, because we don't need to quote the LHS.
No we don't, we only "need" to quote the ones that (can) contain globs. In the general case, there is a school of thought that says you should only quote what you explicitly need to quote. :p Also not really surprising. This is a bash feature, you can do regex comparisons too.
--- test/cases/ftpdir-cleanup.bats | 4 ++-- test/cases/sourceballs.bats | 4 ++-- test/lib/common.bash | 12 +++++++++++- 3 files changed, 15 insertions(+), 5 deletions(-)
diff --git a/test/cases/ftpdir-cleanup.bats b/test/cases/ftpdir-cleanup.bats index fd485f3..8c713c6 100644 --- a/test/cases/ftpdir-cleanup.bats +++ b/test/cases/ftpdir-cleanup.bats @@ -13,8 +13,8 @@ __checkRepoRemovedPackage() { local pkgname
for pkgname in $(__getPackageNamesFromPackageBase ${pkgbase}); do - [[ ! -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-*${PKGEXT} ]] - [[ ! -f ${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}-*${PKGEXT} ]] + ! __isGlobfile "${FTP_BASE}/${PKGPOOL}/${pkgname}"-*"${PKGEXT}" + ! __isGlobfile "${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}"-*"${PKGEXT}" done }
diff --git a/test/cases/sourceballs.bats b/test/cases/sourceballs.bats index a0a2999..df7ddd4 100644 --- a/test/cases/sourceballs.bats +++ b/test/cases/sourceballs.bats @@ -2,12 +2,12 @@ load ../lib/common
__checkSourcePackage() { local pkgbase=$1 - [ -r ${FTP_BASE}/${SRCPOOL}/${pkgbase}-*${SRCEXT} ] + __isGlobfile "${FTP_BASE}/${SRCPOOL}/${pkgbase}"-*"${SRCEXT}" }
__checkRemovedSourcePackage() { local pkgbase=$1 - [ ! -r ${FTP_BASE}/${SRCPOOL}/${pkgbase}-*${SRCEXT} ] + ! __isGlobfile "${FTP_BASE}/${SRCPOOL}/${pkgbase}"-*"${SRCEXT}" }
@test "create simple package sourceballs" { diff --git a/test/lib/common.bash b/test/lib/common.bash index 5411641..6e2b3df 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -14,6 +14,16 @@ __getCheckSum() { echo "${result%% *}" }
+# Proxy function to check if a file exists. Using [[ -f ... ]] directly is not +# always wanted because we might want to expand bash globs first. This way we +# can pass unquoted globs to __isGlobfile() and have them expanded as function +# arguments before being checked. +# +# This is a copy of db-functions is_globfile +__isGlobfile() { + [[ -f $1 ]] +} + __buildPackage() { local pkgdest=${1:-.} local p @@ -203,7 +213,7 @@ checkPackageDB() { for repoarch in ${repoarches[@]}; do # Only 'any' packages can be found in repos of both arches if [[ $pkgarch != any ]]; then - if [[ $pkgarch != ${repoarch} ]]; then + if [[ $pkgarch != "$repoarch" ]]; then continue fi fi
-- Eli Schwartz Bug Wrangler and Trusted User
From: Luke Shumaker <lukeshu@parabola.nu> These are things that were (IMO) missed in 5afac1e. I found them using: git ls-files|xargs grep -E '(plain|msg|msg2|warning|error|die) "[^"]*\$' I went a little above-and-beyond for escaping strings for the error messages in db-functions' arch_repo_add and arch_repo_remove. The code should explain itself, but I wanted to point it out, as it's more complex than the "slap %s in there, and move the ${...} to the right" that is used everywhere else. --- cron-jobs/ftpdir-cleanup | 8 ++++---- cron-jobs/integrity-check | 2 +- cron-jobs/sourceballs | 8 ++++---- db-functions | 8 ++++++-- db-move | 4 ++-- db-remove | 2 +- db-repo-add | 2 +- db-repo-remove | 2 +- db-update | 2 +- testing2x | 2 +- 10 files changed, 22 insertions(+), 18 deletions(-) diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index ff65d46..e24e614 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -50,7 +50,7 @@ for repo in ${PKGREPOS[@]}; do if (( ${#missing_pkgs[@]} >= 1 )); then error "Missing packages in [%s] (%s)..." "$repo" "$arch" for missing_pkg in ${missing_pkgs[@]}; do - msg2 "${missing_pkg}" + msg2 '%s' "${missing_pkg}" done fi @@ -58,7 +58,7 @@ for repo in ${PKGREPOS[@]}; do if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old packages from [%s] (%s)..." "$repo" "$arch" for old_pkg in ${old_pkgs[@]}; do - msg2 "${old_pkg}" + msg2 '%s' "${old_pkg}" clean_pkg "${FTP_BASE}/${repo}/os/${arch}/${old_pkg}" done fi @@ -76,7 +76,7 @@ old_pkgs=($(comm -23 "${WORKDIR}/pool" "${WORKDIR}/db")) if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old packages from package pool..." for old_pkg in ${old_pkgs[@]}; do - msg2 "${old_pkg}" + msg2 '%s' "${old_pkg}" clean_pkg "$FTP_BASE/${PKGPOOL}/${old_pkg}" done fi @@ -91,7 +91,7 @@ done if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do - msg2 "${old_pkg}" + msg2 '%s' "${old_pkg}" if ! ${CLEANUP_DRYRUN}; then rm -f "${CLEANUP_DESTDIR}/${old_pkg}" rm -f "${CLEANUP_DESTDIR}/${old_pkg}.sig" diff --git a/cron-jobs/integrity-check b/cron-jobs/integrity-check index f1e75ab..47cf856 100755 --- a/cron-jobs/integrity-check +++ b/cron-jobs/integrity-check @@ -8,7 +8,7 @@ dirname="$(dirname $0)" script_lock if (( $# != 1 )); then - die "usage: ${0##*/} <mailto>" + die "usage: %s <mailto>" "${0##*/}" fi mailto=$1 diff --git a/cron-jobs/sourceballs b/cron-jobs/sourceballs index 8f089b3..7370594 100755 --- a/cron-jobs/sourceballs +++ b/cron-jobs/sourceballs @@ -109,13 +109,13 @@ for repo in ${PKGREPOS[@]}; do if [ ${#newpkgs[@]} -ge 1 ]; then msg "Adding source packages for [%s]..." "$repo" for new_pkg in ${newpkgs[@]}; do - msg2 "${new_pkg}" + msg2 '%s' "${new_pkg}" done fi if [ ${#failedpkgs[@]} -ge 1 ]; then msg "Failed to create source packages for [%s]..." "$repo" for failed_pkg in ${failedpkgs[@]}; do - msg2 "${failed_pkg}" + msg2 '%s' "${failed_pkg}" done fi done @@ -129,7 +129,7 @@ if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old source packages..." ${SOURCE_CLEANUP_DRYRUN} && warning 'dry run mode is active' for old_pkg in ${old_pkgs[@]}; do - msg2 "${old_pkg}" + msg2 '%s' "${old_pkg}" if ! ${SOURCE_CLEANUP_DRYRUN}; then mv_acl "$FTP_BASE/${SRCPOOL}/${old_pkg}" "${SOURCE_CLEANUP_DESTDIR}/${old_pkg}" touch "${SOURCE_CLEANUP_DESTDIR}/${old_pkg}" @@ -148,7 +148,7 @@ done if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old source packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do - msg2 "${old_pkg}" + msg2 '%s' "${old_pkg}" ${SOURCE_CLEANUP_DRYRUN} || rm -f "${SOURCE_CLEANUP_DESTDIR}/${old_pkg}" done fi diff --git a/db-functions b/db-functions index 8b71cae..6d02c50 100644 --- a/db-functions +++ b/db-functions @@ -446,11 +446,13 @@ arch_repo_add() { local repo=$1 local arch=$2 local pkgs=(${@:3}) + local pkgs_str # package files might be relative to repo dir pushd "${FTP_BASE}/${repo}/os/${arch}" >/dev/null + printf -v pkgs_str -- '%q ' "${pkgs[@]}" /usr/bin/repo-add -q "${repo}${DBEXT}" ${pkgs[@]} \ - || error "repo-add ${repo}${DBEXT} ${pkgs[@]}" + || error 'repo-add %q %s' "${repo}${DBEXT}" "${pkgs_str% }" popd >/dev/null set_repo_permission "${repo}" "${arch}" @@ -462,13 +464,15 @@ arch_repo_remove() { local arch=$2 local pkgs=(${@:3}) local dbfile="${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" + local pkgs_str if [[ ! -f ${dbfile} ]]; then error "No database found at '%s'" "$dbfile" return 1 fi + printf -v pkgs_str -- '%q ' "${pkgs[@]}" /usr/bin/repo-remove -q "${dbfile}" ${pkgs[@]} \ - || error "repo-remove ${dbfile} ${pkgs[@]}" + || error 'repo-remove %q %s' "$dbfile" "${pkgs_str% }" set_repo_permission "${repo}" "${arch}" REPO_MODIFIED=1 diff --git a/db-move b/db-move index fb7ebac..806ad9a 100755 --- a/db-move +++ b/db-move @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions" if (( $# < 3 )); then - msg "usage: ${0##*/} <repo-from> <repo-to> <pkgname|pkgbase> ..." + msg "usage: %s <repo-from> <repo-to> <pkgname|pkgbase> ..." "${0##*/}" exit 1 fi @@ -74,7 +74,7 @@ for pkgbase in ${args[@]:2}; do else tarches=("${pkgarch}") fi - msg2 "${pkgbase} ($(echo ${tarches[@]}))" + msg2 "%s (%s)" "$pkgbase" "${tarches[*]}" pkgnames=($(. "${svnrepo_from}/PKGBUILD"; echo ${pkgname[@]})) pkgver=$(. "${svnrepo_from}/PKGBUILD"; get_full_version) diff --git a/db-remove b/db-remove index 70502bc..e7aed31 100755 --- a/db-remove +++ b/db-remove @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions" if (( $# < 3 )); then - msg "usage: ${0##*/} <repo> <arch> <pkgname|pkgbase> ..." + msg "usage: %s <repo> <arch> <pkgname|pkgbase> ..." "${0##*/}" exit 1 fi diff --git a/db-repo-add b/db-repo-add index d7be302..4ea758f 100755 --- a/db-repo-add +++ b/db-repo-add @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions" if (( $# < 3 )); then - msg "usage: ${0##*/} <repo> <arch> <pkgfile> ..." + msg "usage: %s <repo> <arch> <pkgfile> ..." "${0##*/}" exit 1 fi diff --git a/db-repo-remove b/db-repo-remove index 32d167e..2f24edd 100755 --- a/db-repo-remove +++ b/db-repo-remove @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions" if (( $# < 3 )); then - msg "usage: ${0##*/} <repo> <arch> <pkgname> ..." + msg "usage: %s <repo> <arch> <pkgname> ..." "${0##*/}" exit 1 fi diff --git a/db-update b/db-update index 4e17184..5077389 100755 --- a/db-update +++ b/db-update @@ -78,7 +78,7 @@ for repo in ${repos[@]}; do arch_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-${pkgarch}${PKGEXTS} 2>/dev/null)) for pkg in ${arch_pkgs[@]} ${any_pkgs[@]}; do pkgfile="${pkg##*/}" - msg2 "${pkgfile} (${pkgarch})" + msg2 '%s (%s)' "$pkgfile" "$pkgarch" # any packages might have been moved by the previous run if [[ -f ${pkg} ]]; then mv "${pkg}" "$FTP_BASE/${PKGPOOL}" diff --git a/testing2x b/testing2x index f0d77cb..d68e405 100755 --- a/testing2x +++ b/testing2x @@ -4,7 +4,7 @@ . "$(dirname $0)/db-functions" if (( $# < 1 )); then - msg "usage: ${0##*/} <pkgname|pkgbase> ..." + msg "usage: %s <pkgname|pkgbase> ..." "${0##*/}" exit 1 fi -- 2.16.1
On 02/22/2018 03:43 PM, Luke Shumaker wrote:
From: Luke Shumaker <lukeshu@parabola.nu>
These are things that were (IMO) missed in 5afac1e. I found them using:
git ls-files|xargs grep -E '(plain|msg|msg2|warning|error|die) "[^"]*\$'
Consider using git grep next time :p rather than piping through both xargs and the inferior GNU grep.
I went a little above-and-beyond for escaping strings for the error messages in db-functions' arch_repo_add and arch_repo_remove. The code should explain itself, but I wanted to point it out, as it's more complex than the "slap %s in there, and move the ${...} to the right" that is used everywhere else.
[...] index 8b71cae..6d02c50 100644 --- a/db-functions +++ b/db-functions @@ -446,11 +446,13 @@ arch_repo_add() { local repo=$1 local arch=$2 local pkgs=(${@:3}) + local pkgs_str
# package files might be relative to repo dir pushd "${FTP_BASE}/${repo}/os/${arch}" >/dev/null + printf -v pkgs_str -- '%q ' "${pkgs[@]}" /usr/bin/repo-add -q "${repo}${DBEXT}" ${pkgs[@]} \ - || error "repo-add ${repo}${DBEXT} ${pkgs[@]}" + || error 'repo-add %q %s' "${repo}${DBEXT}" "${pkgs_str% }" popd >/dev/null set_repo_permission "${repo}" "${arch}"
@@ -462,13 +464,15 @@ arch_repo_remove() { local arch=$2 local pkgs=(${@:3}) local dbfile="${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" + local pkgs_str
if [[ ! -f ${dbfile} ]]; then error "No database found at '%s'" "$dbfile" return 1 fi + printf -v pkgs_str -- '%q ' "${pkgs[@]}" /usr/bin/repo-remove -q "${dbfile}" ${pkgs[@]} \ - || error "repo-remove ${dbfile} ${pkgs[@]}" + || error 'repo-remove %q %s' "$dbfile" "${pkgs_str% }" set_repo_permission "${repo}" "${arch}"
I do see what you're doing, I'm just not sure why. Is the whole idea with this extra variable floating around, to avoid tokenizing "${pkgs[@]}" as separate messages? That's why "${pkgs[*]}" tokenizes the members of an array as one word by gluing the members together using the first IFS character (which is a space). You'll note I used this in testing2x. As for using %q for filepaths that can theoretically contain spaces... good point I guess. -- Eli Schwartz Bug Wrangler and Trusted User
On Thu, 22 Feb 2018 16:44:20 -0500, Eli Schwartz wrote:
I went a little above-and-beyond for escaping strings for the error messages in db-functions' arch_repo_add and arch_repo_remove. The code should explain itself, but I wanted to point it out, as it's more complex than the "slap %s in there, and move the ${...} to the right" that is used everywhere else.
[...] index 8b71cae..6d02c50 100644 --- a/db-functions +++ b/db-functions @@ -446,11 +446,13 @@ arch_repo_add() { local repo=$1 local arch=$2 local pkgs=(${@:3}) + local pkgs_str
# package files might be relative to repo dir pushd "${FTP_BASE}/${repo}/os/${arch}" >/dev/null + printf -v pkgs_str -- '%q ' "${pkgs[@]}" /usr/bin/repo-add -q "${repo}${DBEXT}" ${pkgs[@]} \ - || error "repo-add ${repo}${DBEXT} ${pkgs[@]}" + || error 'repo-add %q %s' "${repo}${DBEXT}" "${pkgs_str% }" popd >/dev/null set_repo_permission "${repo}" "${arch}"
@@ -462,13 +464,15 @@ arch_repo_remove() { local arch=$2 local pkgs=(${@:3}) local dbfile="${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" + local pkgs_str
if [[ ! -f ${dbfile} ]]; then error "No database found at '%s'" "$dbfile" return 1 fi + printf -v pkgs_str -- '%q ' "${pkgs[@]}" /usr/bin/repo-remove -q "${dbfile}" ${pkgs[@]} \ - || error "repo-remove ${dbfile} ${pkgs[@]}" + || error 'repo-remove %q %s' "$dbfile" "${pkgs_str% }" set_repo_permission "${repo}" "${arch}"
I do see what you're doing, I'm just not sure why. Is the whole idea with this extra variable floating around, to avoid tokenizing "${pkgs[@]}" as separate messages? That's why "${pkgs[*]}" tokenizes the members of an array as one word by gluing the members together using the first IFS character (which is a space). You'll note I used this in testing2x.
As for using %q for filepaths that can theoretically contain spaces... good point I guess.
It's all about %q. (I did use ${ary[*]} somewhere else in the commit). The separate variable applies %q escaping to each package filename separately. Without it, if I did something like: + || error 'repo-remove %q %q' "$dbfile" "${pkgs[*]}" Then it would also escape the spaces that separate them. Anyway, correctly applying %q escaping probably isn't super-important. But, we don't really expect repo-add or repo-remove to fail; if something is wrong, any of the numerous checks leading up to actually calling them should have already caught that. If we trigger one of these error messages, something *weird* is going on, and I'd like the most precise error message possible. -- Happy hacking, ~ Luke Shumaker
On 02/22/2018 06:54 PM, Luke Shumaker wrote:
I do see what you're doing, I'm just not sure why. Is the whole idea with this extra variable floating around, to avoid tokenizing "${pkgs[@]}" as separate messages? That's why "${pkgs[*]}" tokenizes the members of an array as one word by gluing the members together using the first IFS character (which is a space). You'll note I used this in testing2x.
As for using %q for filepaths that can theoretically contain spaces... good point I guess.
It's all about %q. (I did use ${ary[*]} somewhere else in the commit). The separate variable applies %q escaping to each package filename separately. Without it, if I did something like:
+ || error 'repo-remove %q %q' "$dbfile" "${pkgs[*]}"
Then it would also escape the spaces that separate them.
But, you're using error 'repo-remove %q %s' "$dbfile" "${pkgs_str% }" with a %s which works just as well as it ever did. And you might as well do that with "${pkgs[*]}" since that also works as well as it ever did. Maybe, you should update that to work properly %q then... Or maybe, you should skip the temporary variable and use error 'repo-remove %s %s' "${dbfile@Q}" "${pkgs[*]@Q}" This will result in the variables being passed into error, after being suitably '/path quoted/rather/than/escaped/' See the bash documentation on "Parameter transformation".
Anyway, correctly applying %q escaping probably isn't super-important. But, we don't really expect repo-add or repo-remove to fail; if something is wrong, any of the numerous checks leading up to actually calling them should have already caught that. If we trigger one of these error messages, something *weird* is going on, and I'd like the most precise error message possible.
No, I agree we might as well be careful here! -- Eli Schwartz Bug Wrangler and Trusted User
On Thu, 22 Feb 2018 19:23:17 -0500, Eli Schwartz wrote:
On 02/22/2018 06:54 PM, Luke Shumaker wrote:
I do see what you're doing, I'm just not sure why. Is the whole idea with this extra variable floating around, to avoid tokenizing "${pkgs[@]}" as separate messages? That's why "${pkgs[*]}" tokenizes the members of an array as one word by gluing the members together using the first IFS character (which is a space). You'll note I used this in testing2x.
As for using %q for filepaths that can theoretically contain spaces... good point I guess.
It's all about %q. (I did use ${ary[*]} somewhere else in the commit). The separate variable applies %q escaping to each package filename separately. Without it, if I did something like:
+ || error 'repo-remove %q %q' "$dbfile" "${pkgs[*]}"
Then it would also escape the spaces that separate them.
But, you're using
error 'repo-remove %q %s' "$dbfile" "${pkgs_str% }"
with a %s which works just as well as it ever did. And you might as well do that with "${pkgs[*]}" since that also works as well as it ever did.
I guess I *should* have explained it a bit more; the escaping of the package list happens when assigning pkgs_str: printf -v pkgs_str -- '%q ' "${pkgs[@]}"
Or maybe, you should skip the temporary variable and use
error 'repo-remove %s %s' "${dbfile@Q}" "${pkgs[*]@Q}"
This will result in the variables being passed into error, after being suitably '/path quoted/rather/than/escaped/'
See the bash documentation on "Parameter transformation".
Oohh, a new Bash 4.4 feature that I missed. (It's been >1 year, I don't have an excuse). And, to be fair, I had written those ~4 lines of code (f338cb0 in Parabola's dbscripts) before Bash 4.4 existed! (I knew that I had written code to escape the package list before, and just grabbed those lines from our dbscripts.) @Q generally escapes things by wrapping them in single-quotes; %q generally escapes them by inserting backslashes. Anyway, your simpler version LGTM. Shall I submit a v2 patchset? -- Happy hacking, ~ Luke Shumaker
On 02/22/2018 08:52 PM, Luke Shumaker wrote:
I guess I *should* have explained it a bit more; the escaping of the package list happens when assigning pkgs_str:
printf -v pkgs_str -- '%q ' "${pkgs[@]}"
Hmm, true. But the version without the additional variable wins IMO.
Anyway, your simpler version LGTM. Shall I submit a v2 patchset?
Yes please. -- Eli Schwartz Bug Wrangler and Trusted User
participants (2)
-
Eli Schwartz
-
Luke Shumaker