- Arch-projects - lists.archlinux.org

[arch-projects] [dbscripts] [PATCH 0/3] Misc touchup
by Luke Shumaker 23 Feb '18

23 Feb '18

From: Luke Shumaker <lukeshu(a)parabola.nu> These are a few minor unrelated touch-ups that I noticed when doing the PKGEXT work. There's no real theme to them. Luke Shumaker (3): test: common.bash:__getCheckSum: Don't rely on IFS test: Fixup glob matching Update messages to make fuller use of printf formatters cron-jobs/ftpdir-cleanup | 8 ++++---- cron-jobs/integrity-check | 2 +- cron-jobs/sourceballs | 8 ++++---- db-functions | 8 ++++++-- db-move | 4 ++-- db-remove | 2 +- db-repo-add | 2 +- db-repo-remove | 2 +- db-update | 2 +- test/cases/ftpdir-cleanup.bats | 4 ++-- test/cases/sourceballs.bats | 4 ++-- test/lib/common.bash | 17 ++++++++++++++--- testing2x | 2 +- 13 files changed, 40 insertions(+), 25 deletions(-) -- Happy hacking, ~ Luke Shumaker

2 13

[arch-projects] [dbscripts] [GIT] Official repo DB scripts branch master updated. 20131102-69-ge53cad6
by eschwartz＠archlinux.org 22 Feb '18

22 Feb '18

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "Official repo DB scripts". The branch, master has been updated via e53cad6e4a8284165c6d0b2c7c86f6c077be693b (commit) via c82f4372440b42f879344f105e12fb358c4c0ccf (commit) via 3e01ba9a82234c1d78a14be095c18a626591ad71 (commit) via 8757c459089c0db12e57db66561f640f657e34ac (commit) via 07e6a91da1331ae61b667d878092c5b03aa49ced (commit) via b45650c3e3870b975620dd89bb6aa77585c8ff02 (commit) from 134aff74d16331231ed94f34d8417d1325d4600f (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit e53cad6e4a8284165c6d0b2c7c86f6c077be693b Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Thu Feb 15 22:34:36 2018 -0500 Globally set $PKGEXT to a bash extended glob representing valid choices. The current glob `*.pkg.tar.?z` is both less restrictive and more restrictive than makepkg, as it accepts any valid unicode character. To be more exact, it's almost completely orthogonal to the one in makepkg. makepkg only accepts .tar.gz, .tar.bz2, .tar.xz, .tar.lzo, .tar.lrz, and .tar.Z and most of those fail to match against a two-char compression type. dbscripts accepts .pkg.tar.💩z which incidentally is what I think of cherry-picking xz and gz as supported methods. Since this can be anything makepkg.conf accepts, it needs to be able to match all that, unless we decide to perform additional restrictions in which case we should still explicitly list each allowed extension. Using bash extended globbing allows us to do this relatively painlessly. Document the fact that this has *always* been some sort of glob, and update the two cases where this was (not!) being evaluated by bash [[ ... ]], to use a not-elegant-at-all proxy function is_globfile() to evaluate globs *before* testing if they exist. commit c82f4372440b42f879344f105e12fb358c4c0ccf Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Thu Feb 15 22:32:04 2018 -0500 ftpdir-cleanup,sourceballs: replace external find command with bash globbing This fully removes the use of find from the codebase, leads to a micro-optimization in a couple cases, and ensures that $PKGEXT is consistently treated as a shell globbing character (which is important because it is used as one). Of the eight instances in these files: - One was unnecessary as `cat` can natively consume all files passed to it and no directory traversal was in use. - Two were unnecessary as they were hardcoded to read a single file.... - Another four were only being used to strip leading directory paths, and can be replaced by globstar and ${filepath##*/} - The final two were checking the modification time of the files, and can be replaced with touch(1) and [[ -nt ]]. Although this introduces an additional temporary file, this is not such a big deal. commit 3e01ba9a82234c1d78a14be095c18a626591ad71 Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Thu Feb 15 22:26:24 2018 -0500 db-update: replace external find command with bash globbing Don't bother emitting errors. bash doesn't show globbing errors if it cannot read a directory to try globbing there. And the former code never aborted on errors anyway, as without `set -o pipefail` the sort command swallowed the return code. commit 8757c459089c0db12e57db66561f640f657e34ac Author: Luke Shumaker <lukeshu(a)parabola.nu> Date: Fri Feb 16 12:10:25 2018 -0500 test: db-update: verify that PKGEXT(S) is treated as a glob Commit b61a714 introduced a regression that in a db-functions function called by db-update, config:PKGEXT is treated like a fixed string, instead of as a glob. Because of inadequacy of the test-suite, this did not cause a test failure. https://lists.archlinux.org/pipermail/arch-projects/2018-February/004742.ht… The broken function checks if the repo already contains a package that matches the one being released. This did not trigger a test-suite failure because right after the broken check there is another check that checks for the exact filename of the new package file. In the test-suite, all package files have the same extension, so the latter check always "saved" us. So, modify a relevant test case (a test case that verifies that we can't release the same package twice) such that the second time we release the package, we set PKGEXT to something different (.gz, instead of the normal .xz). This way, we can be sure that when db-update rejects the operation, it's because of the glob check, not the exact-filename check. This is made slightly tricky because the __buildPackage() test helper routine tries to cache builds, to speed up the tests. Without modification, this means that it will just ignore the new PKGEXT. So, modify it to only use the cache if it can successfully copy build artifacts out, in a way that respects PKGEXT. commit 07e6a91da1331ae61b667d878092c5b03aa49ced Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Mon Feb 19 11:12:02 2018 -0500 Fix overloading PKGEXT to mean two things. PKGEXT is a makepkg variable referring to a fixed filename suffix, but we were also using it to mean a bash glob referring to candidate filenames. This is wrong, so rename it to PKGEXTS which is more descriptive of its purpose. Exclude the testsuite from this change, as the testsuite actually uses PKGEXT for its intended purpose. Fix the testsuite to consistently use PKGEXT, as it hardcoded the file extension in several cases, and pin its value to .pkg.tar.xz commit b45650c3e3870b975620dd89bb6aa77585c8ff02 Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Mon Feb 19 11:42:22 2018 -0500 Use even more bashisms. Catch some cases that were missed in the previous run. ----------------------------------------------------------------------- Summary of changes: config | 4 +++- cron-jobs/devlist-mailer | 6 +++--- cron-jobs/ftpdir-cleanup | 36 ++++++++++++++++++++++++------------ cron-jobs/integrity-check | 2 +- cron-jobs/sourceballs | 31 +++++++++++++++++++++---------- cron-jobs/update-web-db | 6 +++--- db-functions | 16 ++++++++++++++-- db-move | 4 ++-- db-update | 14 ++++++++------ test/cases/db-repo-add.bats | 6 +++--- test/cases/db-update.bats | 5 ++--- test/cases/ftpdir-cleanup.bats | 6 +++--- test/lib/common.bash | 8 ++++---- 13 files changed, 91 insertions(+), 53 deletions(-) hooks/post-receive -- Official repo DB scripts

1 0

[arch-projects] [dbscripts] [PATCH 0/3] Fix ambiguous uses of
by Eli Schwartz 20 Feb '18

20 Feb '18

This was sort of cobbled together and not really tested, so I'm not 100% sure it will work, but it looks okay, so I am posting this to get more eyes on it. I think I've actually gotten this to work properly, which is yay, and support multiple extensions, which is meh but we may need this as Luke said, if we ever decide to switch over. Which means it is probably "more proper" to do so... As a bonus, we get to micro-optimize a few external calls away which saves us a handful of forked processes and should be just as fast not counting a fraction of a second gained for all those forks? Eli Schwartz (3): db-update: replace external find command with bash globbing ftpdir-cleanup,sourceballs: replace external find command with bash globbing Globally set $PKGEXT to a bash extended glob representing valid choices. config | 3 ++- cron-jobs/ftpdir-cleanup | 24 ++++++++++++++++++------ cron-jobs/sourceballs | 21 ++++++++++++++++----- db-functions | 13 +++++++++++-- db-update | 9 ++++++--- 5 files changed, 53 insertions(+), 17 deletions(-) -- 2.16.1

4 23

[arch-projects] [dbscripts] [PATCH v3] test: db-update: @test "update same any package to same repository fails": change PKGEXT
by Luke Shumaker 20 Feb '18

20 Feb '18

From: Luke Shumaker <lukeshu(a)parabola.nu> This has the test change PKGEXT the second time it tries to release the package. Currently, this causes the tests to fail. That's a good thing; it's checking for the regression where db-functions:check_pkgrepos isn't treating PKGEXT as a glob. Without this, that regression didn't cause test failure because the checks right after it were tripping anyway. https://lists.archlinux.org/pipermail/arch-projects/2018-February/004742.ht… v2: Follow Eli's suggestion to simplify it using the check in __buildPackage v3: Simplify further by assuming __buildPackage checks PKGEXT, not PKGEXTS --- This is written againt Eli's v2 patchset (my concerns there don't affect this). You can verify--applying this patch first makes the tests fail, then applying Eli's patches make the tests pass again. Dave's objections to the __isGlobfile name and comment apply to this patch as well. test/cases/db-update.bats | 2 +- test/lib/common.bash | 7 ++++++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/test/cases/db-update.bats b/test/cases/db-update.bats index 1da7eef..2395438 100644 --- a/test/cases/db-update.bats +++ b/test/cases/db-update.bats @@ -92,7 +92,7 @@ load ../lib/common db-update checkPackage extra pkg-any-a - releasePackage extra pkg-any-a + PKGEXT=.pkg.tar.gz releasePackage extra pkg-any-a run db-update [ "$status" -ne 0 ] } diff --git a/test/lib/common.bash b/test/lib/common.bash index 540e403..d251259 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -13,6 +13,11 @@ __getCheckSum() { echo ${result[0]} } +# Check if a file exists, even if the file uses wildcards +__isGlobfile() { + [[ -f $1 ]] +} + __buildPackage() { local pkgdest=${1:-.} local p @@ -23,7 +28,7 @@ __buildPackage() { if [[ -n ${BUILDDIR} ]]; then cache=${BUILDDIR}/$(__getCheckSum PKGBUILD) - if [[ -d ${cache} ]]; then + if __isGlobfile "${cache}"/*"${PKGEXT}"; then cp -Lv ${cache}/*${PKGEXT}{,.sig} ${pkgdest} return 0 else -- 2.16.1

2 5

[arch-projects] [dbscripts] [PATCH v4] test: db-update: verify that PKGEXT(S) is treated as a glob
by Luke Shumaker 20 Feb '18

20 Feb '18

From: Luke Shumaker <lukeshu(a)parabola.nu> Commit b61a714 introduced a regression that in a db-functions function called by db-update, config:PKGEXT is treated like a fixed string, instead of as a glob. Because of inadequacy of the test-suite, this did not cause a test failure. https://lists.archlinux.org/pipermail/arch-projects/2018-February/004742.ht… The broken function checks if the repo already contains a package that matches the one being released. This did not trigger a test-suite failure because right after the broken check there is another check that checks for the exact filename of the new package file. In the test-suite, all package files have the same extension, so the latter check always "saved" us. So, modify a relevant test case (a test case that verifies that we can't release the same package twice) such that the second time we release the package, we set PKGEXT to something different (.gz, instead of the normal .xz). This way, we can be sure that when db-update rejects the operation, it's because of the glob check, not the exact-filename check. This is made slightly tricky because the __buildPackage() test helper routine tries to cache builds, to speed up the tests. Without modification, this means that it will just ignore the new PKGEXT. So, modify it to only use the cache if it can successfully copy build artifacts out, in a way that respects PKGEXT. v2: - Follow Eli's suggestion to simplify the test-case by modifying __buildPackage so that the test-case itself doesn't need to worry about the call being cached. v3: - Simplify the test-case further by having it not prune the cache after running. v4: - Follow Eli's suggestion of simplifying __buildPackage by having the copy operation itself be the test on whether to use the cache. - Improve the commit message. --- I've also submitted this as a GitHub PR. https://github.com/archlinux/dbscripts/pull/3 You can verify that applying this without Eli's fix patchset it causes a test failure, but with Eli's fix patchset the tests pass. test/cases/db-update.bats | 2 +- test/lib/common.bash | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/test/cases/db-update.bats b/test/cases/db-update.bats index 1da7eef..2395438 100644 --- a/test/cases/db-update.bats +++ b/test/cases/db-update.bats @@ -92,7 +92,7 @@ load ../lib/common db-update checkPackage extra pkg-any-a - releasePackage extra pkg-any-a + PKGEXT=.pkg.tar.gz releasePackage extra pkg-any-a run db-update [ "$status" -ne 0 ] } diff --git a/test/lib/common.bash b/test/lib/common.bash index 540e403..e9cf6c9 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -23,8 +23,7 @@ __buildPackage() { if [[ -n ${BUILDDIR} ]]; then cache=${BUILDDIR}/$(__getCheckSum PKGBUILD) - if [[ -d ${cache} ]]; then - cp -Lv ${cache}/*${PKGEXT}{,.sig} ${pkgdest} + if cp -Lv ${cache}/*${PKGEXT}{,.sig} ${pkgdest} 2>/dev/null; then return 0 else mkdir -p ${cache} -- 2.16.1

1 0

[arch-projects] [dbscripts] [PATCH v2 0/8] PKGEXT fixup
by Luke Shumaker 19 Feb '18

19 Feb '18

From: Luke Shumaker <lukeshu(a)parabola.nu> This incorporates and improves on work from 3 previously submitted patch sets: 1. My testcase patch, but - take Eli's suggestion to simplify db-update.bats - add another commit so we don't hit a bug in BATS 2. Renaming PKGEXT->PKGEXT_glob from my first fix patchset 3. Eli's fix/extglob patchset, but - re-order things between commits - also accept ".tar" (with no compression suffix) - use `old_pkgs=()` instead of `unset old_pkgs` - replace `[ -ge 1]` with `(( > 0 ))` instead of `(( > 1 ))` - db-update: use `readarray < <(... | sort -u)` instead of the O(n^2) op of using in_array in a for loop and appending to the array - common.bash, sourceballs.bats: update to work with shopt -s extglob nullglob globstar; the tests needed updated too, not just the main code The last 3 commits aren't really related to the goal of the patchset, but I wanted to include all of the work from Eli's fix patchset. I don't mean to take credit away from Eli by re-working his patches (I credit him in the commit messages); I just wanted to make it clearer what is accomplished by each change, and how each of the changes relate to our goals; as well as actually testing each of them against the test suite. Luke Shumaker (8): test: common.bash:__getCheckSum: Don't rely on IFS test: db-update: @test "update same any package to same repository fails": change PKGEXT config: Rename PKGEXT to PKGEXT_glob Correctly treat PKGEXT_glob as a glob config: let PKGEXT_glob be an extglob; have its value match makepkg ftpdir-cleanup: fix typo in a comment ("pacakge") Replace all instances of `find` command with bash globbing ftpdir-cleanup, sourceballs: swap out [ -ge 1 ] for (( > 0 )) config | 4 +++- cron-jobs/ftpdir-cleanup | 24 ++++++++++++++++++------ cron-jobs/sourceballs | 20 +++++++++++++++----- db-functions | 12 ++++++++++-- db-move | 4 ++-- db-update | 16 ++++++++++------ test/cases/db-repo-add.bats | 6 +++--- test/cases/db-update.bats | 5 +++-- test/cases/ftpdir-cleanup.bats | 4 ++-- test/cases/sourceballs.bats | 4 ++-- test/lib/common.bash | 21 ++++++++++++++++----- 11 files changed, 84 insertions(+), 36 deletions(-) -- Happy hacking, ~ Luke Shumaker

3 12

[arch-projects] [dbscripts] [PATCH 0/1] Update tests to check for glob regression
by Luke Shumaker 16 Feb '18

16 Feb '18

From: Luke Shumaker <lukeshu(a)parabola.nu> This updates the tests to check for the regression we've been discussing. This doesn't include a fix. I haven't looked too closely, or actually tested them, but Eli's fix LGTM so far. Luke Shumaker (1): test: db-update: @test "update same any package to same repository fails": change PKGEXT test/cases/db-update.bats | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) -- 2.16.1

2 4

[arch-projects] [dbscripts] [PATCH] test: fix misuse of $PKGEXT
by Eli Schwartz 16 Feb '18

16 Feb '18

As per the previous commits, this never worked and nobody noticed because the use of globbing was undocumented. Fortunately, we can fix that now, by using the new is_globfile() proxy function. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- test/cases/ftpdir-cleanup.bats | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/cases/ftpdir-cleanup.bats b/test/cases/ftpdir-cleanup.bats index 6280ce0..63eed03 100644 --- a/test/cases/ftpdir-cleanup.bats +++ b/test/cases/ftpdir-cleanup.bats @@ -13,8 +13,8 @@ __checkRepoRemovedPackage() { local pkgname for pkgname in $(__getPackageNamesFromPackageBase ${pkgbase}); do - [[ ! -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-*${PKGEXT} ]] - [[ ! -f ${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}-*${PKGEXT} ]] + ! is_globfile "${FTP_BASE}/${PKGPOOL}/${pkgname}"-*${PKGEXT} + ! is_globfile "${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}"-*${PKGEXT} done } -- 2.16.1

1 0

[arch-projects] [dbscripts] [PATCH] Do not support wildcards in PKGEXT, and standardize on xz compression.
by Eli Schwartz 16 Feb '18

16 Feb '18

This results in unpredictable behavior when used across, variously, bash [[, POSIX sh [, and find -name Its usage depended on matching only one result, which is bad practice. Moreover, it never worked in the first place as - The majority of alternative compression extensions available in makepkg do not use *exactly* two chars. - It accepted lots of extensions that aren't valid at all, like for example .pkg.tar.💩z (which is what I think of using nonstandard compression schemes). Since devtools pushes xz, and every single package currently in the repos uses xz, and AFAIK we haven't actually used any others, it makes sense to just assume xz as the only supported PKGEXT. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config b/config index d2c1942..d292c9f 100644 --- a/config +++ b/config @@ -25,7 +25,7 @@ TMPDIR="/var/tmp" ARCHES=(x86_64) DBEXT=".db.tar.gz" FILESEXT=".files.tar.gz" -PKGEXT=".pkg.tar.?z" +PKGEXT=".pkg.tar.xz" SRCEXT=".src.tar.gz" # Allowed licenses: get sourceballs only for licenses in this array -- 2.16.1

2 1

[arch-projects] [dbscripts] [PATCH 0/2] Better handling of PKGEXT
by Luke Shumaker 15 Feb '18

15 Feb '18

From: Luke Shumaker <lukeshu(a)parabola.nu> This fixes a mistake introduced in b61a714 around quoting PKGEXT, then renames PKGEXT to PKGEXT_glob to avoid that type of confusion in the future. Luke Shumaker (2): Don't quote $PKGEXT config: Rename PKGEXT to PKGEXT_glob config | 4 +++- cron-jobs/ftpdir-cleanup | 6 +++--- db-functions | 4 ++-- db-move | 4 ++-- db-update | 8 ++++---- test/cases/db-repo-add.bats | 6 +++--- test/cases/db-update.bats | 2 +- test/cases/ftpdir-cleanup.bats | 4 ++-- test/lib/common.bash | 6 ++++-- 9 files changed, 24 insertions(+), 20 deletions(-) -- Happy hacking, ~ Luke Shumaker

3 10