February 2018 - Arch-projects - lists.archlinux.org

[arch-projects] [dbscripts] [PATCH v3] test: db-update: @test "update same any package to same repository fails": change PKGEXT
by Luke Shumaker 20 Feb '18

20 Feb '18

From: Luke Shumaker <lukeshu(a)parabola.nu> This has the test change PKGEXT the second time it tries to release the package. Currently, this causes the tests to fail. That's a good thing; it's checking for the regression where db-functions:check_pkgrepos isn't treating PKGEXT as a glob. Without this, that regression didn't cause test failure because the checks right after it were tripping anyway. https://lists.archlinux.org/pipermail/arch-projects/2018-February/004742.ht… v2: Follow Eli's suggestion to simplify it using the check in __buildPackage v3: Simplify further by assuming __buildPackage checks PKGEXT, not PKGEXTS --- This is written againt Eli's v2 patchset (my concerns there don't affect this). You can verify--applying this patch first makes the tests fail, then applying Eli's patches make the tests pass again. Dave's objections to the __isGlobfile name and comment apply to this patch as well. test/cases/db-update.bats | 2 +- test/lib/common.bash | 7 ++++++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/test/cases/db-update.bats b/test/cases/db-update.bats index 1da7eef..2395438 100644 --- a/test/cases/db-update.bats +++ b/test/cases/db-update.bats @@ -92,7 +92,7 @@ load ../lib/common db-update checkPackage extra pkg-any-a - releasePackage extra pkg-any-a + PKGEXT=.pkg.tar.gz releasePackage extra pkg-any-a run db-update [ "$status" -ne 0 ] } diff --git a/test/lib/common.bash b/test/lib/common.bash index 540e403..d251259 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -13,6 +13,11 @@ __getCheckSum() { echo ${result[0]} } +# Check if a file exists, even if the file uses wildcards +__isGlobfile() { + [[ -f $1 ]] +} + __buildPackage() { local pkgdest=${1:-.} local p @@ -23,7 +28,7 @@ __buildPackage() { if [[ -n ${BUILDDIR} ]]; then cache=${BUILDDIR}/$(__getCheckSum PKGBUILD) - if [[ -d ${cache} ]]; then + if __isGlobfile "${cache}"/*"${PKGEXT}"; then cp -Lv ${cache}/*${PKGEXT}{,.sig} ${pkgdest} return 0 else -- 2.16.1

2 5

[arch-projects] [dbscripts] [PATCH v4] test: db-update: verify that PKGEXT(S) is treated as a glob
by Luke Shumaker 20 Feb '18

20 Feb '18

From: Luke Shumaker <lukeshu(a)parabola.nu> Commit b61a714 introduced a regression that in a db-functions function called by db-update, config:PKGEXT is treated like a fixed string, instead of as a glob. Because of inadequacy of the test-suite, this did not cause a test failure. https://lists.archlinux.org/pipermail/arch-projects/2018-February/004742.ht… The broken function checks if the repo already contains a package that matches the one being released. This did not trigger a test-suite failure because right after the broken check there is another check that checks for the exact filename of the new package file. In the test-suite, all package files have the same extension, so the latter check always "saved" us. So, modify a relevant test case (a test case that verifies that we can't release the same package twice) such that the second time we release the package, we set PKGEXT to something different (.gz, instead of the normal .xz). This way, we can be sure that when db-update rejects the operation, it's because of the glob check, not the exact-filename check. This is made slightly tricky because the __buildPackage() test helper routine tries to cache builds, to speed up the tests. Without modification, this means that it will just ignore the new PKGEXT. So, modify it to only use the cache if it can successfully copy build artifacts out, in a way that respects PKGEXT. v2: - Follow Eli's suggestion to simplify the test-case by modifying __buildPackage so that the test-case itself doesn't need to worry about the call being cached. v3: - Simplify the test-case further by having it not prune the cache after running. v4: - Follow Eli's suggestion of simplifying __buildPackage by having the copy operation itself be the test on whether to use the cache. - Improve the commit message. --- I've also submitted this as a GitHub PR. https://github.com/archlinux/dbscripts/pull/3 You can verify that applying this without Eli's fix patchset it causes a test failure, but with Eli's fix patchset the tests pass. test/cases/db-update.bats | 2 +- test/lib/common.bash | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/test/cases/db-update.bats b/test/cases/db-update.bats index 1da7eef..2395438 100644 --- a/test/cases/db-update.bats +++ b/test/cases/db-update.bats @@ -92,7 +92,7 @@ load ../lib/common db-update checkPackage extra pkg-any-a - releasePackage extra pkg-any-a + PKGEXT=.pkg.tar.gz releasePackage extra pkg-any-a run db-update [ "$status" -ne 0 ] } diff --git a/test/lib/common.bash b/test/lib/common.bash index 540e403..e9cf6c9 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -23,8 +23,7 @@ __buildPackage() { if [[ -n ${BUILDDIR} ]]; then cache=${BUILDDIR}/$(__getCheckSum PKGBUILD) - if [[ -d ${cache} ]]; then - cp -Lv ${cache}/*${PKGEXT}{,.sig} ${pkgdest} + if cp -Lv ${cache}/*${PKGEXT}{,.sig} ${pkgdest} 2>/dev/null; then return 0 else mkdir -p ${cache} -- 2.16.1

1 0

[arch-projects] [dbscripts] [PATCH v2 0/8] PKGEXT fixup
by Luke Shumaker 19 Feb '18

19 Feb '18

From: Luke Shumaker <lukeshu(a)parabola.nu> This incorporates and improves on work from 3 previously submitted patch sets: 1. My testcase patch, but - take Eli's suggestion to simplify db-update.bats - add another commit so we don't hit a bug in BATS 2. Renaming PKGEXT->PKGEXT_glob from my first fix patchset 3. Eli's fix/extglob patchset, but - re-order things between commits - also accept ".tar" (with no compression suffix) - use `old_pkgs=()` instead of `unset old_pkgs` - replace `[ -ge 1]` with `(( > 0 ))` instead of `(( > 1 ))` - db-update: use `readarray < <(... | sort -u)` instead of the O(n^2) op of using in_array in a for loop and appending to the array - common.bash, sourceballs.bats: update to work with shopt -s extglob nullglob globstar; the tests needed updated too, not just the main code The last 3 commits aren't really related to the goal of the patchset, but I wanted to include all of the work from Eli's fix patchset. I don't mean to take credit away from Eli by re-working his patches (I credit him in the commit messages); I just wanted to make it clearer what is accomplished by each change, and how each of the changes relate to our goals; as well as actually testing each of them against the test suite. Luke Shumaker (8): test: common.bash:__getCheckSum: Don't rely on IFS test: db-update: @test "update same any package to same repository fails": change PKGEXT config: Rename PKGEXT to PKGEXT_glob Correctly treat PKGEXT_glob as a glob config: let PKGEXT_glob be an extglob; have its value match makepkg ftpdir-cleanup: fix typo in a comment ("pacakge") Replace all instances of `find` command with bash globbing ftpdir-cleanup, sourceballs: swap out [ -ge 1 ] for (( > 0 )) config | 4 +++- cron-jobs/ftpdir-cleanup | 24 ++++++++++++++++++------ cron-jobs/sourceballs | 20 +++++++++++++++----- db-functions | 12 ++++++++++-- db-move | 4 ++-- db-update | 16 ++++++++++------ test/cases/db-repo-add.bats | 6 +++--- test/cases/db-update.bats | 5 +++-- test/cases/ftpdir-cleanup.bats | 4 ++-- test/cases/sourceballs.bats | 4 ++-- test/lib/common.bash | 21 ++++++++++++++++----- 11 files changed, 84 insertions(+), 36 deletions(-) -- Happy hacking, ~ Luke Shumaker

3 12

[arch-projects] [dbscripts] [PATCH 0/1] Update tests to check for glob regression
by Luke Shumaker 16 Feb '18

16 Feb '18

From: Luke Shumaker <lukeshu(a)parabola.nu> This updates the tests to check for the regression we've been discussing. This doesn't include a fix. I haven't looked too closely, or actually tested them, but Eli's fix LGTM so far. Luke Shumaker (1): test: db-update: @test "update same any package to same repository fails": change PKGEXT test/cases/db-update.bats | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) -- 2.16.1

2 4

[arch-projects] [dbscripts] [PATCH] test: fix misuse of $PKGEXT
by Eli Schwartz 16 Feb '18

16 Feb '18

As per the previous commits, this never worked and nobody noticed because the use of globbing was undocumented. Fortunately, we can fix that now, by using the new is_globfile() proxy function. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- test/cases/ftpdir-cleanup.bats | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/cases/ftpdir-cleanup.bats b/test/cases/ftpdir-cleanup.bats index 6280ce0..63eed03 100644 --- a/test/cases/ftpdir-cleanup.bats +++ b/test/cases/ftpdir-cleanup.bats @@ -13,8 +13,8 @@ __checkRepoRemovedPackage() { local pkgname for pkgname in $(__getPackageNamesFromPackageBase ${pkgbase}); do - [[ ! -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-*${PKGEXT} ]] - [[ ! -f ${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}-*${PKGEXT} ]] + ! is_globfile "${FTP_BASE}/${PKGPOOL}/${pkgname}"-*${PKGEXT} + ! is_globfile "${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}"-*${PKGEXT} done } -- 2.16.1

1 0

[arch-projects] [dbscripts] [PATCH] Do not support wildcards in PKGEXT, and standardize on xz compression.
by Eli Schwartz 16 Feb '18

16 Feb '18

This results in unpredictable behavior when used across, variously, bash [[, POSIX sh [, and find -name Its usage depended on matching only one result, which is bad practice. Moreover, it never worked in the first place as - The majority of alternative compression extensions available in makepkg do not use *exactly* two chars. - It accepted lots of extensions that aren't valid at all, like for example .pkg.tar.💩z (which is what I think of using nonstandard compression schemes). Since devtools pushes xz, and every single package currently in the repos uses xz, and AFAIK we haven't actually used any others, it makes sense to just assume xz as the only supported PKGEXT. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config b/config index d2c1942..d292c9f 100644 --- a/config +++ b/config @@ -25,7 +25,7 @@ TMPDIR="/var/tmp" ARCHES=(x86_64) DBEXT=".db.tar.gz" FILESEXT=".files.tar.gz" -PKGEXT=".pkg.tar.?z" +PKGEXT=".pkg.tar.xz" SRCEXT=".src.tar.gz" # Allowed licenses: get sourceballs only for licenses in this array -- 2.16.1

2 1

[arch-projects] [dbscripts] [PATCH 0/2] Better handling of PKGEXT
by Luke Shumaker 15 Feb '18

15 Feb '18

From: Luke Shumaker <lukeshu(a)parabola.nu> This fixes a mistake introduced in b61a714 around quoting PKGEXT, then renames PKGEXT to PKGEXT_glob to avoid that type of confusion in the future. Luke Shumaker (2): Don't quote $PKGEXT config: Rename PKGEXT to PKGEXT_glob config | 4 +++- cron-jobs/ftpdir-cleanup | 6 +++--- db-functions | 4 ++-- db-move | 4 ++-- db-update | 8 ++++---- test/cases/db-repo-add.bats | 6 +++--- test/cases/db-update.bats | 2 +- test/cases/ftpdir-cleanup.bats | 4 ++-- test/lib/common.bash | 6 ++++-- 9 files changed, 24 insertions(+), 20 deletions(-) -- Happy hacking, ~ Luke Shumaker

3 10

[arch-projects] [dbscripts] [GIT] Official repo DB scripts branch master updated. 20131102-59-g36b71d3
by eschwartz＠archlinux.org 15 Feb '18

15 Feb '18

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "Official repo DB scripts". The branch, master has been updated discards be2971eca4cb08aa5c128d8b206f6943d58a7dd9 (commit) via 36b71d3231aca071ad635b995342b786676ef8fe (commit) via b61a7148eaf546a31597b74d9dd8948e4a39dea1 (commit) via f4f9d1a0099c1f784c4a964e2b5651b56f629b82 (commit) via a7b497e8347fc964f8738d4dfc3c3ef23806fdc7 (commit) via 5afac1ed83479c5ff7aab134b54245ec4f5b59fe (commit) via eec8e35eba84658bdd03230c83f449d2bb437b10 (commit) This update added new revisions after undoing existing revisions. That is to say, the old revision is not a strict subset of the new revision. This situation occurs when you --force push a change and generate a repository containing something like this: * -- * -- B -- O -- O -- O (be2971eca4cb08aa5c128d8b206f6943d58a7dd9) \ N -- N -- N (36b71d3231aca071ad635b995342b786676ef8fe) When this happens we assume that you've already had alert emails for all of the O revisions, and so we here report only the revisions in the N branch from the common base, B. Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 36b71d3231aca071ad635b995342b786676ef8fe Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Mon Feb 12 20:53:06 2018 -0500 db-functions: deduplicate some repeated logic reuse getpkgfile() in getpkgfiles() commit b61a7148eaf546a31597b74d9dd8948e4a39dea1 Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Mon Feb 12 20:50:57 2018 -0500 Use more bashisms Fix numerous instances of POSIX `[ ... ]`, including reliance on ugly deprecated constructs like POSIX `-a`. Since we require bash regardless, it makes sense to take full advantage of it. bash `[[ ... ]]` does not require quoting variables as the shell natively recognizes them as variables rather than expanded strings. Use shell arithmetic rather than test, when checking numerical values. commit f4f9d1a0099c1f784c4a964e2b5651b56f629b82 Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Mon Feb 12 20:39:30 2018 -0500 Use modern bash to append to strings/arrays. Rather than using ugly hacks like arr[${#arr[*]}]="foo", bash 3.1 has the += operator. Update strings to use the same operator while we are at it. commit a7b497e8347fc964f8738d4dfc3c3ef23806fdc7 Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Mon Feb 12 19:41:29 2018 -0500 ARCHES is an array, do not attempt to call it as a string This meant only the first array element was checked in check_repo_permission(). Although arguably this should never cause real issues as something else would have to be broken if multiple architectures have different permissions, we should catch this now anyway. commit 5afac1ed83479c5ff7aab134b54245ec4f5b59fe Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Mon Feb 12 17:53:10 2018 -0500 Update messages to make full use of printf formatters libmakepkg messaging functions provide automatic access to gettext (which we do not currently make use of) in addition to cleanly separating data from message strings. In order for this to work properly, pass argv correctly from die() to libmakepkg's error() commit eec8e35eba84658bdd03230c83f449d2bb437b10 Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Mon Feb 12 17:21:39 2018 -0500 Use return codes properly when checking for failed commands. Rather than doing a test(1) on the numerical value of the return code, use the `if !` and `||` operators during control flow. ----------------------------------------------------------------------- Summary of changes: config | 2 +- cron-jobs/ftpdir-cleanup | 4 +- cron-jobs/sourceballs | 4 +- cron-jobs/update-web-db | 4 +- db-functions | 177 ++++++++++++++++++++------------------------ db-move | 28 +++---- db-remove | 18 ++--- db-repo-add | 12 +-- db-repo-remove | 8 +- db-update | 47 ++++++------ test/cases/db-repo-add.bats | 2 +- testing2x | 16 ++-- 12 files changed, 151 insertions(+), 171 deletions(-) hooks/post-receive -- Official repo DB scripts

3 3

[arch-projects] [dbscripts] [GIT] Official repo DB scripts branch master updated. 20131102-63-g134aff7
by pierre＠archlinux.org 15 Feb '18

15 Feb '18

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "Official repo DB scripts". The branch, master has been updated via 134aff74d16331231ed94f34d8417d1325d4600f (commit) via c7f1d85c0e11aaf3c9acd944388f664aea40037b (commit) via 24671e6cb2fa343fd6f15fe54b16840aed091ad4 (commit) via 883005f5450ed6a278b5cef46be1245783840b29 (commit) from 36b71d3231aca071ad635b995342b786676ef8fe (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 134aff74d16331231ed94f34d8417d1325d4600f Author: Pierre Schmitz <pierre(a)archlinux.de> Date: Thu Feb 15 17:29:57 2018 +0100 Add usage tests commit c7f1d85c0e11aaf3c9acd944388f664aea40037b Author: Pierre Schmitz <pierre(a)archlinux.de> Date: Thu Feb 15 17:07:02 2018 +0100 PR-2: Default to x86_64 for check_packages commit 24671e6cb2fa343fd6f15fe54b16840aed091ad4 Author: Pierre Schmitz <pierre(a)archlinux.de> Date: Thu Feb 15 17:04:29 2018 +0100 PR-2: Allow i686 for tests commit 883005f5450ed6a278b5cef46be1245783840b29 Author: Florian Pritz <bluewind(a)xinu.at> Date: Thu Feb 15 16:01:07 2018 +0100 Remove i686 from config We dropped it long ago and this should be reflected here rather than being changed on our servers only. Signed-off-by: Florian Pritz <bluewind(a)xinu.at> ----------------------------------------------------------------------- Summary of changes: config | 2 +- cron-jobs/check_archlinux/check_packages.py | 8 ++++---- cron-jobs/check_archlinux/parse_pkgbuilds.sh | 2 +- test/cases/common.bats | 10 ++++++++++ test/lib/common.bash | 1 + 5 files changed, 17 insertions(+), 6 deletions(-) create mode 100644 test/cases/common.bats hooks/post-receive -- Official repo DB scripts

1 0

[arch-projects] [dbscripts] [GIT] Official repo DB scripts branch master updated. 20131102-54-gbe2971e
by eschwartz＠archlinux.org 15 Feb '18

15 Feb '18

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "Official repo DB scripts". The branch, master has been updated via be2971eca4cb08aa5c128d8b206f6943d58a7dd9 (commit) from 1cad51c5af4b0aa1e1196d737edaa00938af937f (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit be2971eca4cb08aa5c128d8b206f6943d58a7dd9 Author: Luke Shumaker <LukeShu(a)sbcglobal.net> Date: Sun May 17 19:44:11 2015 -0400 Use += instead of jumping through hoops. The += operator was introduced in Bash 3.1, and was already used in some places in dbscripts, but not everywhere. For normal strings, this isn't a big deal, but appending to an array without using += is nasty. [gabrielfrancosouza(a)gmail.com: Adjust for removed files] Signed-off-by: Gabriel Souza Franco <gabrielfrancosouza(a)gmail.com> ----------------------------------------------------------------------- Summary of changes: cron-jobs/sourceballs | 6 +++--- db-functions | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) hooks/post-receive -- Official repo DB scripts

1 0