[arch-projects] [dbscripts] [PATCH 1/3] testing2x: be more generic and accept multiple testing repos
Currently multilib is a second-class citizen the way it is lumped into community, and dbscripts cannot even keep track of whether it constitutes a testing repo. Teach config to track both testing and staging repos just like the stable ones, and teach testing2x to search through TESTING_REPOS to determine which one it is operating on. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- config | 3 ++- config.local.svn-community | 3 ++- config.local.svn-packages | 3 ++- test/lib/common.bash | 3 ++- testing2x | 26 ++++++++++++++++---------- 5 files changed, 24 insertions(+), 14 deletions(-) diff --git a/config b/config index 1cfe11f4..4c76558b 100644 --- a/config +++ b/config @@ -4,7 +4,8 @@ FTP_BASE="/srv/ftp" PKGREPOS=() PKGPOOL='' SRCPOOL='' -TESTING_REPO='' +STAGING_REPOS=() +TESTING_REPOS=() STABLE_REPOS=() # VCS backend diff --git a/config.local.svn-community b/config.local.svn-community index 5d61b5ea..493f34e3 100644 --- a/config.local.svn-community +++ b/config.local.svn-community @@ -5,7 +5,8 @@ PKGPOOL='pool/community' SRCPOOL='sources/community' SVNREPO='file:///srv/repos/svn-community/svn' SVNUSER='svn-community' -TESTING_REPO='community-testing' +STAGING_REPOS=('community-staging' 'multilib-staging') +TESTING_REPOS=('community-testing' 'multilib-testing') STABLE_REPOS=('community') CLEANUP_DESTDIR="/srv/repos/svn-community/package-cleanup" diff --git a/config.local.svn-packages b/config.local.svn-packages index 34aab35c..567d002e 100644 --- a/config.local.svn-packages +++ b/config.local.svn-packages @@ -5,7 +5,8 @@ PKGPOOL='pool/packages' SRCPOOL='sources/packages' SVNREPO='file:///srv/repos/svn-packages/svn' SVNUSER='svn-packages' -TESTING_REPO='testing' +STAGING_REPOS=('staging') +TESTING_REPOS=('testing') STABLE_REPOS=('core' 'extra') CLEANUP_DESTDIR="/srv/repos/svn-packages/package-cleanup" diff --git a/test/lib/common.bash b/test/lib/common.bash index bc2b4e6d..a92a01ee 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -103,7 +103,8 @@ setup() { PKGREPOS=('core' 'extra' 'testing') PKGPOOL='pool/packages' SRCPOOL='sources/packages' - TESTING_REPO='testing' + STAGING_REPOS=('staging') + TESTING_REPOS=('testing') STABLE_REPOS=('core' 'extra') CLEANUP_DESTDIR="${TMP}/package-cleanup" SOURCE_CLEANUP_DESTDIR="${TMP}/source-cleanup" diff --git a/testing2x b/testing2x index 14970a9f..215be39c 100755 --- a/testing2x +++ b/testing2x @@ -10,7 +10,7 @@ fi # Lock everything to reduce possibility of interfering task between the different repo-updates script_lock -for repo in "${TESTING_REPO}" "${STABLE_REPOS[@]}"; do +for repo in "${TESTING_REPOS[@]}" "${STABLE_REPOS[@]}"; do for pkgarch in "${ARCHES[@]}"; do repo_lock "${repo}" "${pkgarch}" || exit 1 done @@ -18,30 +18,36 @@ done declare -A pkgs +testing_repo= for pkgbase in "$@"; do for pkgarch in "${ARCHES[@]}"; do - vcsrepo_from="$(find_repo_for_package "${pkgbase}" "${pkgarch[@]}" "${TESTING_REPO}")" + vcsrepo_from="$(find_repo_for_package "${pkgbase}" "${pkgarch[@]}" "${TESTING_REPOS[@]}")" vcsrepo_to="$(find_repo_for_package "${pkgbase}" "${pkgarch[@]}" "${STABLE_REPOS[@]}")" + repo_from=${vcsrepo_from%-@(any|${pkgarch})} + repo_to="${vcsrepo_to%-@(any|${pkgarch})}" + if [[ -z ${testing_repo} ]]; then + testing_repo=${repo_from} + elif [[ ${testing_repo} != ${repo_from} ]]; then + die "Cannot move packages from multiple repos at a time: %s" "${testing_repo} ${repo_from}" + fi if [[ ${vcsrepo_from} && ${vcsrepo_to} ]]; then - move_to="${vcsrepo_to%-@(any|${pkgarch})}" - pkgs[${move_to}]+="${pkgbase} " + pkgs[${repo_to}]+="${pkgbase} " break fi done - [[ ${vcsrepo_from} ]] || die "%s not found in [%s]" "$pkgbase" "$TESTING_REPO" + [[ ${vcsrepo_from} ]] || die "%s not found in any of these repos: %s" "${pkgbase}" "${TESTING_REPOS[@]}" [[ ${vcsrepo_to} ]] || die "%s not found in any of these repos: %s" "$pkgbase" "${STABLE_REPOS[*]}" done -for pkgarch in "${ARCHES[@]}"; do - repo_unlock "${TESTING_REPO}" "${pkgarch}" -done -for repo in "${STABLE_REPOS[@]}"; do +for repo in "${TESTING_REPOS[@]}" "${STABLE_REPOS[@]}"; do for pkgarch in "${ARCHES[@]}"; do repo_unlock "${repo}" "${pkgarch}" done +done +for repo in "${STABLE_REPOS[@]}"; do if [[ -n ${pkgs[${repo}]} ]]; then - "$(dirname "$(readlink -e "$0")")/db-move" "${TESTING_REPO}" "${repo}" ${pkgs[${repo}]} + "$(dirname "$(readlink -e "$0")")/db-move" "${testing_repo}" "${repo}" ${pkgs[${repo}]} fi done -- 2.19.2
A semi-common pattern is for one maintainer to stage a rebuild of a package due to e.g. cascading repository-wide python/boost/whatever rebuilds, and then for the original maintainer of the package to not notice and update the package in the stable repo, leaving an out of date rebuild in staging or testing. Then the the out of date package gets moved and ends up breaking things, possibly via a package downgrade, possibly via breaking compatibility with a much more targeted rebuild uploaded all at once. Ultimately, Things Happen™ and the repository hierarchy gets broken. Prevent this by enforcing for all packages that exist in multiple levels of the repo: staging -> testing -> stable That updates to one must come with an update to all the others. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- db-functions | 21 +++++++++++++++++++++ db-update | 5 +++++ test/cases/db-update.bats | 13 +++++++++++++ 3 files changed, 39 insertions(+) diff --git a/db-functions b/db-functions index 7aeedced..b47ad570 100644 --- a/db-functions +++ b/db-functions @@ -374,6 +374,27 @@ check_pkgrepos() { return 0 } +check_stagingrepos() { + local pkgfile=${1} + local pkgrepo=${2} + local pkgname=$(getpkgname "${pkgfile}") + local pkgarch=$(getpkgarch "${pkgfile}") + local candidate + + if in_array "${pkgrepo}" "${STABLE_REPOS[@]}"; then + candidate=$(find_repo_for_package "${pkgname}" "${pkgarch}" "${TESTING_REPOS[@]}") || return 0 + elif in_array "${pkgrepo}" "${TESTING_REPOS}"; then + candidate=$(find_repo_for_package "${pkgname}" "${pkgarch}" "${STAGING_REPOS[@]}") || return 0 + fi + + printf '%s\n' "${candidate%-*}" + for candidate in "${STAGING}/${candidate%-*}"/*${PKGEXTS}; do + [[ ${pkgname} = $(getpkgname "${candidate}" 2>/dev/null) ]] && return 0 + done + + return 1 +} + #usage: chk_license ${license[@]}" chk_license() { local l diff --git a/db-update b/db-update index 313fb999..f07a8a45 100755 --- a/db-update +++ b/db-update @@ -52,6 +52,11 @@ for repo in "${repos[@]}"; do if ! check_pkgrepos "${pkg}"; then die "Package %s already exists in another repository" "$repo/${pkg##*/}" fi + set -x + if ! missing_repo="$(check_stagingrepos "${pkg}" "${repo}")"; then + die "Package %s in %s needs to be updated in unstable repos as well: %s" "${pkg}" "${repo}" "${missing_repo}" + fi + set +x if ! check_packager "${pkg}"; then die "Package %s does not have a valid packager" "$repo/${pkg##*/}" fi diff --git a/test/cases/db-update.bats b/test/cases/db-update.bats index 9ee06321..f2e7fcab 100644 --- a/test/cases/db-update.bats +++ b/test/cases/db-update.bats @@ -87,6 +87,19 @@ load ../lib/common checkPackage testing pkg-any-a 1-2 } +@test "update any package to stable repo without updating testing package fails" { + releasePackage extra pkg-any-a + db-update + updatePackage pkg-any-a + releasePackage testing pkg-any-a + db-update + updatePackage pkg-any-a + releasePackage extra pkg-any-a + + run db-update + [ "$status" -ne 0 ] +} + @test "update same any package to same repository fails" { releasePackage extra pkg-any-a db-update -- 2.19.2
Don't allow anomalous testing packages floating around after a rebuild which are older than stable. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- db-move | 9 +++++++++ test/cases/db-move.bats | 14 ++++++++++++++ test/lib/common.bash | 2 ++ 3 files changed, 25 insertions(+) diff --git a/db-move b/db-move index 72aa0b35..7ded7de1 100755 --- a/db-move +++ b/db-move @@ -14,6 +14,11 @@ repo_to="${args[1]}" ftppath_from="${FTP_BASE}/${repo_from}/os/" ftppath_to="${FTP_BASE}/${repo_to}/os/" +check_leapfrog=false +if in_array "${repo_from}" "${STAGING_REPOS}" && in_array "${repo_to}" "${STABLE_REPOS}"; then + check_leapfrog=true +fi + if ! check_repo_permission "$repo_to" || ! check_repo_permission "$repo_from"; then die "You don't have permission to move packages from %s to %s" "$repo_from" "$repo_to" fi @@ -42,6 +47,10 @@ for pkgbase in "${args[@]:2}"; do die "Could not read pkgver" fi + if [[ ${check_leapfrog} = true ]] && leapfrog=$(find_repo_for_package "${pkgbase}" "${pkgarch}" "${TESTING_REPOS[@]}"); then + die "Package %s cannnot be moved from %s to %s while it exists in %s" "${pkgbase}" "${repo_from}" "${repo_to}" "${leapfrog}" + fi + for pkgname in "${pkgnames[@]}"; do getpkgfile "${ftppath_from}/${pkgarch}/${pkgname}-${pkgver}-${vcsrepo_from##*-}"${PKGEXTS} >/dev/null done diff --git a/test/cases/db-move.bats b/test/cases/db-move.bats index eb77ef9c..f4407168 100644 --- a/test/cases/db-move.bats +++ b/test/cases/db-move.bats @@ -39,6 +39,20 @@ load ../lib/common done } +@test "move package from staging to extra while a testing package exists fails" { + releasePackage extra pkg-any-a + db-update + updatePackage pkg-any-a + releasePackage testing pkg-any-a + db-update + updatePackage pkg-any-a + releasePackage staging pkg-any-a + db-update + + run db-move staging extra pkg-any-a + [ "$status" -ne 0 ] +} + @test "move single-arch packages" { local arches=('i686' 'x86_64') local pkgs=('pkg-single-arch' 'pkg-simple-b') diff --git a/test/lib/common.bash b/test/lib/common.bash index a92a01ee..03089efc 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -25,6 +25,8 @@ __isGlobfile() { [[ -f $1 ]] } +do:() { echo "${@@Q}"; "$@"; } + __buildPackage() { local pkgdest=${1:-.} local p -- 2.19.2
A semi-common pattern is for one maintainer to stage a rebuild of a package due to e.g. cascading repository-wide python/boost/whatever rebuilds, and then for the original maintainer of the package to not notice and update the package in the stable repo, leaving an out of date rebuild in staging or testing. Then the the out of date package gets moved and ends up breaking things, possibly via a package downgrade, possibly via breaking compatibility with a much more targeted rebuild uploaded all at once. Ultimately, Things Happen™ and the repository hierarchy gets broken. Prevent this by enforcing for all packages that exist in multiple levels of the repo: staging -> testing -> stable That updates to one must come with an update to all the others. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- v2: add check for stable & staging, add relevant testsuite, fix incorrectly accessed array. db-functions | 25 +++++++++++++++++++++++++ db-update | 3 +++ test/cases/db-update.bats | 27 +++++++++++++++++++++++++++ test/lib/common.bash | 2 +- 4 files changed, 56 insertions(+), 1 deletion(-) diff --git a/db-functions b/db-functions index 0e4dd939..5c363578 100644 --- a/db-functions +++ b/db-functions @@ -377,6 +377,31 @@ check_pkgrepos() { return 0 } +check_stagingrepos() { + local pkgfile=${1} + local pkgrepo=${2} + local pkgname=$(getpkgname "${pkgfile}") + local pkgarch=$(getpkgarch "${pkgfile}") + local candidate candidates=() + + if in_array "${pkgrepo}" "${STABLE_REPOS[@]}"; then + candidates+=($(find_repo_for_package "${pkgname}" "${pkgarch}" "${TESTING_REPOS[@]}")) + fi + if in_array "${pkgrepo}" "${STABLE_REPOS[@]}" "${TESTING_REPOS[@]}"; then + candidates+=($(find_repo_for_package "${pkgname}" "${pkgarch}" "${STAGING_REPOS[@]}")) + fi + (( ${#candidates[@]} == 0 )) && return 0 + + printf '%s\n' "${candidates[@]%-*}" + for candidate in "${candidates[@]}"; do + for candidate in "${STAGING}/${candidate%-*}"/*${PKGEXTS}; do + [[ ${pkgname} = $(getpkgname "${candidate}" 2>/dev/null) ]] && return 0 + done + done + + return 1 +} + #usage: chk_license ${license[@]}" chk_license() { local l diff --git a/db-update b/db-update index 4848fef0..b85295f3 100755 --- a/db-update +++ b/db-update @@ -52,6 +52,9 @@ for repo in "${repos[@]}"; do if ! check_pkgrepos "${pkg}"; then die "Package %s already exists in another repository" "$repo/${pkg##*/}" fi + if ! missing_repo="$(check_stagingrepos "${pkg}" "${repo}")"; then + die "Package %s in %s needs to be updated in unstable repos as well: %s" "${pkg}" "${repo}" "${missing_repo}" + fi if ! check_packager "${pkg}"; then die "Package %s does not have a valid packager" "$repo/${pkg##*/}" fi diff --git a/test/cases/db-update.bats b/test/cases/db-update.bats index e1782a59..ddeb4060 100644 --- a/test/cases/db-update.bats +++ b/test/cases/db-update.bats @@ -94,6 +94,33 @@ load ../lib/common [[ -f ${ARCHIVE_BASE}/packages/p/pkg-any-a/pkg-any-a-1-1-any${PKGEXT}.sig ]] } +@test "update any package to stable repo without updating testing package fails" { + releasePackage extra pkg-any-a + db-update + updatePackage pkg-any-a + releasePackage testing pkg-any-a + db-update + updatePackage pkg-any-a + releasePackage extra pkg-any-a + + run db-update + [ "$status" -ne 0 ] +} + +@test "update any package to stable repo without updating staging package fails" { + releasePackage extra pkg-any-a + db-update + updatePackage pkg-any-a + releasePackage staging pkg-any-a + db-update + updatePackage pkg-any-a + releasePackage extra pkg-any-a + + run db-update + echo "$output" + [ "$status" -ne 0 ] +} + @test "update same any package to same repository fails" { releasePackage extra pkg-any-a db-update diff --git a/test/lib/common.bash b/test/lib/common.bash index b99ad15f..02693809 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -102,7 +102,7 @@ setup() { ARCHIVE_BASE="${TMP}/archive" ARCHIVEUSER="" SVNREPO="file://${TMP}/svn-packages-repo" - PKGREPOS=('core' 'extra' 'testing') + PKGREPOS=('core' 'extra' 'testing' 'staging') PKGPOOL='pool/packages' SRCPOOL='sources/packages' STAGING_REPOS=('staging') -- 2.20.1
Don't allow anomalous testing packages floating around after a rebuild which are older than stable. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- v2: fix incorrectly accessed array db-move | 9 +++++++++ test/cases/db-move.bats | 14 ++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/db-move b/db-move index 72aa0b35..6f20397c 100755 --- a/db-move +++ b/db-move @@ -14,6 +14,11 @@ repo_to="${args[1]}" ftppath_from="${FTP_BASE}/${repo_from}/os/" ftppath_to="${FTP_BASE}/${repo_to}/os/" +check_leapfrog=false +if in_array "${repo_from}" "${STAGING_REPOS[@]}" && in_array "${repo_to}" "${STABLE_REPOS[@]}"; then + check_leapfrog=true +fi + if ! check_repo_permission "$repo_to" || ! check_repo_permission "$repo_from"; then die "You don't have permission to move packages from %s to %s" "$repo_from" "$repo_to" fi @@ -42,6 +47,10 @@ for pkgbase in "${args[@]:2}"; do die "Could not read pkgver" fi + if [[ ${check_leapfrog} = true ]] && leapfrog=$(find_repo_for_package "${pkgbase}" "${pkgarch}" "${TESTING_REPOS[@]}"); then + die "Package %s cannnot be moved from %s to %s while it exists in %s" "${pkgbase}" "${repo_from}" "${repo_to}" "${leapfrog}" + fi + for pkgname in "${pkgnames[@]}"; do getpkgfile "${ftppath_from}/${pkgarch}/${pkgname}-${pkgver}-${vcsrepo_from##*-}"${PKGEXTS} >/dev/null done diff --git a/test/cases/db-move.bats b/test/cases/db-move.bats index eb77ef9c..f4407168 100644 --- a/test/cases/db-move.bats +++ b/test/cases/db-move.bats @@ -39,6 +39,20 @@ load ../lib/common done } +@test "move package from staging to extra while a testing package exists fails" { + releasePackage extra pkg-any-a + db-update + updatePackage pkg-any-a + releasePackage testing pkg-any-a + db-update + updatePackage pkg-any-a + releasePackage staging pkg-any-a + db-update + + run db-move staging extra pkg-any-a + [ "$status" -ne 0 ] +} + @test "move single-arch packages" { local arches=('i686' 'x86_64') local pkgs=('pkg-single-arch' 'pkg-simple-b') -- 2.20.1
participants (1)
-
Eli Schwartz