[arch-projects] [dbscripts] [PATCH 0/5] Be less tightly coupled with SVN
From: Luke Shumaker lukeshu@parabola.nu
This patchset is intended to clean up dbscripts' interaction with SVN. Besides (IMO) improved code clarity, this should make it easier for those who wish to replace SVN with something else.
There are two major themes here: 1. Avoid asking SVN for information; to the extent possible, get all information directly from the pacman repos. 2. Isolate SVN-interfacing code in to functions; making it clear what interfaces the PKGBUILD tracking needs to provide.
This does not touch the test suite, which is still firmly SVN-dependent.
I've cc'd Florian Pritz because of his related work on migrating to git. https://wiki.archlinux.org/index.php/User:Bluewind/dbscripts-rewrite
Luke Shumaker (5): db-move, db-remove: Don't parse PKGBUILD files db-move: Clarify the flow of the preflight check testing2x: Discover correct repos based on the DBEXT files, not SVN Centralize all SVN access into 'abs_*' functions in a new 'db-abs' file db-move, db-abs: Simplify the abs_move_* functions
cron-jobs/sourceballs | 4 +- db-abs | 114 ++++++++++++++++++++++++++++++++++++++++++ db-functions | 61 +++++++++++++++------- db-move | 95 +++++++++-------------------------- db-remove | 18 ++----- testing2x | 40 +++++++-------- 6 files changed, 204 insertions(+), 128 deletions(-) create mode 100644 db-abs
From: Luke Shumaker lukeshu@parabola.nu
Don't try to parse PKGBUILD files from SVN; all of the information we need is already in the DBEXT files. Several programs use [[ -f PKGBUILD ]] or [[ -r PKGBUILD ]] on files from SVN; those checks can stay, just remove all instances of actually trying to *parse* those files.
As an exception, don't modify parse_pkgbuilds.sh (which is called by check_packages.py, which is called by cron-jobs/integrity-check).
Most of the attributes we need have been present in the DBEXT files for as long as `repo-add` has been part of pacman:
attribute | git | git date | ver | ver date ----------+----------+------------+-------+----------- FILENAME | aa1c0ba9 | 2006-11-20 | 3.0.0 | 2007-03-25 NAME | aa1c0ba9 | 2006-11-20 | 3.0.0 | 2007-03-25 VERSION | aa1c0ba9 | 2006-11-20 | 3.0.0 | 2007-03-25 ARCH | aa1c0ba9 | 2006-11-20 | 3.0.0 | 2007-03-25 BASE | 4b21504f | 2009-07-22 | 3.3.0 | 2009-08-02
However, `BASE` for split packages is slightly newer, and its presence relies not only on the `repo-add` version, but also the `makepkg` version used to build the package.
As of 2018-06-15, the oldest package in any of the Arch Linux repos is community/gimp-refocus, built on 2013-07-22; I don't believe it is necessary to handle packages from before that change was made (before `BASE` was set).
"Ignore space change" might be useful when viewing this diff. --- db-functions | 38 ++++++++++++++++++-- db-move | 99 ++++++++++++++++++---------------------------------- db-remove | 11 +++--- 3 files changed, 74 insertions(+), 74 deletions(-)
diff --git a/db-functions b/db-functions index 0491c22..69f35b4 100644 --- a/db-functions +++ b/db-functions @@ -6,9 +6,6 @@ shopt -s extglob globstar nullglob
-# Some PKGBUILDs need CARCH to be set -CARCH="x86_64" - # Useful functions UMASK="" set_umask () { @@ -294,6 +291,41 @@ getpkgfiles() { echo "${files[@]}" }
+# usage: getdbinfo repo arch <pkgbase|pkgname> fields +getdbinfo() { + local repo=$1 + local arch=$2 + local pkgbase=$3 + local fields=$4 + + local dbfile="${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" + local dbdir="${WORKDIR}/dbfiles/${repo}-${arch}" + if ! [[ -d $dbdir ]]; then + mkdir -p "$dbdir" + bsdtar -xf "$dbfile" -C "$dbdir" --include='*/desc' + fi + + # The grep/xargs is just a fast filter with possible + # false-postives (like a Bloom filter), because awk is + # comparatively slow. You could remove the grep/xargs line, + # and append `"$dbdir"/*/desc` to the awk command, and it + # would do the same thing; just several times slower. + grep -r -lZ -Fx -e "$pkgbase" -- "$dbdir" | xargs -r0 \ + awk -v pkgbase="$pkgbase" -v fields="$fields" -vFS='\n' -vRS='' ' + { dat[gensub(/^%(.*)%$/, "\1", 1, $1)] = gensub(FS, ",", "g", gensub($1 FS, "", 1)) } + ENDFILE { + if (dat["BASE"] == pkgbase || dat["NAME"] == pkgbase) { + n=split(fields, fieldlist, ",") + str = "" + for (i=1; i<=n; i++) + str = str (i==1?"":" ") dat[fieldlist[i]] + print str + } + delete dat + } + ' +} + check_pkgfile() { local pkgfile=$1
diff --git a/db-move b/db-move index 63e5c14..03debfc 100755 --- a/db-move +++ b/db-move @@ -28,32 +28,14 @@ done arch_svn checkout -q -N "${SVNREPO}" "${WORKDIR}/svn" >/dev/null for pkgbase in "${args[@]:2}"; do arch_svn up -q "${WORKDIR}/svn/${pkgbase}" >/dev/null - for pkgarch in "${ARCHES[@]}" 'any'; do - svnrepo_from="${WORKDIR}/svn/${pkgbase}/repos/${repo_from}-${pkgarch}" - if [[ -r ${svnrepo_from}/PKGBUILD ]]; then - pkgnames=($(. "${svnrepo_from}/PKGBUILD"; echo "${pkgname[@]}")) - if (( ${#pkgnames[@]} < 1 )); then - die "Could not read pkgname" + for tarch in "${ARCHES[@]}"; do + while read -r pkgarch pkgfile; do + svnrepo_from="${WORKDIR}/svn/${pkgbase}/repos/${repo_from}-${pkgarch}" + if [[ -r ${svnrepo_from}/PKGBUILD ]]; then + getpkgfile "${ftppath_from}/${tarch}/${pkgfile}" >/dev/null + continue 3 fi - - pkgver=$(. "${svnrepo_from}/PKGBUILD"; get_full_version) - if [[ -z ${pkgver} ]]; then - die "Could not read pkgver" - fi - - if [[ "${pkgarch}" = any ]]; then - tarches=("${ARCHES[@]}") - else - tarches=("${pkgarch}") - fi - - for pkgname in "${pkgnames[@]}"; do - for tarch in "${tarches[@]}"; do - getpkgfile "${ftppath_from}/${tarch}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXTS} >/dev/null - done - done - continue 2 - fi + done < <(getdbinfo "$repo_from" "$tarch" "$pkgbase" ARCH,FILENAME) done die "%s not found in %s" "$pkgbase" "$repo_from" done @@ -64,49 +46,36 @@ declare -A add_pkgs declare -A remove_pkgs for pkgbase in "${args[@]:2}"; do tag_list="" - for pkgarch in "${ARCHES[@]}" 'any'; do - svnrepo_from="${WORKDIR}/svn/${pkgbase}/repos/${repo_from}-${pkgarch}" - svnrepo_to="${WORKDIR}/svn/${pkgbase}/repos/${repo_to}-${pkgarch}" - - if [[ -f ${svnrepo_from}/PKGBUILD ]]; then - if [[ ${pkgarch} = any ]]; then - tarches=("${ARCHES[@]}") - else - tarches=("${pkgarch}") - fi - msg2 "%s (%s)" "$pkgbase" "${tarches[*]}" - pkgnames=($(. "${svnrepo_from}/PKGBUILD"; echo "${pkgname[@]}")) - pkgver=$(. "${svnrepo_from}/PKGBUILD"; get_full_version) - - if [[ -d ${svnrepo_to} ]]; then - for file in $(arch_svn ls "${svnrepo_to}"); do - arch_svn rm -q "${svnrepo_to}/$file@" + for tarch in "${ARCHES[@]}"; do + while read -r pkgname pkgver pkgarch pkgfile; do + svnrepo_from="${WORKDIR}/svn/${pkgbase}/repos/${repo_from}-${pkgarch}" + svnrepo_to="${WORKDIR}/svn/${pkgbase}/repos/${repo_to}-${pkgarch}" + if [[ -f ${svnrepo_from}/PKGBUILD ]]; then + msg2 "%s (%s)" "$pkgbase" "$pkgarch" + + if [[ -d ${svnrepo_to} ]]; then + for file in $(arch_svn ls "${svnrepo_to}"); do + arch_svn rm -q "${svnrepo_to}/$file@" + done + else + mkdir "${svnrepo_to}" + arch_svn add -q "${svnrepo_to}" + fi + + for file in $(arch_svn ls "${svnrepo_from}"); do + arch_svn mv -q -r HEAD "${svnrepo_from}/$file@" "${svnrepo_to}/" done - else - mkdir "${svnrepo_to}" - arch_svn add -q "${svnrepo_to}" + arch_svn rm --force -q "${svnrepo_from}" + tag_list+=", $pkgarch" fi
- for file in $(arch_svn ls "${svnrepo_from}"); do - arch_svn mv -q -r HEAD "${svnrepo_from}/$file@" "${svnrepo_to}/" - done - arch_svn rm --force -q "${svnrepo_from}" - tag_list+=", $pkgarch" - - for pkgname in "${pkgnames[@]}"; do - for tarch in "${tarches[@]}"; do - pkgpath=$(getpkgfile "${ftppath_from}/${tarch}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXTS}) - pkgfile="${pkgpath##*/}" - - ln -s "../../../${PKGPOOL}/${pkgfile}" "${ftppath_to}/${tarch}/" - if [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile}.sig ]]; then - ln -s "../../../${PKGPOOL}/${pkgfile}.sig" "${ftppath_to}/${tarch}/" - fi - add_pkgs[${tarch}]+="${FTP_BASE}/${PKGPOOL}/${pkgfile} " - remove_pkgs[${tarch}]+="${pkgname} " - done - done - fi + ln -s "../../../${PKGPOOL}/${pkgfile}" "${ftppath_to}/${tarch}/" + if [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile}.sig ]]; then + ln -s "../../../${PKGPOOL}/${pkgfile}.sig" "${ftppath_to}/${tarch}/" + fi + add_pkgs[${tarch}]+="${FTP_BASE}/${PKGPOOL}/${pkgfile} " + remove_pkgs[${tarch}]+="${pkgname} " + done < <(getdbinfo "$repo_from" "$tarch" "$pkgbase" NAME,VERSION,ARCH,FILENAME) done tag_list="${tag_list#, }" arch_svn commit -q "${WORKDIR}/svn/${pkgbase}" -m "${0##*/}: moved ${pkgbase} from [${repo_from}] to [${repo_to}] (${tag_list})" diff --git a/db-remove b/db-remove index ac9a168..3017026 100755 --- a/db-remove +++ b/db-remove @@ -32,17 +32,16 @@ done remove_pkgs=() for pkgbase in "${pkgbases[@]}"; do msg "Removing %s from [%s]..." "$pkgbase" "$repo" - arch_svn checkout -q "${SVNREPO}/${pkgbase}" "${WORKDIR}/svn/${pkgbase}" >/dev/null
+ mapfile -t pkgnames < <(getdbinfo "$repo" "${tarches[0]}" "$pkgbase" NAME) + remove_pkgs+=("${pkgnames[@]}") + + arch_svn checkout -q "${SVNREPO}/${pkgbase}" "${WORKDIR}/svn/${pkgbase}" >/dev/null if [[ -d ${WORKDIR}/svn/$pkgbase/repos/$svnrepo ]]; then - remove_pkgs+=($(. "${WORKDIR}/svn/$pkgbase/repos/$svnrepo/PKGBUILD"; echo ${pkgname[@]})) arch_svn rm --force -q "${WORKDIR}/svn/$pkgbase/repos/$svnrepo" arch_svn commit -q "${WORKDIR}/svn/$pkgbase" -m "${0##*/}: $pkgbase removed by $(id -un)" else - warning "%s not found in %s" "$pkgbase" "$svnrepo" - warning "Removing only %s from the repo" "$pkgbase" - warning "If it was a split package you have to remove the others yourself!" - remove_pkgs+=("$pkgbase") + warning "pkgbase '%s' not found in svn; unable to commit removal to svn" "$pkgbase" fi done
From: Luke Shumaker lukeshu@parabola.nu
This should have the same result, but be a bit less confusing to read. --- db-move | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/db-move b/db-move index 03debfc..5f197b8 100755 --- a/db-move +++ b/db-move @@ -29,15 +29,22 @@ arch_svn checkout -q -N "${SVNREPO}" "${WORKDIR}/svn" >/dev/null for pkgbase in "${args[@]:2}"; do arch_svn up -q "${WORKDIR}/svn/${pkgbase}" >/dev/null for tarch in "${ARCHES[@]}"; do + found=false while read -r pkgarch pkgfile; do + svnrepo_from="${WORKDIR}/svn/${pkgbase}/repos/${repo_from}-${pkgarch}" - if [[ -r ${svnrepo_from}/PKGBUILD ]]; then - getpkgfile "${ftppath_from}/${tarch}/${pkgfile}" >/dev/null - continue 3 + if ! [[ -r ${svnrepo_from}/PKGBUILD ]]; then + die "%s not found in %s" "$pkgbase" "$repo_from" fi + + # getpkgfile will `exit` for us if it fails; + # no need to check its result + getpkgfile "${ftppath_from}/${tarch}/${pkgfile}" >/dev/null + + found=true done < <(getdbinfo "$repo_from" "$tarch" "$pkgbase" ARCH,FILENAME) + [[ $found = true ]] || die "%s not found in %s" "$pkgbase" "$repo_from" done - die "%s not found in %s" "$pkgbase" "$repo_from" done
msg "Moving packages from [%s] to [%s]..." "$repo_from" "$repo_to"
From: Luke Shumaker lukeshu@parabola.nu
testing2x needs to: 1. Verify that the new version of the package is found in TESTING_REPO 2. Determine which of STABLE_REPOS the old version of the package is in
Currently, to do those things, it checks for the existence of PKGBUILD files in SVN. That information is already stored in the DBEXT files; get it from there, instead of talking to SVN.
"Ignore space change" might be useful when viewing this diff. --- testing2x | 40 +++++++++++++++++----------------------- 1 file changed, 17 insertions(+), 23 deletions(-)
diff --git a/testing2x b/testing2x index d3b096b..f455a93 100755 --- a/testing2x +++ b/testing2x @@ -19,31 +19,25 @@ done declare -A pkgs
for pkgbase in "$@"; do - if [[ ! -d ${WORKDIR}/${pkgbase} ]]; then - arch_svn export -q "${SVNREPO}/${pkgbase}/repos" "${WORKDIR}/${pkgbase}" >/dev/null - - found_source=false - for pkgarch in "${ARCHES[@]}" 'any'; do - svnrepo_from="${WORKDIR}/${pkgbase}/${TESTING_REPO}-${pkgarch}" - if [[ -r ${svnrepo_from}/PKGBUILD ]]; then - found_source=true - break + found_source=false + for tarch in "${ARCHES[@]}"; do + if [[ -n $(getdbinfo "$TESTING_REPO" "$tarch" "$pkgbase" NAME) ]]; then + found_source=true + break + fi + done + [[ $found_source = true ]] || die "%s not found in [%s]" "$pkgbase" "$TESTING_REPO" + found_target=false + for tarch in "${ARCHES[@]}"; do + for repo in "${STABLE_REPOS[@]}"; do + if [[ -n $(getdbinfo "$repo" "$tarch" "$pkgbase" NAME) ]]; then + found_target=true + pkgs[${repo}]+="${pkgbase} " + break 2 fi done - [[ $found_source = true ]] || die "%s not found in [%s]" "$pkgbase" "$TESTING_REPO" - found_target=false - for pkgarch in "${ARCHES[@]}" 'any'; do - for repo in "${STABLE_REPOS[@]}"; do - svnrepo_to="${WORKDIR}/${pkgbase}/${repo}-${pkgarch}" - if [[ -r ${svnrepo_to}/PKGBUILD ]]; then - found_target=true - pkgs[${repo}]+="${pkgbase} " - break 2 - fi - done - done - [[ $found_target = true ]] || die "%s not found in any of these repos: %s" "$pkgbase" "${STABLE_REPOS[*]}" - fi + done + [[ $found_target = true ]] || die "%s not found in any of these repos: %s" "$pkgbase" "${STABLE_REPOS[*]}" done
for pkgarch in "${ARCHES[@]}"; do
From: Luke Shumaker lukeshu@parabola.nu
Moving all SVN code in to a separate file means both that: 1. It is easier to identify the interactions with SVN, when considering a replacement. 2. It is easier to swap out the one file if/when replacing SVN with something else.
Put another way: try to be less tightly coupled with SVN. --- cron-jobs/sourceballs | 4 +- db-abs | 106 ++++++++++++++++++++++++++++++++++++++++++ db-functions | 23 ++++----- db-move | 31 ++---------- db-remove | 9 +--- 5 files changed, 121 insertions(+), 52 deletions(-) create mode 100644 db-abs
diff --git a/cron-jobs/sourceballs b/cron-jobs/sourceballs index 6be28ab..f56f0cf 100755 --- a/cron-jobs/sourceballs +++ b/cron-jobs/sourceballs @@ -79,8 +79,8 @@ for repo in "${PKGREPOS[@]}"; do
# Get the sources from svn mkdir -p -m0770 "${WORKDIR}/pkgbuilds/${repo}-${pkgarch}" - arch_svn export -q "${SVNREPO}/${pkgbase}/repos/${repo}-${pkgarch}" \ - "${WORKDIR}/pkgbuilds/${repo}-${pkgarch}/${pkgbase}" >/dev/null 2>&1 + abs_export "$repo" "$pkgarch" "$pkgbase" \ + "${WORKDIR}/pkgbuilds/${repo}-${pkgarch}/${pkgbase}" if (( $? >= 1 )); then failedpkgs+=("${pkgbase}-${pkgver}${SRCEXT}") continue diff --git a/db-abs b/db-abs new file mode 100644 index 0000000..5a34b84 --- /dev/null +++ b/db-abs @@ -0,0 +1,106 @@ +#!/hint/bash + +arch_svn() { + if [[ -z ${SVNUSER} ]]; then + /usr/bin/svn "${@}" + else + sudo -u "${SVNUSER}" -- /usr/bin/svn --username "${USER}" "${@}" + fi +} + +_abs_checkout() { + local pkgbase=$1 + if ! [[ -d ${WORKDIR}/svn ]]; then + arch_svn checkout -q -N "${SVNREPO}" "${WORKDIR}/svn" >/dev/null + fi + if ! [[ -d ${WORKDIR}/svn/${pkgbase} ]]; then + arch_svn up -q "${WORKDIR}/svn/${pkgbase}" >/dev/null + fi +} + +abs_move_preflight_check() { + local repo_from=$1 + local pkgarch=$2 + local pkgbase=$3 + + _abs_checkout "$pkgbase" + local svnrepo_from="${WORKDIR}/svn/${pkgbase}/repos/${repo_from}-${pkgarch}" + [[ -r ${svnrepo_from}/PKGBUILD ]] +} + +abs_move_start() { + abs_move_repo_from=$1 + abs_move_repo_to=$2 + abs_move_pkgbase=$3 + + abs_move_tag_list="" +} + +# If the "from" PKGBUILD doesn't exist, this is a no-op (not an +# error), so that it can be run for each arch, and the invoker doesn't +# need to worry about hoisting it out of the loop if arch=(any). If +# the nonexistence is such that it should be an error, we count on +# abs_move_preflight_check having already caught that. +abs_move_arch() { + local pkgarch=$1 + + local repo_from=$abs_move_repo_from + local repo_to=$abs_move_repo_to + local pkgbase=$abs_move_pkgbase + + local svnrepo_from="${WORKDIR}/svn/${pkgbase}/repos/${repo_from}-${pkgarch}" + local svnrepo_to="${WORKDIR}/svn/${pkgbase}/repos/${repo_to}-${pkgarch}" + if [[ -f ${svnrepo_from}/PKGBUILD ]]; then + msg2 "%s (%s)" "$pkgbase" "$pkgarch" + + if [[ -d ${svnrepo_to} ]]; then + for file in $(arch_svn ls "${svnrepo_to}"); do + arch_svn rm -q "${svnrepo_to}/$file@" + done + else + mkdir "${svnrepo_to}" + arch_svn add -q "${svnrepo_to}" + fi + + for file in $(arch_svn ls "${svnrepo_from}"); do + arch_svn mv -q -r HEAD "${svnrepo_from}/$file@" "${svnrepo_to}/" + done + arch_svn rm --force -q "${svnrepo_from}" + abs_move_tag_list+=", $pkgarch" + fi +} + +abs_move_finish() { + local repo_from=$abs_move_repo_from + local repo_to=$abs_move_repo_to + local pkgbase=$abs_move_pkgbase + + local tag_list="${abs_move_tag_list#, }" + arch_svn commit -q "${WORKDIR}/svn/${pkgbase}" -m "${0##*/}: moved ${pkgbase} from [${repo_from}] to [${repo_to}] (${tag_list})" +} + +abs_remove() { + local repo=$1 + local arch=$2 + local pkgbase=$3 + + local svnrepo="$repo-$arch" + + _abs_checkout "$pkgbase" + if [[ -d ${WORKDIR}/svn/$pkgbase/repos/$svnrepo ]]; then + arch_svn rm --force -q "${WORKDIR}/svn/$pkgbase/repos/$svnrepo" + arch_svn commit -q "${WORKDIR}/svn/$pkgbase" -m "${0##*/}: $pkgbase removed by $(id -un)" + else + warning "pkgbase '%s' not found in svn; unable to commit removal to svn" "$pkgbase" + fi +} + +abs_export() { + local repo=$1 + local pkgarch=$2 + local pkgbase=$3 + local dest=$4 + + arch_svn export -q "${SVNREPO}/${pkgbase}/repos/${repo}-${pkgarch}" \ + "${dest}" >/dev/null 2>&1 +} diff --git a/db-functions b/db-functions index 69f35b4..340c794 100644 --- a/db-functions +++ b/db-functions @@ -1,6 +1,7 @@ #!/hint/bash
. /usr/share/makepkg/util.sh +. "$(dirname "${BASH_SOURCE[0]}")/db-abs"
# global shell options for enhanced bash scripting shopt -s extglob globstar nullglob @@ -348,16 +349,16 @@ check_pkgsvn() {
in_array "${repo}" "${PKGREPOS[@]}" || return 1
- if [[ ! -f ${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase} ]]; then + if [[ ! -f ${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}/PKGBUILD ]]; then mkdir -p "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}" - arch_svn export -q "${SVNREPO}/${_pkgbase}/repos/${repo}-${_pkgarch}/PKGBUILD" \ + abs_export "$repo" "$_pkgarch" "$_pkgbase" \ "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}" >/dev/null || return 1 fi
- local svnver="$(. "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}"; get_full_version)" + local svnver="$(. "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}/PKGBUILD"; get_full_version)" [[ "${svnver}" = "${_pkgver}" ]] || return 1
- local svnnames=($(. "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}"; echo "${pkgname[@]}")) + local svnnames=($(. "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}/PKGBUILD"; echo "${pkgname[@]}")) in_array "${_pkgname}" "${svnnames[@]}" || return 1
return 0 @@ -382,13 +383,13 @@ check_splitpkgs() { mkdir -p "${repo}/${_pkgarch}/${_pkgbase}" echo "${_pkgname}" >> "${repo}/${_pkgarch}/${_pkgbase}/staging"
- if [[ ! -f ${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase} ]]; then + if [[ ! -f ${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}/PKGBUILD ]]; then mkdir -p "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}" - arch_svn export -q "${SVNREPO}/${_pkgbase}/repos/${repo}-${_pkgarch}/PKGBUILD" \ + abs_export "$repo" "$_pkgarch" "$_pkgbase" \ "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}" >/dev/null || return 1 fi
- local svnnames=($(. "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}"; echo "${pkgname[@]}")) + local svnnames=($(. "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}/PKGBUILD"; echo "${pkgname[@]}")) printf '%s\n' "${svnnames[@]}" >> "${repo}/${_pkgarch}/${_pkgbase}/svn" done popd >/dev/null @@ -489,11 +490,3 @@ arch_repo_modify() {
REPO_MODIFIED=1 } - -arch_svn() { - if [[ -z ${SVNUSER} ]]; then - /usr/bin/svn "${@}" - else - sudo -u "${SVNUSER}" -- /usr/bin/svn --username "${USER}" "${@}" - fi -} diff --git a/db-move b/db-move index 5f197b8..8a1c24d 100755 --- a/db-move +++ b/db-move @@ -25,15 +25,12 @@ for pkgarch in "${ARCHES[@]}"; do done
# check if packages to be moved exist in svn and ftp dir -arch_svn checkout -q -N "${SVNREPO}" "${WORKDIR}/svn" >/dev/null for pkgbase in "${args[@]:2}"; do - arch_svn up -q "${WORKDIR}/svn/${pkgbase}" >/dev/null for tarch in "${ARCHES[@]}"; do found=false while read -r pkgarch pkgfile; do
- svnrepo_from="${WORKDIR}/svn/${pkgbase}/repos/${repo_from}-${pkgarch}" - if ! [[ -r ${svnrepo_from}/PKGBUILD ]]; then + if ! abs_move_preflight_check "$repo_from" "$pkgarch" "$pkgbase"; then die "%s not found in %s" "$pkgbase" "$repo_from" fi
@@ -52,29 +49,10 @@ msg "Moving packages from [%s] to [%s]..." "$repo_from" "$repo_to" declare -A add_pkgs declare -A remove_pkgs for pkgbase in "${args[@]:2}"; do - tag_list="" + abs_move_start "$repo_from" "$repo_to" "$pkgbase" for tarch in "${ARCHES[@]}"; do while read -r pkgname pkgver pkgarch pkgfile; do - svnrepo_from="${WORKDIR}/svn/${pkgbase}/repos/${repo_from}-${pkgarch}" - svnrepo_to="${WORKDIR}/svn/${pkgbase}/repos/${repo_to}-${pkgarch}" - if [[ -f ${svnrepo_from}/PKGBUILD ]]; then - msg2 "%s (%s)" "$pkgbase" "$pkgarch" - - if [[ -d ${svnrepo_to} ]]; then - for file in $(arch_svn ls "${svnrepo_to}"); do - arch_svn rm -q "${svnrepo_to}/$file@" - done - else - mkdir "${svnrepo_to}" - arch_svn add -q "${svnrepo_to}" - fi - - for file in $(arch_svn ls "${svnrepo_from}"); do - arch_svn mv -q -r HEAD "${svnrepo_from}/$file@" "${svnrepo_to}/" - done - arch_svn rm --force -q "${svnrepo_from}" - tag_list+=", $pkgarch" - fi + abs_move_arch "$pkgarch"
ln -s "../../../${PKGPOOL}/${pkgfile}" "${ftppath_to}/${tarch}/" if [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile}.sig ]]; then @@ -84,8 +62,7 @@ for pkgbase in "${args[@]:2}"; do remove_pkgs[${tarch}]+="${pkgname} " done < <(getdbinfo "$repo_from" "$tarch" "$pkgbase" NAME,VERSION,ARCH,FILENAME) done - tag_list="${tag_list#, }" - arch_svn commit -q "${WORKDIR}/svn/${pkgbase}" -m "${0##*/}: moved ${pkgbase} from [${repo_from}] to [${repo_to}] (${tag_list})" + abs_move_finish done
for tarch in "${ARCHES[@]}"; do diff --git a/db-remove b/db-remove index 3017026..93430b6 100755 --- a/db-remove +++ b/db-remove @@ -13,7 +13,6 @@ arch="$2" pkgbases=("${@:3}")
ftppath="$FTP_BASE/$repo/os" -svnrepo="$repo-$arch"
if ! check_repo_permission "$repo"; then die "You don't have permission to remove packages from %s" "$repo" @@ -36,13 +35,7 @@ for pkgbase in "${pkgbases[@]}"; do mapfile -t pkgnames < <(getdbinfo "$repo" "${tarches[0]}" "$pkgbase" NAME) remove_pkgs+=("${pkgnames[@]}")
- arch_svn checkout -q "${SVNREPO}/${pkgbase}" "${WORKDIR}/svn/${pkgbase}" >/dev/null - if [[ -d ${WORKDIR}/svn/$pkgbase/repos/$svnrepo ]]; then - arch_svn rm --force -q "${WORKDIR}/svn/$pkgbase/repos/$svnrepo" - arch_svn commit -q "${WORKDIR}/svn/$pkgbase" -m "${0##*/}: $pkgbase removed by $(id -un)" - else - warning "pkgbase '%s' not found in svn; unable to commit removal to svn" "$pkgbase" - fi + abs_remove "$repo" "$arch" "$pkgbase" done
for tarch in "${tarches[@]}"; do
From: Luke Shumaker lukeshu@parabola.nu
Currently, abs_move_preflight_check and abs_move_arch require the exact pkgarch (i.e. "any" or "x86_64"). Modify them so that they take the tarch (i.e. "x86_64") and figure out themselves if it needs to set pkgarch=$tarch or pkgarch=any.
abs_export could probably stand to be modified to do something similar, but it's more work to adjust its callers.
"Ignore space change" might be useful when viewing this diff. --- db-abs | 56 ++++++++++++++++++++++++++++++++------------------------ db-move | 20 +++++++++----------- 2 files changed, 41 insertions(+), 35 deletions(-)
diff --git a/db-abs b/db-abs index 5a34b84..c510bad 100644 --- a/db-abs +++ b/db-abs @@ -20,12 +20,12 @@ _abs_checkout() {
abs_move_preflight_check() { local repo_from=$1 - local pkgarch=$2 + local tarch=$2 local pkgbase=$3
_abs_checkout "$pkgbase" - local svnrepo_from="${WORKDIR}/svn/${pkgbase}/repos/${repo_from}-${pkgarch}" - [[ -r ${svnrepo_from}/PKGBUILD ]] + local reposdir="${WORKDIR}/svn/${pkgbase}/repos" + [[ -r ${reposdir}/${repo_from}-${tarch}/PKGBUILD || -r ${reposdir}/${repo_from}-any/PKGBUILD ]] }
abs_move_start() { @@ -37,37 +37,45 @@ abs_move_start() { }
# If the "from" PKGBUILD doesn't exist, this is a no-op (not an -# error), so that it can be run for each arch, and the invoker doesn't -# need to worry about hoisting it out of the loop if arch=(any). If -# the nonexistence is such that it should be an error, we count on -# abs_move_preflight_check having already caught that. +# error). This is because we expect to be called exactly once for +# each tarch (eg: x86_64, i686), but if arch=(any) then we only need +# do the work once; on the subsequent calls the "from" PKGBUILD won't +# exist anymore. If the source PKGBUILD never existed, we expect that +# to have already been caught by abs_move_preflight_check. abs_move_arch() { - local pkgarch=$1 + local tarch=$1
local repo_from=$abs_move_repo_from local repo_to=$abs_move_repo_to local pkgbase=$abs_move_pkgbase
+ local pkgarch + if [[ -r "${WORKDIR}/svn/${pkgbase}/repos/${repo_from}-${tarch}/PKGBUILD" ]]; then + pkgarch=$tarch + elif [[ -r "${WORKDIR}/svn/${pkgbase}/repos/${repo_from}-any/PKGBUILD" ]]; then + pkgarch=any + else + return 0 + fi + local svnrepo_from="${WORKDIR}/svn/${pkgbase}/repos/${repo_from}-${pkgarch}" local svnrepo_to="${WORKDIR}/svn/${pkgbase}/repos/${repo_to}-${pkgarch}" - if [[ -f ${svnrepo_from}/PKGBUILD ]]; then - msg2 "%s (%s)" "$pkgbase" "$pkgarch" - - if [[ -d ${svnrepo_to} ]]; then - for file in $(arch_svn ls "${svnrepo_to}"); do - arch_svn rm -q "${svnrepo_to}/$file@" - done - else - mkdir "${svnrepo_to}" - arch_svn add -q "${svnrepo_to}" - fi - - for file in $(arch_svn ls "${svnrepo_from}"); do - arch_svn mv -q -r HEAD "${svnrepo_from}/$file@" "${svnrepo_to}/" + msg2 "%s (%s)" "$pkgbase" "$pkgarch" + + if [[ -d ${svnrepo_to} ]]; then + for file in $(arch_svn ls "${svnrepo_to}"); do + arch_svn rm -q "${svnrepo_to}/$file@" done - arch_svn rm --force -q "${svnrepo_from}" - abs_move_tag_list+=", $pkgarch" + else + mkdir "${svnrepo_to}" + arch_svn add -q "${svnrepo_to}" fi + + for file in $(arch_svn ls "${svnrepo_from}"); do + arch_svn mv -q -r HEAD "${svnrepo_from}/$file@" "${svnrepo_to}/" + done + arch_svn rm --force -q "${svnrepo_from}" + abs_move_tag_list+=", $pkgarch" }
abs_move_finish() { diff --git a/db-move b/db-move index 8a1c24d..7fc0041 100755 --- a/db-move +++ b/db-move @@ -27,19 +27,18 @@ done # check if packages to be moved exist in svn and ftp dir for pkgbase in "${args[@]:2}"; do for tarch in "${ARCHES[@]}"; do - found=false - while read -r pkgarch pkgfile; do - - if ! abs_move_preflight_check "$repo_from" "$pkgarch" "$pkgbase"; then - die "%s not found in %s" "$pkgbase" "$repo_from" - fi + if ! abs_move_preflight_check "$repo_from" "$tarch" "$pkgbase"; then + die "%s not found in %s" "$pkgbase" "$repo_from" + fi
+ found=false + while read -r pkgfile; do # getpkgfile will `exit` for us if it fails; # no need to check its result getpkgfile "${ftppath_from}/${tarch}/${pkgfile}" >/dev/null
found=true - done < <(getdbinfo "$repo_from" "$tarch" "$pkgbase" ARCH,FILENAME) + done < <(getdbinfo "$repo_from" "$tarch" "$pkgbase" FILENAME) [[ $found = true ]] || die "%s not found in %s" "$pkgbase" "$repo_from" done done @@ -51,16 +50,15 @@ declare -A remove_pkgs for pkgbase in "${args[@]:2}"; do abs_move_start "$repo_from" "$repo_to" "$pkgbase" for tarch in "${ARCHES[@]}"; do - while read -r pkgname pkgver pkgarch pkgfile; do - abs_move_arch "$pkgarch" - + abs_move_arch "$tarch" + while read -r pkgname pkgver pkgfile; do ln -s "../../../${PKGPOOL}/${pkgfile}" "${ftppath_to}/${tarch}/" if [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile}.sig ]]; then ln -s "../../../${PKGPOOL}/${pkgfile}.sig" "${ftppath_to}/${tarch}/" fi add_pkgs[${tarch}]+="${FTP_BASE}/${PKGPOOL}/${pkgfile} " remove_pkgs[${tarch}]+="${pkgname} " - done < <(getdbinfo "$repo_from" "$tarch" "$pkgbase" NAME,VERSION,ARCH,FILENAME) + done < <(getdbinfo "$repo_from" "$tarch" "$pkgbase" NAME,VERSION,FILENAME) done abs_move_finish done
On 06/19/2018 12:20 AM, Luke Shumaker wrote:
From: Luke Shumaker lukeshu@parabola.nu
This patchset is intended to clean up dbscripts' interaction with SVN. Besides (IMO) improved code clarity, this should make it easier for those who wish to replace SVN with something else.
There are two major themes here:
- Avoid asking SVN for information; to the extent possible, get all information directly from the pacman repos.
Why is this a goal? Especially, because currently dbscripts will error out on you if you do something like create a PKGBUILD in a directory that doesn't match the pkgbase. This is a feature, because we don't want that to happen.
We want the PKGBUILDs repository to match the expected pacman repository action, so this information should always be sourced from the PKGBUILD.
There's a WIP rewrite here: https://wiki.archlinux.org/index.php/User:Bluewind/dbscripts-rewrite
Which has e.g. get_file_from_pkgrepo which cat's the PKGBUILD in a VCS-agnostic manner. This is the direction I think we want to take, and IMHO also has a lot more code clarity than a mini awk program reading from pacman's technically undocumented database format (makepkg/PKGBUILD is quite well documented, I'm not sure we make any guarantees about the database format except that libalpm can read it).
- Isolate SVN-interfacing code in to functions; making it clear what interfaces the PKGBUILD tracking needs to provide.
The dbscripts-rewrite aims to do this too, but in a very different way. I'd like to keep to those semantics in order to reduce the effort it takes to merge those changes.
This does not touch the test suite, which is still firmly SVN-dependent.
I've cc'd Florian Pritz because of his related work on migrating to git. https://wiki.archlinux.org/index.php/User:Bluewind/dbscripts-rewrite
Luke Shumaker (5): db-move, db-remove: Don't parse PKGBUILD files db-move: Clarify the flow of the preflight check testing2x: Discover correct repos based on the DBEXT files, not SVN Centralize all SVN access into 'abs_*' functions in a new 'db-abs' file db-move, db-abs: Simplify the abs_move_* functions
cron-jobs/sourceballs | 4 +- db-abs | 114 ++++++++++++++++++++++++++++++++++++++++++ db-functions | 61 +++++++++++++++------- db-move | 95 +++++++++-------------------------- db-remove | 18 ++----- testing2x | 40 +++++++-------- 6 files changed, 204 insertions(+), 128 deletions(-) create mode 100644 db-abs
On Fri, 22 Jun 2018 12:52:04 -0400, Eli Schwartz wrote:
On 06/19/2018 12:20 AM, Luke Shumaker wrote:
From: Luke Shumaker lukeshu@parabola.nu
This patchset is intended to clean up dbscripts' interaction with SVN. Besides (IMO) improved code clarity, this should make it easier for those who wish to replace SVN with something else.
There are two major themes here:
- Avoid asking SVN for information; to the extent possible, get all information directly from the pacman repos.
Why is this a goal? Especially, because currently dbscripts will error out on you if you do something like create a PKGBUILD in a directory that doesn't match the pkgbase. This is a feature, because we don't want that to happen.
That should not change: - db-update will still do the check_pkgsvn and such that check the PKGBUILD - db-move, db-remove will still move/remove the PKGBUILD, and will error if it isn't where they expect it.
Replacing SVN with something else (whether it be in Parabola, or in dbscripts-rewrite) involves considering many interactions between dbscripts and the VCS repo. That's hard to do (as evidenced by issues in the Parabola bug tracker, and by "TODO" comments in dbscripts-rewrite). By removing pointless interactions, we can come up with a clean interface between dbscripts and the VCS (and I called that interface `db-abs`)
We want the PKGBUILDs repository to match the expected pacman repository action, so this information should always be sourced from the PKGBUILD.
And we want the pacman repository to match the PKGBUILDs repository, so this information should always be sourced from the .db?
There's a WIP rewrite here: https://wiki.archlinux.org/index.php/User:Bluewind/dbscripts-rewrite
Yes, I am aware of Florian's work (I did include the same URL in my cover letter).
Though I haven't spoken with Florian, one of my big goals with this is to make is work easier. I believe that if this patchset is applied, then the *only*[1] changes he will have to make are contained to the `db-abs` file, which defines just a handful of functions that provide a clean VCS-agnostic interface.
Which has e.g. get_file_from_pkgrepo which cat's the PKGBUILD in a VCS-agnostic manner. This is the direction I think we want to take,
That's not entirely different than the direction I took it: I included an abs_export function, which writes the files for a package (PKGBUILD and other) to a given directory, in a VCS-agnostic manner. I just named it abs_export, rather than get_dir_from_pkgrepo.
andIMHO also has a lot more code clarity than a mini awk program reading from pacman's technically undocumented database format (makepkg/PKGBUILD is quite well documented, I'm not sure we make any guarantees about the database format except that libalpm can read it).
For what it's worth, the added AWK program is very similar to the one already in `sourceballs`.
- Isolate SVN-interfacing code in to functions; making it clear what interfaces the PKGBUILD tracking needs to provide.
The dbscripts-rewrite aims to do this too, but in a very different way. I'd like to keep to those semantics in order to reduce the effort it takes to merge those changes.
Like I said, one of my goals is to reduce that effort.
Unless I'm looking at the wrong git repo (https://github.com/gbsf/dbscripts), it's already a lot of work; and several of the interactions haven't been considered. The check_pkgsvn and check_splitpkgs consistency checks are simply removed and replaced with "TODO" comments. The bit where db-remove looks at SVN to handle split packages (that I replaced with looking at the .db file) is similarly replaced with a "TODO" comment.
This patchset is intended to make those tricky bits trivial.
This does not touch the test suite, which is still firmly SVN-dependent.
[1]: Ok, I lied; they'll have to edit `db-abs` *and* the test suite.
On Fri, 22 Jun 2018 15:49:19 -0400, Luke Shumaker wrote:
On Fri, 22 Jun 2018 12:52:04 -0400, Eli Schwartz wrote:
andIMHO also has a lot more code clarity than a mini awk program reading from pacman's technically undocumented database format (makepkg/PKGBUILD is quite well documented, I'm not sure we make any guarantees about the database format except that libalpm can read it).
For what it's worth, the added AWK program is very similar to the one already in `sourceballs`.
Come to think of it, would you be happier with a Python program using pyalpm to parse it?
libalpm doesn't offer a convenient way to say "please parse this .db file", but I think that can be worked around by synthesizing a config that has a repo pointing to it via a "Server=file:///..." setting.
participants (2)
-
Eli Schwartz
-
Luke Shumaker