- Arch-projects - lists.archlinux.org

[arch-projects] [devtools] [PATCH] makechrootpkg: Skip PGP checks inside the chroot
by Alexander Shpilkin 13 Jul '18

13 Jul '18

As with the integrity checks, these have already been done in download_sources(), and the GPG keyring is unavailable inside the chroot anyway. --- makechrootpkg.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index d81be84..a26815e 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -304,7 +304,7 @@ move_products() { # }}} main() { - default_makepkg_args=(--syncdeps --noconfirm --log --holdver --skipinteg) + default_makepkg_args=(--syncdeps --noconfirm --log --holdver --skipinteg --skippgpcheck) makepkg_args=("${default_makepkg_args[@]}") keepbuilddir=false update_first=false --- 2.18.0

1 0

[arch-projects] [dbscripts] [PATCH v2] Don't parse .db files ourselves; use expac instead
by Luke Shumaker 10 Jul '18

10 Jul '18

From: Luke Shumaker <lukeshu(a)parabola.nu> In a patchset that I recently submitted, Eli was concerned that I was parsing .db files with bsdtar+awk, when the format of .db files isn't "public"; the only guarantees made about it are that libalpm can parse it. https://lists.archlinux.org/pipermail/arch-projects/2018-June/004932.html I wasn't too concerned, because `ftpdir-cleanup` and `sourceballs` already parse the .db files in the same way. Nonetheless, I think Eli is right: we shouldn't be parsing these files ourselves. So, add an `arch_expac` function that uses `expac` to parse the .db files. `expac` is particularly palatable as a new dependency, as it seems likely that it will be included in future releases of pacman. `expac` (like the underlying libalpm) doesn't offer an easy way to say "parse this DB file for me"; instead, we must construct a pacman.conf that has a repo pointing to that file with a `Server=file://...` line. --- v2: - use expac, not pyalpm cron-jobs/ftpdir-cleanup | 2 +- cron-jobs/sourceballs | 21 ++++++++------------- db-functions | 24 ++++++++++++++++++++++++ test/Dockerfile | 2 +- 4 files changed, 34 insertions(+), 15 deletions(-) diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index 9df5f99..e64b7e4 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -44,7 +44,7 @@ for repo in "${PKGREPOS[@]}"; do fi done | sort > "${WORKDIR}/repo-${repo}-${arch}" # get a list of package files defined in the repo db - bsdtar -xOf "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" | awk '/^%FILENAME%/{getline;print}' | sort > "${WORKDIR}/db-${repo}-${arch}" + arch_expac "$repo" "$arch" '%f' | sort > "${WORKDIR}/db-${repo}-${arch}" missing_pkgs=($(comm -13 "${WORKDIR}/repo-${repo}-${arch}" "${WORKDIR}/db-${repo}-${arch}")) if (( ${#missing_pkgs[@]} >= 1 )); then diff --git a/cron-jobs/sourceballs b/cron-jobs/sourceballs index 6be28ab..e24102b 100755 --- a/cron-jobs/sourceballs +++ b/cron-jobs/sourceballs @@ -19,24 +19,19 @@ renice +10 -p $$ > /dev/null # Create a readable file for each repo with the following format # <pkgbase|pkgname> <pkgver>-<pkgrel> <arch> <license>[ <license>] for repo in "${PKGREPOS[@]}"; do + # The %n+awk+cut bit is because we can't trust %e, as makepkg + # <5.1 didn't set pkgbase in .PKGINFO for non-split packages + # when when pkgbase==pkgname; so until all packages have been + # rebuilt with makepkg 5.1, we need to look at both pkgbase + # (%e) and pkgname (%n) to reliably get pkgbase. for arch in "${ARCHES[@]}"; do # Repo does not exist; skip it if [[ ! -f ${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT} ]]; then continue fi - bsdtar -xOf "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" \ - | awk '/^%NAME%/ { getline b }; - /^%BASE%/ { getline b }; - /^%VERSION%/ { getline v }; - /^%LICENSE%/,/^$/ { - if ( !/^%LICENSE%/ ) { l=l" "$0 } - }; - /^%ARCH%/ { - getline a; - printf "%s %s %s %s\n", b, v, a, l; - l=""; - }' - done | sort -u > "${WORKDIR}/db-${repo}" + arch_expac "$repo" "$arch" '%n %e %v %a %L' + done | awk -F' ' '$2 == "(null)" { $2 = $1 } { print }' | cut -d' ' -f2- | + sort -u > "${WORKDIR}/db-${repo}" done for repo in "${PKGREPOS[@]}"; do diff --git a/db-functions b/db-functions index 0491c22..cb55f87 100644 --- a/db-functions +++ b/db-functions @@ -436,6 +436,30 @@ set_repo_permission() { fi } +# usage: arch_expac repo arch expac_args... +arch_expac() { + local repo=$1 + local arch=$2 + local args=("${@:3}") + local dir="${WORKDIR}/expac-${repo}-${arch}" + + if ! [[ -d $dir ]]; then + mkdir -p -- "${dir}/root" + cat >"${dir}/pacman.conf" <<-EOT + [options] + RootDir = ${dir}/root + DBPath = ${dir}/root + Architecture = ${arch} + + [$repo] + Server = file://${FTP_BASE}/\$repo/os/\$arch + EOT + fi + + fakeroot pacman --config="${dir}/pacman.conf" -Sy >/dev/null + expac --config="${dir}/pacman.conf" --sync "${args[@]}" +} + arch_repo_modify() { local action=$1 local repo=$2 diff --git a/test/Dockerfile b/test/Dockerfile index 83c8449..cb40570 100644 --- a/test/Dockerfile +++ b/test/Dockerfile @@ -1,5 +1,5 @@ FROM archlinux/base -RUN pacman -Syu --noconfirm --needed sudo fakeroot awk subversion make kcov bash-bats gettext grep +RUN pacman -Syu --noconfirm --needed sudo fakeroot awk subversion make kcov bash-bats gettext grep expac RUN pacman-key --init RUN echo '%wheel ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/wheel RUN useradd -N -g users -G wheel -d /build -m tester -- 2.17.1

1 0

[arch-projects] [dbscripts] [PATCH] Don't parse .db files ourselves; use pyalpm instead
by Luke Shumaker 10 Jul '18

10 Jul '18

From: Luke Shumaker <lukeshu(a)parabola.nu> In a patchset that I recently submitted, Eli was concerned that I was parsing .db files with bsdtar+awk, when the format of .db files isn't "public"; the only guarantees made about it are that libalpm can parse it. https://lists.archlinux.org/pipermail/arch-projects/2018-June/004932.html I wasn't too concerned, because `ftpdir-cleanup` and `sourceballs` already parse the .db files in the same way. Nonetheless, I think Eli is right: we shouldn't be parsing these files ourselves. So, add a `dbquery` function that uses pyalpm to parse the .db files: - It takes as arguments Python 3 expressions; 1. one that that returns a bool deciding whether we want to print information on a package, and 2. another that returns the string to print for a package. Currently, all callers use "True" for the decider expression, as ftpdir-cleanup and sourceballs operate on *every* package. However, I'm including a way to filter packages because, I'm coming at this from the context that I want to parse .db files in other places too. - libalpm doesn't offer an easy way to say "parse this DB file for me"; instead, we must construct a configuration that has a syncdb pointing to that file, which we then have it sync in to a temporary directory. As a final note, when re-writing the bit of sourceballs to use dbquery instead of AWK, I realized that it does not correctly handle licenses that have a space in them (as of 2018-07-07 there are 67 packages in the Arch repos that have license containing a space). I did not fix this bug; I merely translated it from AWK to Python, as the program would also need to be adjusted elsewhere. --- cron-jobs/ftpdir-cleanup | 2 +- cron-jobs/sourceballs | 14 ++------------ db-functions | 25 +++++++++++++++++++++++++ test/Dockerfile | 2 +- 4 files changed, 29 insertions(+), 14 deletions(-) diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index 9df5f99..77e49c8 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -44,7 +44,7 @@ for repo in "${PKGREPOS[@]}"; do fi done | sort > "${WORKDIR}/repo-${repo}-${arch}" # get a list of package files defined in the repo db - bsdtar -xOf "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" | awk '/^%FILENAME%/{getline;print}' | sort > "${WORKDIR}/db-${repo}-${arch}" + dbquery "$repo" "$arch" True pkg.filename | sort > "${WORKDIR}/db-${repo}-${arch}" missing_pkgs=($(comm -13 "${WORKDIR}/repo-${repo}-${arch}" "${WORKDIR}/db-${repo}-${arch}")) if (( ${#missing_pkgs[@]} >= 1 )); then diff --git a/cron-jobs/sourceballs b/cron-jobs/sourceballs index 6be28ab..784b48b 100755 --- a/cron-jobs/sourceballs +++ b/cron-jobs/sourceballs @@ -24,18 +24,8 @@ for repo in "${PKGREPOS[@]}"; do if [[ ! -f ${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT} ]]; then continue fi - bsdtar -xOf "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" \ - | awk '/^%NAME%/ { getline b }; - /^%BASE%/ { getline b }; - /^%VERSION%/ { getline v }; - /^%LICENSE%/,/^$/ { - if ( !/^%LICENSE%/ ) { l=l" "$0 } - }; - /^%ARCH%/ { - getline a; - printf "%s %s %s %s\n", b, v, a, l; - l=""; - }' + dbquery "$repo" "$arch" True \ + 'f"{pkg.base or pkg.name} {pkg.version} {pkg.arch} {'\'' '\''.join(pkg.licenses)}"' done | sort -u > "${WORKDIR}/db-${repo}" done diff --git a/db-functions b/db-functions index 0491c22..f1d821a 100644 --- a/db-functions +++ b/db-functions @@ -294,6 +294,31 @@ getpkgfiles() { echo "${files[@]}" } +# usage: dbquery repo arch filter_expr output_expr +dbquery() { + local repo=$1 + local arch=$2 + local filter=$3 + local output=$4 + local dbfile="${FTP_BASE}/${repo}/os/${arch}/${repo}.db" + + python3 - "$dbfile" "$filter" "$output" <<-'EOT' + import os.path + import sys + import tempfile + import pyalpm + db_dir, db_file = os.path.split(os.path.abspath(sys.argv[1])) + with tempfile.TemporaryDirectory() as tmpdirname: + handle = pyalpm.Handle(tmpdirname, tmpdirname) + db = handle.register_syncdb(db_file[:-3], 0) + db.servers = ["file://{}".format(db_dir)] + db.update(False) + for pkg in db.search(".*"): + if eval(sys.argv[2], {}, {"pkg": pkg}): + print(eval(sys.argv[3], {}, {"pkg": pkg})) + EOT +} + check_pkgfile() { local pkgfile=$1 diff --git a/test/Dockerfile b/test/Dockerfile index 83c8449..0d01a75 100644 --- a/test/Dockerfile +++ b/test/Dockerfile @@ -1,5 +1,5 @@ FROM archlinux/base -RUN pacman -Syu --noconfirm --needed sudo fakeroot awk subversion make kcov bash-bats gettext grep +RUN pacman -Syu --noconfirm --needed sudo fakeroot awk subversion make kcov bash-bats gettext grep pyalpm RUN pacman-key --init RUN echo '%wheel ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/wheel RUN useradd -N -g users -G wheel -d /build -m tester -- 2.17.1

2 4

[arch-projects] [namcap] [PATCH] Fix testsuite to not hardcode the libalpm version.
by Eli Schwartz 06 Jul '18

06 Jul '18

Namcap already depends on pyelftools, which contains an elegant way to retrieve the soname information directly from libalpm.so (demonstrated in readelf.py). Using hardcoded strings means the testsuite periodically breaks (every time the external pacman project bumps their soname), for no good reason and at least a few bad ones. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- Namcap/tests/package/test_sodepends.py | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/Namcap/tests/package/test_sodepends.py b/Namcap/tests/package/test_sodepends.py index 3a3d7b6..f94b3e9 100644 --- a/Namcap/tests/package/test_sodepends.py +++ b/Namcap/tests/package/test_sodepends.py @@ -2,7 +2,7 @@ # # namcap tests - sodepends # Copyright (C) 2011 Rémy Oudompheng <remy(a)archlinux.org> -# +# # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or @@ -17,11 +17,23 @@ # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 # USA -# +# import os from Namcap.tests.makepkg import MakepkgTest import Namcap.rules.sodepends +from elftools.elf.dynamic import DynamicSection +from elftools.elf.elffile import ELFFile + +def get_soname(filename): + with open(filename, 'rb') as f: + alpm = ELFFile(f) + for section in alpm.iter_sections(): + if not isinstance(section, DynamicSection): + continue + for tag in section.iter_tags(): + if tag.entry.d_tag == 'DT_SONAME': + return tag.soname class SoDependsTest(MakepkgTest): pkgbuild = """ @@ -47,6 +59,7 @@ package() { def test_sodepends(self): "Package with missing pacman dependency" pkgfile = "__namcap_test_sodepends-1.0-1-%(arch)s.pkg.tar" % { "arch": self.arch } + alpm_filename = os.path.join('usr/lib', get_soname('/usr/lib/libalpm.so')) with open(os.path.join(self.tmpdir, "PKGBUILD"), "w") as f: f.write(self.pkgbuild) self.run_makepkg() @@ -56,15 +69,15 @@ package() { ) self.assertEqual(pkg.detected_deps['pacman'], [ ('libraries-needed %s %s', - (str(['usr/lib/libalpm.so.10']), str(["usr/bin/main"])) + (str([alpm_filename]), str(["usr/bin/main"])) )] ) e, w, i = Namcap.depends.analyze_depends(pkg) self.assertEqual(e, [ ('dependency-detected-not-included %s (%s)', - ('pacman', "libraries ['usr/lib/libalpm.so.10'] needed in files ['usr/bin/main']")) + ('pacman', "libraries ['%s'] needed in files ['usr/bin/main']" % alpm_filename)) + ]) self.assertEqual(w, []) # vim: set ts=4 sw=4 noet: - -- 2.18.0

1 0

[arch-projects] [namcap] [patch] Python 3.7 cleanups
by Jelle van der Waa 05 Jul '18

05 Jul '18

This patch series cleans up unused imports and resolves the deprecation warning showns with Python 3.7

2 4

[arch-projects] [namcap] [PATCH] tests: pacman 5.1
by Michael Straube 05 Jul '18

05 Jul '18

Signed-off-by: Michael Straube <michael.straube(a)posteo.de> --- Namcap/tests/package/test_sodepends.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Namcap/tests/package/test_sodepends.py b/Namcap/tests/package/test_sodepends.py index 3a3d7b6..4188c35 100644 --- a/Namcap/tests/package/test_sodepends.py +++ b/Namcap/tests/package/test_sodepends.py @@ -56,13 +56,13 @@ package() { ) self.assertEqual(pkg.detected_deps['pacman'], [ ('libraries-needed %s %s', - (str(['usr/lib/libalpm.so.10']), str(["usr/bin/main"])) + (str(['usr/lib/libalpm.so.11']), str(["usr/bin/main"])) )] ) e, w, i = Namcap.depends.analyze_depends(pkg) self.assertEqual(e, [ ('dependency-detected-not-included %s (%s)', - ('pacman', "libraries ['usr/lib/libalpm.so.10'] needed in files ['usr/bin/main']")) + ('pacman', "libraries ['usr/lib/libalpm.so.11'] needed in files ['usr/bin/main']")) ]) self.assertEqual(w, []) -- 2.18.0

2 1

[arch-projects] [dbscripts] [PATCH 1/2] Preliminary work to break out svn-specific code.
by Eli Schwartz 04 Jul '18

04 Jul '18

Introduce "db-functions-$VCS" which will eventually contain all VCS-specific code, and make this configurable in config. Move private arch_svn function and svn acl handling here, and introduce a new source_pkgbuild function to handle discovering PKGBUILDs from the configured VCS and sourcing them to extract metadata. The PKGBUILD is the only file we ever check out from version control, and only ever to scrape information from it, except for when we actually want to db-move a whole directory (which is by necessity considerably dependent on the VCS in use). source_pkgbuild is inspired by commits from the dbscripts rewrite, authored by Florian Pritz <bluewind(a)xinu.at> Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- config | 7 +++++-- db-functions | 50 +++++++++++++++--------------------------------- db-functions-svn | 27 ++++++++++++++++++++++++++ db-move | 5 +++-- db-remove | 5 ++--- db-update | 4 ++-- 6 files changed, 54 insertions(+), 44 deletions(-) create mode 100644 db-functions-svn diff --git a/config b/config index fd79b6ee..1cfe11f4 100644 --- a/config +++ b/config @@ -1,14 +1,17 @@ #!/hint/bash FTP_BASE="/srv/ftp" -SVNREPO='' -SVNUSER='' PKGREPOS=() PKGPOOL='' SRCPOOL='' TESTING_REPO='' STABLE_REPOS=() +# VCS backend +VCS=svn +SVNREPO='' +SVNUSER='' + CLEANUP_DESTDIR="/var/tmp" CLEANUP_DRYRUN=false # Time in days to keep moved packages diff --git a/db-functions b/db-functions index 0491c22d..3757b726 100644 --- a/db-functions +++ b/db-functions @@ -39,11 +39,6 @@ mv_acl() { # set up general environment WORKDIR=$(mktemp -dt "${0##*/}.XXXXXXXXXX") -if [[ -n ${SVNUSER} ]]; then - setfacl -m u:"${SVNUSER}":rwx "${WORKDIR}" - setfacl -m d:u:"${USER}":rwx "${WORKDIR}" - setfacl -m d:u:"${SVNUSER}":rwx "${WORKDIR}" -fi LOCKS=() REPO_MODIFIED=0 @@ -306,27 +301,24 @@ check_pkgfile() { [[ ${pkgfile##*/} = "${pkgname}-${pkgver}-${pkgarch}"* ]] } -check_pkgsvn() { +# Check that the package file is consistent with the PKGBUILD in version control +check_pkgvcs() { local pkgfile="${1}" + local repo="${2}" local _pkgbase="$(getpkgbase "${pkgfile}")" || return 1 local _pkgname="$(getpkgname "${pkgfile}")" || return 1 local _pkgver="$(getpkgver "${pkgfile}")" || return 1 local _pkgarch="$(getpkgarch "${pkgfile}")" || return 1 - local repo="${2}" in_array "${repo}" "${PKGREPOS[@]}" || return 1 - if [[ ! -f ${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase} ]]; then - mkdir -p "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}" - arch_svn export -q "${SVNREPO}/${_pkgbase}/repos/${repo}-${_pkgarch}/PKGBUILD" \ - "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}" >/dev/null || return 1 - fi + local vcsver vcsnames=() + read -rd'\n' vcsver vcsnames < <(source_pkgbuild "${_pkgbase}" "repos/${repo}-${_pkgarch}"; \ + get_full_version; echo "${pkgname[@]}") + read -ra vcsnames <<<"${vcsnames}" - local svnver="$(. "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}"; get_full_version)" - [[ "${svnver}" = "${_pkgver}" ]] || return 1 - - local svnnames=($(. "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}"; echo "${pkgname[@]}")) - in_array "${_pkgname}" "${svnnames[@]}" || return 1 + [[ "${vcsver}" = "${_pkgver}" ]] || return 1 + in_array "${_pkgname}" "${vcsnames[@]}" || return 1 return 0 } @@ -337,7 +329,7 @@ check_splitpkgs() { local pkgfiles=("${@}") local pkgfile local pkgdir - local svnname + local vcsname mkdir -p "${WORKDIR}/check_splitpkgs/" pushd "${WORKDIR}/check_splitpkgs" >/dev/null @@ -350,22 +342,16 @@ check_splitpkgs() { mkdir -p "${repo}/${_pkgarch}/${_pkgbase}" echo "${_pkgname}" >> "${repo}/${_pkgarch}/${_pkgbase}/staging" - if [[ ! -f ${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase} ]]; then - mkdir -p "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}" - arch_svn export -q "${SVNREPO}/${_pkgbase}/repos/${repo}-${_pkgarch}/PKGBUILD" \ - "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}" >/dev/null || return 1 - fi - - local svnnames=($(. "${WORKDIR}/pkgbuilds/${repo}-${_pkgarch}/${_pkgbase}"; echo "${pkgname[@]}")) - printf '%s\n' "${svnnames[@]}" >> "${repo}/${_pkgarch}/${_pkgbase}/svn" + local vcsnames=($(source_pkgbuild "${_pkgbase}" "repos/${repo}-${_pkgarch}"; echo "${pkgname[@]}")) + printf '%s\n' "${vcsnames[@]}" >> "${repo}/${_pkgarch}/${_pkgbase}/vcs" done popd >/dev/null for pkgdir in "${WORKDIR}/check_splitpkgs/${repo}"/*/*; do [[ ! -d ${pkgdir} ]] && continue sort -u "${pkgdir}/staging" -o "${pkgdir}/staging" - sort -u "${pkgdir}/svn" -o "${pkgdir}/svn" - if [[ ! -z "$(comm -13 "${pkgdir}/staging" "${pkgdir}/svn")" ]]; then + sort -u "${pkgdir}/vcs" -o "${pkgdir}/vcs" + if [[ ! -z "$(comm -13 "${pkgdir}/staging" "${pkgdir}/vcs")" ]]; then return 1 fi done @@ -458,10 +444,4 @@ arch_repo_modify() { REPO_MODIFIED=1 } -arch_svn() { - if [[ -z ${SVNUSER} ]]; then - /usr/bin/svn "${@}" - else - sudo -u "${SVNUSER}" -- /usr/bin/svn --username "${USER}" "${@}" - fi -} +. "$(dirname "$(readlink -e "${BASH_SOURCE[0]}")")/db-functions-${VCS}" diff --git a/db-functions-svn b/db-functions-svn new file mode 100644 index 00000000..aa7952d8 --- /dev/null +++ b/db-functions-svn @@ -0,0 +1,27 @@ +#!/hint/bash + +if [[ -n ${SVNUSER} ]]; then + setfacl -m u:"${SVNUSER}":rwx "${WORKDIR}" + setfacl -m d:u:"${USER}":rwx "${WORKDIR}" + setfacl -m d:u:"${SVNUSER}":rwx "${WORKDIR}" +fi + +arch_svn() { + if [[ -z ${SVNUSER} ]]; then + /usr/bin/svn "${@}" + else + sudo -u "${SVNUSER}" -- /usr/bin/svn --username "${USER}" "${@}" + fi +} + +# source_pkgbuild pkgbase tag +# +# Source the PKGBUILD from the package's git/svn/whatever repo. +# Depending on how the VCS is used the tag might be "trunk" or "repos/$repo-$arch" +# or the full package version (epoch:pkgver-pkgrel) or any other recognized tag. +source_pkgbuild() { + local pkgbase="$1" + local tag="$2" + + . <(arch_svn cat "${SVNREPO}/${pkgbase}/${tag}/PKGBUILD" 2>/dev/null || echo "false") +} diff --git a/db-move b/db-move index b6448898..2a1da68d 100755 --- a/db-move +++ b/db-move @@ -77,8 +77,9 @@ for pkgbase in "${args[@]:2}"; do tarches=("${pkgarch}") fi msg2 "%s (%s)" "$pkgbase" "${tarches[*]}" - pkgnames=($(. "${svnrepo_from}/PKGBUILD"; echo "${pkgname[@]}")) - pkgver=$(. "${svnrepo_from}/PKGBUILD"; get_full_version) + read -rd'\n' pkgver pkgnames < <(source_pkgbuild "${pkgbase}" "repos/${repo_from}-${pkgarch}"; \ + get_full_version; echo "${pkgname[@]}") + read -ra pkgnames <<<"$pkgnames" if [[ -d ${svnrepo_to} ]]; then for file in $(arch_svn ls "${svnrepo_to}"); do diff --git a/db-remove b/db-remove index ac9a1688..b794edc2 100755 --- a/db-remove +++ b/db-remove @@ -32,10 +32,9 @@ done remove_pkgs=() for pkgbase in "${pkgbases[@]}"; do msg "Removing %s from [%s]..." "$pkgbase" "$repo" - arch_svn checkout -q "${SVNREPO}/${pkgbase}" "${WORKDIR}/svn/${pkgbase}" >/dev/null - if [[ -d ${WORKDIR}/svn/$pkgbase/repos/$svnrepo ]]; then - remove_pkgs+=($(. "${WORKDIR}/svn/$pkgbase/repos/$svnrepo/PKGBUILD"; echo ${pkgname[@]})) + if remove_pkgs+=($(. source_pkgbuild "${pkgbase}" "repos/${svnrepo}" && echo ${pkgname[@]})); then + arch_svn checkout -q "${SVNREPO}/${pkgbase}" "${WORKDIR}/svn/${pkgbase}" >/dev/null arch_svn rm --force -q "${WORKDIR}/svn/$pkgbase/repos/$svnrepo" arch_svn commit -q "${WORKDIR}/svn/$pkgbase" -m "${0##*/}: $pkgbase removed by $(id -un)" else diff --git a/db-update b/db-update index 8eb27dfe..313fb999 100755 --- a/db-update +++ b/db-update @@ -46,8 +46,8 @@ for repo in "${repos[@]}"; do if ! pacman-key -v "${pkg}.sig" >/dev/null 2>&1; then die "Package %s does not have a valid signature" "$repo/${pkg##*/}" fi - if ! check_pkgsvn "${pkg}" "${repo}"; then - die "Package %s is not consistent with svn repository" "$repo/${pkg##*/}" + if ! check_pkgvcs "${pkg}" "${repo}"; then + die "Package %s is not consistent with %s repository" "$repo/${pkg##*/}" "${VCS}" fi if ! check_pkgrepos "${pkg}"; then die "Package %s already exists in another repository" "$repo/${pkg##*/}" -- 2.18.0

1 1

[arch-projects] [dbscripts] [GIT] Official repo DB scripts branch master updated. 20180603
by eschwartz＠archlinux.org 04 Jul '18

04 Jul '18

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "Official repo DB scripts". The branch, master has been updated via 08995618eb606523b7d05ac987b1243981290839 (commit) via 9de966ba08e4d697404cc630ffa514ba18b1bace (commit) via 01712670c40df875c6ac6e1d5a88063fab0b020f (commit) from 05dd9be0db06d7f7cea0eff9f9f6f1993eb6d463 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 08995618eb606523b7d05ac987b1243981290839 Author: Luke Shumaker <lukeshu(a)parabola.nu> Date: Wed Apr 4 16:22:20 2018 -0400 test: Fix typoed variables - ftpdir-cleanup: ${pkg[0]} -> ${pkgs[0]} This mistake was introduced in 7628525156110022fa70ad91e4bc13ee8a3cceb0 - test: common.bash: $pkgarches[@] -> ${pkgarches[@]} This mistake was introduced in 7628525156110022fa70ad91e4bc13ee8a3cceb0 commit 9de966ba08e4d697404cc630ffa514ba18b1bace Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Wed May 23 11:11:05 2018 -0400 hoist $tarch handling above $pkgname, since it is more efficient commit 01712670c40df875c6ac6e1d5a88063fab0b020f Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Wed May 23 10:53:45 2018 -0400 db-move: Don't store filepaths as a string with whitespace splitting Use arrays via nameref, since makepkg does not support multidimensional arrays, and assigning to/retrieving from an array variable with an unknown name requires this. Requires bash 4.3, requires that architectures never contain chars that are invalid in a variable name -- which makepkg explicitly requires in order to support architecture-dependent sources/dependencies, and also incorporates into linting checks which forbid anything but [[:alnum:]_] so that's all good. ----------------------------------------------------------------------- Summary of changes: db-move | 24 +++++++++++++++--------- test/cases/ftpdir-cleanup.bats | 2 +- test/lib/common.bash | 2 +- 3 files changed, 17 insertions(+), 11 deletions(-) hooks/post-receive -- Official repo DB scripts

1 0

[arch-projects] [dbscripts] [GIT] Official repo DB scripts annotated tag 20180603 created. 20180603
by eschwartz＠archlinux.org 04 Jul '18

04 Jul '18

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "Official repo DB scripts". The annotated tag, 20180603 has been created at b9e1a487c7b89072904a5d6aabd8b827693e440f (tag) tagging 08995618eb606523b7d05ac987b1243981290839 (commit) replaces 20131102 tagged by Eli Schwartz on Tue Jul 3 22:05:48 2018 -0400 - Log ----------------------------------------------------------------- Lots of fixes which presumably make us handle data more safely -----BEGIN PGP SIGNATURE----- iQJMBAABCgA2FiEEYEETBMCdNmKDQO7/zrFn77VyK9YFAls8LNEYHGVzY2h3YXJ0 ekBhcmNobGludXgub3JnAAoJEM6xZ++1civWMiIP/j3vBx4Ei2A7AvSn6/BDE54p CpPTe5GD4DBp5x/X1Ja0aMpSvsMLs5dguesXglxBwZVPWNuDYcf/flP05V0uwK38 /0+b1dy/2Sn+tBh0fe/BxdiXfF6thH8axi8+92yonzEFhWz+TXRyg7IjGLALNaLu DkWgC4YuLan+uuCqXBbQc7h0z+ZCE95fi4ymVrgA83VjFUp4Nkc46taQFgmN4lj7 mTCr962+9js2wm6shfeqbhSbYmj8KBXG9zdelE0uWgcpFz+Q9LgddDvqXFA4J9lW ku+2AV9P+b+Zy838iduLDgshO2ZCd25tgSo87YngSLLxKfrpbf1HKcxW2ggZsWtk xoO4AX3ElElSiFZL8gDUL9AQNpskfAck1Hrup71xKg8d2B3OrfAdrPef/8+cPLAR 5CQ+7a+NjmAcUbLo0fG5eAntgEM+foRqdRnnw2KgwvMwTj90BXi4soPPpf8CQ1gG o0P/gnp0/f8IrzCd/PPOTRFnLfE/sGJFO65R2hVmlk3Y+xS7V+982sQirXj22wbF 3nTI1+eGrCLONhxGFM6moxQs/mF2FLQUJDCTC3jQdcv4d8/cKocKAWfSOuQHK8pH ij9uu1qqKaBmW0VXu5pOpssg4hPZfFCWXtVWYXfKoDS62//08o5nVQPyq/iBMwOZ C2MSqCydTte4YATTwkfH =ihXJ -----END PGP SIGNATURE----- Dave Reisner (1): disallow packages without a valid PACKAGER Eli Schwartz (17): Use return codes properly when checking for failed commands. Update messages to make full use of printf formatters ARCHES is an array, do not attempt to call it as a string Use modern bash to append to strings/arrays. Use more bashisms db-functions: deduplicate some repeated logic Use even more bashisms. Fix overloading PKGEXT to mean two things. db-update: replace external find command with bash globbing ftpdir-cleanup,sourceballs: replace external find command with bash globbing Globally set $PKGEXT to a bash extended glob representing valid choices. db-functions: unify the repo-add/repo-remove caller fixup! Update messages to make fuller use of printf formatters pacman 5.1 compatibility test: BUILDDIR must be owned by build user db-move: Don't store filepaths as a string with whitespace splitting hoist $tarch handling above $pkgname, since it is more efficient Florian Pritz (1): Remove i686 from config Jelle van der Waa (1): Disable integrity check for i686 Johannes Löthberg (1): config: add GPL3/LGPL3 to ALLOWED_LICENSES Levente Polyak (1): disallow packages not built in a chroot Luke Shumaker (18): test: db-update: verify that PKGEXT(S) is treated as a glob Update messages to make fuller use of printf formatters test: common.bash:__getCheckSum: Don't rely on IFS test: Fixup glob matching Export TMPDIR, and use mktemp -t instead of making it part of the template Normalize to tab indent Add "#!/hint/bash" to the beginning of several files Don't use `grep -q` when operating on piped stdin Remove uses of the "v=true; if $v ..." anti-pattern Fixups near unquoted variables Clean up printf-formatters for user messages README.md: fix spelling mistake: "overriden"->"overridden" README.md: Write an overview of the programs included test/: Verify that db-update won't release duplicate packages in the same transaction Fixup quoting around variables, especially arrays Consistently use "$(dirname "$(readlink -e "$0")")" devlist-mailer: Make LIST and FROM configurable test: Fix typoed variables Pierre Schmitz (52): cron-jobs: Fix issues with empty repositories Remove separate creation of file databases Update bundled makepkg.conf Run tests using docker Add travis configuration Add editor configuration Compile tests into one file to reduce time spent on setup Simplify test setup Use a minimal travis environment testSuite is a simple file Fix package test Fix test package dependency Add ability to generate test coverage Replace bash script with Makefile Make test suite independent from code under test db-repo-add: Fix adding of 'any' packages db-repo-remove: Test removal of 'any' packages db-update: Check that packages with invalid signatures wont get added Signing packages is no longer optional ftpdir-cleanup: Test removal of old packages db-update: Disallow invalid package names db-update: Test if package cannot be added if svn is inconsistent db-update: Test package validation Make coverage folder writeable to the test user Avoid replacing the temporary dir by accident use makepkg library instead of local function copies Move package build and update functions to common library Remove test for file database as this feature was moved to pacman/repo-add a long time ago Mark private test functions Set PATH outside the test environment Switch from shuint2 to bats testing framework Re-add docker image dependency Disable docker terminal to avoid issues with the CI system Cache already built packages Extend check functions to handle 'any' packages as well Move common checks to functions Remove useless code Use tmpfs as build dir Allways build packages for all supported architectures Remove dependency on devtools Reduce test dependencies Correctly use checkRemovedPackage Move package and db checks from test cases to library Move test cases and fixtures to more meaningful directories Document test cases in a more readable format Add initial README file Add travis build status to README.md Use archlinux/base image Install required packages for testing PR-2: Allow i686 for tests PR-2: Default to x86_64 for check_packages Add usage tests ----------------------------------------------------------------------- hooks/post-receive -- Official repo DB scripts

1 0

[arch-projects] [dbscripts] [PATCH 1/2] db-move: Don't store filepaths as a string with whitespace splitting
by Eli Schwartz 02 Jul '18

02 Jul '18

Use arrays via nameref, since makepkg does not support multidimensional arrays, and assigning to/retrieving from an array variable with an unknown name requires this. Requires bash 4.3, requires that architectures never contain chars that are invalid in a variable name -- which makepkg explicitly requires in order to support architecture-dependent sources/dependencies, and also incorporates into linting checks which forbid anything but [[:alnum:]_] so that's all good. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- db-move | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/db-move b/db-move index 63e5c14c..c4da5bf9 100755 --- a/db-move +++ b/db-move @@ -60,8 +60,10 @@ done msg "Moving packages from [%s] to [%s]..." "$repo_from" "$repo_to" -declare -A add_pkgs -declare -A remove_pkgs +for arch in "${ARCHES[@]}"; do + declare -a add_pkgs_$arch + declare -a remove_pkgs_$arch +done for pkgbase in "${args[@]:2}"; do tag_list="" for pkgarch in "${ARCHES[@]}" 'any'; do @@ -95,6 +97,8 @@ for pkgbase in "${args[@]:2}"; do for pkgname in "${pkgnames[@]}"; do for tarch in "${tarches[@]}"; do + declare -n add_pkgs="add_pkgs_${tarch}" + declare -n remove_pkgs="remove_pkgs_${tarch}" pkgpath=$(getpkgfile "${ftppath_from}/${tarch}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXTS}) pkgfile="${pkgpath##*/}" @@ -102,8 +106,8 @@ for pkgbase in "${args[@]:2}"; do if [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile}.sig ]]; then ln -s "../../../${PKGPOOL}/${pkgfile}.sig" "${ftppath_to}/${tarch}/" fi - add_pkgs[${tarch}]+="${FTP_BASE}/${PKGPOOL}/${pkgfile} " - remove_pkgs[${tarch}]+="${pkgname} " + add_pkgs+=("${FTP_BASE}/${PKGPOOL}/${pkgfile}") + remove_pkgs+=("${pkgname}") done done fi @@ -113,9 +117,11 @@ for pkgbase in "${args[@]:2}"; do done for tarch in "${ARCHES[@]}"; do - if [[ -n ${add_pkgs[${tarch}]} ]]; then - arch_repo_modify add "${repo_to}" "${tarch}" ${add_pkgs[${tarch}]} - arch_repo_modify remove "${repo_from}" "${tarch}" ${remove_pkgs[${tarch}]} + declare -n add_pkgs="add_pkgs_${tarch}" + declare -n remove_pkgs="remove_pkgs_${tarch}" + if [[ -n ${add_pkgs[@]} ]]; then + arch_repo_modify add "${repo_to}" "${tarch}" "${add_pkgs[@]}" + arch_repo_modify remove "${repo_from}" "${tarch}" "${remove_pkgs[@]}" fi done -- 2.18.0

1 1