February 2018 - Arch-projects - lists.archlinux.org

[arch-projects] [mkinitcpio] [PATCH] --rsyncable option
by Charles Duffy 16 Dec '18

16 Dec '18

At present, mkinitcpio outputs are not particularly stable: Generating two consecutive initramfs images from the same inputs can generate archives which are quite entirely unlike each other from the perspective of binary diff tools (rsync, casync, rdiff-backup, xdelta, etc). Making some adjustments to compression and archival can greatly reduce the size of this delta. A ticket with a proposed patch (adding a --rsyncable option to mkinitcpio) has been opened at https://bugs.archlinux.org/task/56128 -- however, as the advice there has been to consult this mailing list, it may no longer be open as of this writing. :) The change consists of the following: - Sort the list of files (in the C locale) before passing it to cpio. - When --rsyncable is passed, set all datestamps to the UNIX epoch before generating the cpio archive. - When --rsyncable is passed, pass the --reproducible option to cpio (this creates a dependency on GNU cpio 2.12, prevents device numbers from being included in the archive, and renumbers inode numbers to be consistent when file ordering is unchanged). - When --rsyncable is passed, pass the --rsyncable and --no-name options to gzip (the former tells it to reset the compression table on occasion, sacrificing compression ratio to prevent the effect from changes from cascading throughout too much of the stream; the latter prevents name and timestamp data from being stored in the gzip header itself). A version of the patch squashing together the two commits already present on the ticket is attached.

1 1

[arch-projects] [devtools] [PATCH v2] makechrootpkg: respect GNUPGHOME
by Emiel Wiedijk 01 Jun '18

01 Jun '18

Previously, makechrootpkg hardcoded ~/.gnupg. Therefore, if a user uses a custom GPG home directory, the siganture checking would fail. Now makechrootpkg uses $GNUPGHOME, with a fallback to ~/.gnupg. Signed-off-by: Emiel Wiedijk <me(a)aimileus.nl> --- makechrootpkg.in | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index afcd121..653847f 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -252,7 +252,8 @@ download_sources() { chmod 1777 "$builddir" # Ensure sources are downloaded - sudo -u "$makepkg_user" env SRCDEST="$SRCDEST" BUILDDIR="$builddir" \ + sudo -u "$makepkg_user" --preserve-env=GNUPGHOME \ + env SRCDEST="$SRCDEST" BUILDDIR="$builddir" \ makepkg --config="$copydir/etc/makepkg.conf" --verifysource -o || die "Could not download sources." @@ -341,7 +342,7 @@ main() { [[ -n $makepkg_user && -z $(id -u "$makepkg_user") ]] && die 'Invalid makepkg user.' makepkg_user=${makepkg_user:-${SUDO_USER:-$USER}} - check_root SOURCE_DATE_EPOCH + check_root SOURCE_DATE_EPOCH,GNUPGHOME # Canonicalize chrootdir, getting rid of trailing / chrootdir=$(readlink -e "$passeddir") -- 2.16.2

2 1

[arch-projects] [PATCH] [namcap] Treat all non-tar files as PKGBUILD files, regardless of filename
by Luke Shumaker 16 Apr '18

16 Apr '18

From: Luke Shumaker <lukeshu(a)parabola.nu> https://bugs.archlinux.org/task/57193 If the user mistakenly passes in some other (non-package, non-PKGBUILD) file, instead of refusing to process it (saying "Error: Cannot process ${filename}"), it will now try to parse it as a PKGBUILD, probably encounter and print at least one error, but finally say "Error: ${filename} is not a valid PKGBUILD". This is actually probably *more* helpful to a confused user who passed in the wrong file. --- namcap.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/namcap.py b/namcap.py index b62a2fa..fd62d3c 100755 --- a/namcap.py +++ b/namcap.py @@ -248,9 +248,7 @@ for package in packages: if os.path.isfile(package) and tarfile.is_tarfile(package): process_realpackage(package, active_modules) - elif package.endswith('PKGBUILD'): + else: process_pkgbuild(package, active_modules) - else: - print("Error: Cannot process %s" % package) # vim: set ts=4 sw=4 noet: -- 2.15.1

3 3

[arch-projects] [dbscripts] [PATCH] db-functions: unify the repo-add/repo-remove caller
by Eli Schwartz 14 Mar '18

14 Mar '18

Use less duplicated logic, opening the way to more elegantly change the interface for the repo-add/repo-remove scripts which are themselves symlinked together and mostly have the same interface. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- db-functions | 34 ++++++++++++---------------------- db-move | 4 ++-- db-remove | 2 +- db-repo-add | 2 +- db-repo-remove | 2 +- db-update | 2 +- 6 files changed, 18 insertions(+), 28 deletions(-) diff --git a/db-functions b/db-functions index 8b71cae..653f130 100644 --- a/db-functions +++ b/db-functions @@ -442,34 +442,24 @@ set_repo_permission() { fi } -arch_repo_add() { - local repo=$1 - local arch=$2 - local pkgs=(${@:3}) - - # package files might be relative to repo dir - pushd "${FTP_BASE}/${repo}/os/${arch}" >/dev/null - /usr/bin/repo-add -q "${repo}${DBEXT}" ${pkgs[@]} \ - || error "repo-add ${repo}${DBEXT} ${pkgs[@]}" - popd >/dev/null - set_repo_permission "${repo}" "${arch}" - - REPO_MODIFIED=1 -} - -arch_repo_remove() { - local repo=$1 - local arch=$2 - local pkgs=(${@:3}) +arch_repo_modify() { + local action=$1 + local repo=$2 + local arch=$3 + local pkgs=("${@:4}") local dbfile="${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" - if [[ ! -f ${dbfile} ]]; then + if [[ ${action} = remove && ! -f ${dbfile} ]]; then error "No database found at '%s'" "$dbfile" return 1 fi - /usr/bin/repo-remove -q "${dbfile}" ${pkgs[@]} \ - || error "repo-remove ${dbfile} ${pkgs[@]}" + + # package files for repo-add might be relative to repo dir + pushd "${dbfile%/*}" >/dev/null + /usr/bin/repo-${action} -q "${dbfile}" ${pkgs[@]} \ + || error "repo-${action} ${dbfile} ${pkgs[@]}" set_repo_permission "${repo}" "${arch}" + popd >/dev/null REPO_MODIFIED=1 } diff --git a/db-move b/db-move index fb7ebac..e9c9fa7 100755 --- a/db-move +++ b/db-move @@ -114,8 +114,8 @@ done for tarch in ${ARCHES[@]}; do if [[ -n ${add_pkgs[${tarch}]} ]]; then - arch_repo_add "${repo_to}" "${tarch}" ${add_pkgs[${tarch}]} - arch_repo_remove "${repo_from}" "${tarch}" ${remove_pkgs[${tarch}]} + arch_repo_modify add "${repo_to}" "${tarch}" ${add_pkgs[${tarch}]} + arch_repo_modify remove "${repo_from}" "${tarch}" ${remove_pkgs[${tarch}]} fi done diff --git a/db-remove b/db-remove index 70502bc..08a95f9 100755 --- a/db-remove +++ b/db-remove @@ -47,6 +47,6 @@ for pkgbase in ${pkgbases[@]}; do done for tarch in ${tarches[@]}; do - arch_repo_remove "${repo}" "${tarch}" ${remove_pkgs[@]} + arch_repo_modify remove "${repo}" "${tarch}" ${remove_pkgs[@]} repo_unlock $repo $tarch done diff --git a/db-repo-add b/db-repo-add index d7be302..4b16f7e 100755 --- a/db-repo-add +++ b/db-repo-add @@ -36,6 +36,6 @@ for tarch in ${tarches[@]}; do msg "Adding %s to [%s]..." "$pkgfile" "$repo" fi done - arch_repo_add "${repo}" "${tarch}" ${pkgfiles[@]} + arch_repo_modify add "${repo}" "${tarch}" ${pkgfiles[@]} repo_unlock $repo $tarch done diff --git a/db-repo-remove b/db-repo-remove index 32d167e..3077786 100755 --- a/db-repo-remove +++ b/db-repo-remove @@ -32,6 +32,6 @@ for tarch in ${tarches[@]}; do for pkgname in ${pkgnames[@]}; do msg "Removing %s from [%s]..." "$pkgname" "$repo" done - arch_repo_remove "${repo}" "${tarch}" ${pkgnames[@]} + arch_repo_modify remove "${repo}" "${tarch}" ${pkgnames[@]} repo_unlock $repo $tarch done diff --git a/db-update b/db-update index 4e17184..e8ae08d 100755 --- a/db-update +++ b/db-update @@ -94,7 +94,7 @@ for repo in ${repos[@]}; do add_pkgs+=("${pkgfile}") done if (( ${#add_pkgs[@]} >= 1 )); then - arch_repo_add "${repo}" "${pkgarch}" ${add_pkgs[@]} + arch_repo_modify add "${repo}" "${pkgarch}" ${add_pkgs[@]} fi done done -- 2.16.2

2 1

Re: [arch-projects] [devtools] makechrootpkg: respect GNUPGHOME
by Emiel Wiedijk 27 Feb '18

27 Feb '18

On Tue, Feb 27, 2018 at 03:44:07PM +0100, Eli Schwartz via arch-projects wrote: > On 02/27/2018 05:41 AM, Emiel Wiedijk via arch-projects wrote: > > Correct, but makepkg --verifysource is run with sudo -u $myuser, and sudo > > resets the environment. And the code that copies ~/.gnupg to the chroot > > apparantly hasn't been removed yet (as of 38c7a391b043547b946a99731a56a233458ba7a2). > > I just assumed (apparantly wrongly) that it was for GnuPG related tasks in the > > PKGBUILD, and adjusted the code to copy the correct directory. > > My point is that there is really no point in trying to preserve it in > the chroot, since that section is dead code to begin with, and if you do > anyways then your patch may clash with other pending patches. So of the > three changes your patch made, you should probably only make the second > and third. > > I don't blame you for actually thinking dead code did something. :D > > -- > Eli Schwartz > Bug Wrangler and Trusted User > Fair enough, I sent an updated patch :-). Emiel Wiedijk

1 0

[arch-projects] [devtools] makechrootpkg: respect GNUPGHOME
by Emiel Wiedijk 27 Feb '18

27 Feb '18

Previously, makechrootpkg hardcoded ~/.gnupg. Therefore, if a user uses a custom GPG home directory, the siganture checking would fail. Now makechrootpkg uses $GNUPGHOME, with a fallback to ~/.gnupg. Signed-off-by: Emiel Wiedijk <me(a)aimileus.nl> --- makechrootpkg.in | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index afcd121..5a79dc0 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -182,9 +182,10 @@ prepare_chroot() { $install -d "$copydir"/{build,build/.gnupg,startdir,{pkg,srcpkg,src,log}dest} - for x in .gnupg/pubring.{kbx,gpg}; do - [[ -r $USER_HOME/$x ]] || continue - $install -m 644 "$USER_HOME/$x" "$copydir/build/$x" + for x in pubring.{kbx,gpg}; do + local pubring="${GNUPGHOME:-$USER_HOME/.gnupg}/$x" + [[ -r "$pubring" ]] || continue + $install -m 644 "$pubring" "$copydir/build/.gnupg/$x" done sed -e '/^MAKEFLAGS=/d' -e '/^PACKAGER=/d' -i "$copydir/etc/makepkg.conf" @@ -252,7 +253,8 @@ download_sources() { chmod 1777 "$builddir" # Ensure sources are downloaded - sudo -u "$makepkg_user" env SRCDEST="$SRCDEST" BUILDDIR="$builddir" \ + sudo -u "$makepkg_user" --preserve-env=GNUPGHOME \ + env SRCDEST="$SRCDEST" BUILDDIR="$builddir" \ makepkg --config="$copydir/etc/makepkg.conf" --verifysource -o || die "Could not download sources." @@ -341,7 +343,7 @@ main() { [[ -n $makepkg_user && -z $(id -u "$makepkg_user") ]] && die 'Invalid makepkg user.' makepkg_user=${makepkg_user:-${SUDO_USER:-$USER}} - check_root SOURCE_DATE_EPOCH + check_root SOURCE_DATE_EPOCH,GNUPGHOME # Canonicalize chrootdir, getting rid of trailing / chrootdir=$(readlink -e "$passeddir") -- 2.16.2

2 3

[arch-projects] [dbscripts] [PATCH v2 0/3] Misc touchup
by Luke Shumaker 27 Feb '18

27 Feb '18

From: Luke Shumaker <lukeshu(a)parabola.nu> These are a few minor unrelated touch-ups that I noticed when doing the PKGEXT work. There's no real theme to them. v2: - Use Bash 4.4 feature to avoid introducing temporary variables in the printf commit - Mention splitting declaring and setting the result variable in the __getCheckSum commit message. Luke Shumaker (3): test: common.bash:__getCheckSum: Don't rely on IFS test: Fixup glob matching Update messages to make fuller use of printf formatters cron-jobs/ftpdir-cleanup | 8 ++++---- cron-jobs/integrity-check | 2 +- cron-jobs/sourceballs | 8 ++++---- db-functions | 4 ++-- db-move | 4 ++-- db-remove | 2 +- db-repo-add | 2 +- db-repo-remove | 2 +- db-update | 2 +- test/cases/ftpdir-cleanup.bats | 4 ++-- test/cases/sourceballs.bats | 4 ++-- test/lib/common.bash | 17 ++++++++++++++--- testing2x | 2 +- 13 files changed, 36 insertions(+), 25 deletions(-) -- 2.16.1

2 7

[arch-projects] [dbscripts] [PATCH 0/3] Misc touchup
by Luke Shumaker 23 Feb '18

23 Feb '18

From: Luke Shumaker <lukeshu(a)parabola.nu> These are a few minor unrelated touch-ups that I noticed when doing the PKGEXT work. There's no real theme to them. Luke Shumaker (3): test: common.bash:__getCheckSum: Don't rely on IFS test: Fixup glob matching Update messages to make fuller use of printf formatters cron-jobs/ftpdir-cleanup | 8 ++++---- cron-jobs/integrity-check | 2 +- cron-jobs/sourceballs | 8 ++++---- db-functions | 8 ++++++-- db-move | 4 ++-- db-remove | 2 +- db-repo-add | 2 +- db-repo-remove | 2 +- db-update | 2 +- test/cases/ftpdir-cleanup.bats | 4 ++-- test/cases/sourceballs.bats | 4 ++-- test/lib/common.bash | 17 ++++++++++++++--- testing2x | 2 +- 13 files changed, 40 insertions(+), 25 deletions(-) -- Happy hacking, ~ Luke Shumaker

2 13

[arch-projects] [dbscripts] [GIT] Official repo DB scripts branch master updated. 20131102-69-ge53cad6
by eschwartz＠archlinux.org 22 Feb '18

22 Feb '18

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "Official repo DB scripts". The branch, master has been updated via e53cad6e4a8284165c6d0b2c7c86f6c077be693b (commit) via c82f4372440b42f879344f105e12fb358c4c0ccf (commit) via 3e01ba9a82234c1d78a14be095c18a626591ad71 (commit) via 8757c459089c0db12e57db66561f640f657e34ac (commit) via 07e6a91da1331ae61b667d878092c5b03aa49ced (commit) via b45650c3e3870b975620dd89bb6aa77585c8ff02 (commit) from 134aff74d16331231ed94f34d8417d1325d4600f (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit e53cad6e4a8284165c6d0b2c7c86f6c077be693b Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Thu Feb 15 22:34:36 2018 -0500 Globally set $PKGEXT to a bash extended glob representing valid choices. The current glob `*.pkg.tar.?z` is both less restrictive and more restrictive than makepkg, as it accepts any valid unicode character. To be more exact, it's almost completely orthogonal to the one in makepkg. makepkg only accepts .tar.gz, .tar.bz2, .tar.xz, .tar.lzo, .tar.lrz, and .tar.Z and most of those fail to match against a two-char compression type. dbscripts accepts .pkg.tar.💩z which incidentally is what I think of cherry-picking xz and gz as supported methods. Since this can be anything makepkg.conf accepts, it needs to be able to match all that, unless we decide to perform additional restrictions in which case we should still explicitly list each allowed extension. Using bash extended globbing allows us to do this relatively painlessly. Document the fact that this has *always* been some sort of glob, and update the two cases where this was (not!) being evaluated by bash [[ ... ]], to use a not-elegant-at-all proxy function is_globfile() to evaluate globs *before* testing if they exist. commit c82f4372440b42f879344f105e12fb358c4c0ccf Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Thu Feb 15 22:32:04 2018 -0500 ftpdir-cleanup,sourceballs: replace external find command with bash globbing This fully removes the use of find from the codebase, leads to a micro-optimization in a couple cases, and ensures that $PKGEXT is consistently treated as a shell globbing character (which is important because it is used as one). Of the eight instances in these files: - One was unnecessary as `cat` can natively consume all files passed to it and no directory traversal was in use. - Two were unnecessary as they were hardcoded to read a single file.... - Another four were only being used to strip leading directory paths, and can be replaced by globstar and ${filepath##*/} - The final two were checking the modification time of the files, and can be replaced with touch(1) and [[ -nt ]]. Although this introduces an additional temporary file, this is not such a big deal. commit 3e01ba9a82234c1d78a14be095c18a626591ad71 Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Thu Feb 15 22:26:24 2018 -0500 db-update: replace external find command with bash globbing Don't bother emitting errors. bash doesn't show globbing errors if it cannot read a directory to try globbing there. And the former code never aborted on errors anyway, as without `set -o pipefail` the sort command swallowed the return code. commit 8757c459089c0db12e57db66561f640f657e34ac Author: Luke Shumaker <lukeshu(a)parabola.nu> Date: Fri Feb 16 12:10:25 2018 -0500 test: db-update: verify that PKGEXT(S) is treated as a glob Commit b61a714 introduced a regression that in a db-functions function called by db-update, config:PKGEXT is treated like a fixed string, instead of as a glob. Because of inadequacy of the test-suite, this did not cause a test failure. https://lists.archlinux.org/pipermail/arch-projects/2018-February/004742.ht… The broken function checks if the repo already contains a package that matches the one being released. This did not trigger a test-suite failure because right after the broken check there is another check that checks for the exact filename of the new package file. In the test-suite, all package files have the same extension, so the latter check always "saved" us. So, modify a relevant test case (a test case that verifies that we can't release the same package twice) such that the second time we release the package, we set PKGEXT to something different (.gz, instead of the normal .xz). This way, we can be sure that when db-update rejects the operation, it's because of the glob check, not the exact-filename check. This is made slightly tricky because the __buildPackage() test helper routine tries to cache builds, to speed up the tests. Without modification, this means that it will just ignore the new PKGEXT. So, modify it to only use the cache if it can successfully copy build artifacts out, in a way that respects PKGEXT. commit 07e6a91da1331ae61b667d878092c5b03aa49ced Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Mon Feb 19 11:12:02 2018 -0500 Fix overloading PKGEXT to mean two things. PKGEXT is a makepkg variable referring to a fixed filename suffix, but we were also using it to mean a bash glob referring to candidate filenames. This is wrong, so rename it to PKGEXTS which is more descriptive of its purpose. Exclude the testsuite from this change, as the testsuite actually uses PKGEXT for its intended purpose. Fix the testsuite to consistently use PKGEXT, as it hardcoded the file extension in several cases, and pin its value to .pkg.tar.xz commit b45650c3e3870b975620dd89bb6aa77585c8ff02 Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Mon Feb 19 11:42:22 2018 -0500 Use even more bashisms. Catch some cases that were missed in the previous run. ----------------------------------------------------------------------- Summary of changes: config | 4 +++- cron-jobs/devlist-mailer | 6 +++--- cron-jobs/ftpdir-cleanup | 36 ++++++++++++++++++++++++------------ cron-jobs/integrity-check | 2 +- cron-jobs/sourceballs | 31 +++++++++++++++++++++---------- cron-jobs/update-web-db | 6 +++--- db-functions | 16 ++++++++++++++-- db-move | 4 ++-- db-update | 14 ++++++++------ test/cases/db-repo-add.bats | 6 +++--- test/cases/db-update.bats | 5 ++--- test/cases/ftpdir-cleanup.bats | 6 +++--- test/lib/common.bash | 8 ++++---- 13 files changed, 91 insertions(+), 53 deletions(-) hooks/post-receive -- Official repo DB scripts

1 0

[arch-projects] [dbscripts] [PATCH 0/3] Fix ambiguous uses of
by Eli Schwartz 20 Feb '18

20 Feb '18

This was sort of cobbled together and not really tested, so I'm not 100% sure it will work, but it looks okay, so I am posting this to get more eyes on it. I think I've actually gotten this to work properly, which is yay, and support multiple extensions, which is meh but we may need this as Luke said, if we ever decide to switch over. Which means it is probably "more proper" to do so... As a bonus, we get to micro-optimize a few external calls away which saves us a handful of forked processes and should be just as fast not counting a fraction of a second gained for all those forks? Eli Schwartz (3): db-update: replace external find command with bash globbing ftpdir-cleanup,sourceballs: replace external find command with bash globbing Globally set $PKGEXT to a bash extended glob representing valid choices. config | 3 ++- cron-jobs/ftpdir-cleanup | 24 ++++++++++++++++++------ cron-jobs/sourceballs | 21 ++++++++++++++++----- db-functions | 13 +++++++++++-- db-update | 9 ++++++--- 5 files changed, 53 insertions(+), 17 deletions(-) -- 2.16.1

4 23