- Pacman-dev - lists.archlinux.org

[pacman-dev] [PATCH] libmakepkg: Support zstd decompression
by Morten Linderud 21 Feb '21

21 Feb '21

From: Morten Linderud <morten(a)linderud.pw> This enables us to extract files in the source array and ensures that we can decompress files if the uncompressed signature is served. Signed-off-by: Morten Linderud <morten(a)linderud.pw> --- scripts/libmakepkg/integrity/verify_signature.sh.in | 3 ++- scripts/libmakepkg/source/file.sh.in | 5 +++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/scripts/libmakepkg/integrity/verify_signature.sh.in b/scripts/libmakepkg/integrity/verify_signature.sh.in index 2be5c493..2960ebbe 100644 --- a/scripts/libmakepkg/integrity/verify_signature.sh.in +++ b/scripts/libmakepkg/integrity/verify_signature.sh.in @@ -134,7 +134,7 @@ verify_file_signature() { fi found=0 - for ext in "" gz bz2 xz lrz lzo Z; do + for ext in "" gz bz2 xz lrz lzo Z zst; do if sourcefile="$(get_filepath "${file%.*}${ext:+.$ext}")"; then found=1 break @@ -152,6 +152,7 @@ verify_file_signature() { xz) decompress="xz -c -d" ;; lrz) decompress="lrzip -q -d" ;; lzo) decompress="lzop -c -d -q" ;; + zst) decompress="zstd -d -q -f" ;; Z) decompress="uncompress -c -f" ;; "") decompress="cat" ;; esac diff --git a/scripts/libmakepkg/source/file.sh.in b/scripts/libmakepkg/source/file.sh.in index b29aeb04..cbaab315 100644 --- a/scripts/libmakepkg/source/file.sh.in +++ b/scripts/libmakepkg/source/file.sh.in @@ -118,6 +118,11 @@ extract_file() { xz) cmd="xz" ;; *) return;; esac ;; + *application/zstd*) + case "$ext" in + zst) cmd="zstd" ;; + *) return;; + esac ;; *) # See if bsdtar can recognize the file if bsdtar -tf "$file" -q '*' &>/dev/null; then -- 2.30.1

1 0

[pacman-dev] Improve locking?
by lilydjwg 20 Feb '21

20 Feb '21

I still have issues with leftover lock file. I have some questions: In pacman.c cleanup function, the signal handler is removed before unlocking because > The soft interrupt handler dereferences config, causing a segfault if > it is called during cleanup. (commit a8e2578febbb71d1b788c7270c2be57ca1e64dec) Why wasn't it to check if config is not NULL instead? Why wasn't flock used? I mean, in addition to current lock mechanism, flock the file. I know flock doesn't work for a couple of places (NFS, non-Linux, etc), but if it works for someone's situation, it would help a lot. Thanks in advance. -- Best regards, lilydjwg

1 0

[pacman-dev] Feature: -T0 flag as default ZSTDCOMPRESSION arg
by Frederik Enste 09 Feb '21

09 Feb '21

Hello pacman devs, today I installed a big package from the AUR on a clean system and forgot to adjust the makepkg.conf beforehand. As makepkg.conf is set up to compresses all packages with zstd before installing them onto the system the compression took a while. Because of that, I would like to add the -T0 flag to the list of default ZSTCOMPRESSION args. This would allow zstd to spawn multiple compression flags (equal to the number of physical processors) and thus speed up package compression. I have created a simple patch for this feature but as this is my first contribution to this project I am not entirely sure that I adjusted all necessary files: From 6a8d811fbced5a799d88ee357d4b0c694d79e0f7 Mon Sep 17 00:00:00 2001 From: Frederik Enste <frederik(a)fenste.de> Date: Tue, 9 Feb 2021 22:43:56 +0100 Subject: [PATCH] added -T0 flag to default zstd compression Signed-off-by: Frederik Enste <frederik(a)fenste.de> --- doc/makepkg.conf.5.asciidoc | 2 +- etc/makepkg.conf.in | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/makepkg.conf.5.asciidoc b/doc/makepkg.conf.5.asciidoc index 04cc5ea9..40129a01 100644 --- a/doc/makepkg.conf.5.asciidoc +++ b/doc/makepkg.conf.5.asciidoc @@ -259,7 +259,7 @@ Options **COMPRESSGZ=**"(gzip -c -f -n)":: **COMPRESSBZ2=**"(bzip2 -c -f)":: **COMPRESSXZ=**"(xz -c -z -)":: -**COMPRESSZST=**"(zstd -c -z -)":: +**COMPRESSZST=**"(zstd -c -z -T0 -)":: **COMPRESSLZO**"(lzop -q)":: **COMPRESSLRZ=**"(lrzip -q)":: **COMPRESSLZ4=**"(lz4 -q)":: diff --git a/etc/makepkg.conf.in b/etc/makepkg.conf.in index b13b1d5d..8d655bf8 100644 --- a/etc/makepkg.conf.in +++ b/etc/makepkg.conf.in @@ -133,7 +133,7 @@ DBGSRCDIR="/usr/src/debug" COMPRESSGZ=(gzip -c -f -n) COMPRESSBZ2=(bzip2 -c -f) COMPRESSXZ=(xz -c -z -) -COMPRESSZST=(zstd -c -z -q -) +COMPRESSZST=(zstd -c -z -q -T0 -) COMPRESSLRZ=(lrzip -q) COMPRESSLZO=(lzop -q) COMPRESSZ=(compress -c -f) -- 2.30.0 Of course, I would understand if this "feature" is not important or requested by other users as everybody could change the makepkg.conf in seconds but please let me know what you think about this. Best regards, Frederik Enste

3 2

[pacman-dev] [GIT] The official pacman repository branch, master, updated. v6.0.0alpha1-70-g17d3da47
by Allan McRae 08 Feb '21

08 Feb '21

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "The official pacman repository". The branch, master has been updated via 17d3da477770313a16a4f29ecae8476613ad3277 (commit) via cc1d23d333992611c4f0bd7e95c4d11fd5c7eaff (commit) via 9e425cd1ea4a5fe7e7d7f1daa225eed37650dafc (commit) via 2b8d00b7993435a8dc6f560b508a33050830ceec (commit) via 3a23abb2ec0c99d74719f97dcc9d097a105fe42b (commit) via 88d054093c1c99a697d95b26bd9aad5bc4d8e170 (commit) from ab549c846707230e44e2b7e08c6a04d44af916cd (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 17d3da477770313a16a4f29ecae8476613ad3277 Author: Emil Velikov <emil.l.velikov(a)gmail.com> Date: Sat Jan 23 21:40:04 2021 +0000 Revert "ci: cache packages" This reverts commit e348ba38814c9cc7c1c9892d0451096234dc39ab. With the above commit we started caching the downloaded packages. Based on some testing and, it saves ~30s in the "step_script" stage while adding 18s for "Restoring/Saving cache". A net saving of ~10s. With earlier commit, we no longer use an ancient image which also pulls base-devel - thus the packages we have to download is minimal. Now comparing the uncached "step_script", vs the cached one - it is slowed by 2-3 seconds (1:01 -> 1:03), while we eliminate the 18s (and growing) caching. Tl:Dr: With up-to date image, package caching in not worth it - be that time, disk or network wise. Signed-off-by: Emil Velikov <emil.l.velikov(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit cc1d23d333992611c4f0bd7e95c4d11fd5c7eaff Author: Emil Velikov <emil.l.velikov(a)gmail.com> Date: Sat Jan 23 21:40:03 2021 +0000 ci: use official image The archlinux/base have been deprecated. Since we depend on base-devel simply use archlinux:base-devel Signed-off-by: Emil Velikov <emil.l.velikov(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 9e425cd1ea4a5fe7e7d7f1daa225eed37650dafc Author: Matti Niemenmaa <matti.niemenmaa+git(a)iki.fi> Date: Sat Jan 30 19:46:40 2021 +0200 makepkg: Don't double-layer distcc on ccache buildenv is set once for build() and a second time for package(). When using both distcc and ccache, this lead to CCACHE_PREFIX="distcc distcc" in package(), which breaks PKGBUILDs that execute the compiler in package() because distcc complains: distcc[383041] (main) CRITICAL! distcc seems to have invoked itself recursively! Avoid causing this error by only adding "distcc" to CCACHE_PREFIX if it's not yet there. Signed-off-by: Matti Niemenmaa <matti.niemenmaa+git(a)iki.fi> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 2b8d00b7993435a8dc6f560b508a33050830ceec Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Wed Feb 3 16:57:33 2021 -0500 doxygen: fix missing parameter name This was raising a warning during the build. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 3a23abb2ec0c99d74719f97dcc9d097a105fe42b Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Sun Feb 7 19:55:17 2021 -0500 support xattr when extracting packages This permits storing the result of setcap during package() and applying the resulting capabilities to the installed program. Formerly, it was necessary to edit the binary after the fact (and thus dirty the file according to -Qkk) by using an install scriptlet. One problem that needs to be solved before this is useful, is preventing the strip routine from destroying xattrs. This is taken care of in the previous patch. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 88d054093c1c99a697d95b26bd9aad5bc4d8e170 Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Sun Feb 7 23:09:42 2021 -0500 makepkg: don't let the strip routine mess up file attributes It updates the stripped/objcopied file by creating a temp file, chown/chmodding it, and replacing the original file. But upstream binutils has CVE-worthy issues with this if running strip as root, and some recent versions of strip don't play nicely with fakeroot. Also, this has always destroyed xattrs. :/ Sidestep the issue by telling strip/objcopy to write to a temporary file, and manually dump the contents of that back into the original binary. Since the original binary is intact, albeit with different contents, it retains its correct attributes in fakeroot. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> Signed-off-by: Allan McRae <allan(a)archlinux.org> ----------------------------------------------------------------------- Summary of changes: .gitlab-ci.yml | 20 +++++--------------- lib/libalpm/add.c | 1 + lib/libalpm/alpm_list.h | 2 +- scripts/libmakepkg/buildenv/compiler.sh.in | 4 +++- scripts/libmakepkg/tidy/strip.sh.in | 11 +++++++++-- 5 files changed, 19 insertions(+), 19 deletions(-) hooks/post-receive -- The official pacman repository

1 0

[pacman-dev] [PATCH] makepkg: Don't double-layer distcc on ccache
by Matti Niemenmaa 08 Feb '21

08 Feb '21

From: Matti Niemenmaa <matti.niemenmaa+git(a)iki.fi> buildenv is set once for build() and a second time for package(). When using both distcc and ccache, this lead to CCACHE_PREFIX="distcc distcc" in package(), which breaks PKGBUILDs that execute the compiler in package() because distcc complains: distcc[383041] (main) CRITICAL! distcc seems to have invoked itself recursively! Avoid causing this error by only adding "distcc" to CCACHE_PREFIX if it's not yet there. Signed-off-by: Matti Niemenmaa <matti.niemenmaa+git(a)iki.fi> --- scripts/libmakepkg/buildenv/compiler.sh.in | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) ncurses is an example of a package that runs into this problem. Setting buildenv twice was done on purpose in commit 02a0bf550a22e199f48537b7eee87361b112e8a0, but at a glance I'm not sure whether it provides any benefit. All the meaningful changes are to environment variables that are exported from the parent process anyway, so all the changes are either repeated or even duplicated (like in PATH, or this CCACHE_PREFIX issue). Maybe it would be better to simply remove the prepare_buildenv call from the INFAKEROOT case in makepkg.sh.in? diff --git a/scripts/libmakepkg/buildenv/compiler.sh.in b/scripts/libmakepkg/buildenv/compiler.sh.in index 69f58a29..c93c77b4 100644 --- a/scripts/libmakepkg/buildenv/compiler.sh.in +++ b/scripts/libmakepkg/buildenv/compiler.sh.in @@ -44,7 +44,9 @@ buildenv_ccache() { buildenv_distcc() { if check_buildoption "distcc" "y"; then if (( using_ccache )); then - export CCACHE_PREFIX="${CCACHE_PREFIX:+$CCACHE_PREFIX }distcc" + if ! [[ "$CCACHE_PREFIX" =~ (^| )distcc($| ) ]]; then + export CCACHE_PREFIX="${CCACHE_PREFIX:+$CCACHE_PREFIX }distcc" + fi export CCACHE_BASEDIR="$srcdir" elif [[ -d /usr/lib/distcc/bin ]]; then export PATH="/usr/lib/distcc/bin:$PATH" -- 2.24.1

3 6

[pacman-dev] [PATCH] doxygen: fix missing parameter name
by Eli Schwartz 08 Feb '21

08 Feb '21

This was raising a warning during the build. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- lib/libalpm/alpm_list.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/libalpm/alpm_list.h b/lib/libalpm/alpm_list.h index a0299a704..f76961d47 100644 --- a/lib/libalpm/alpm_list.h +++ b/lib/libalpm/alpm_list.h @@ -61,7 +61,7 @@ typedef struct __alpm_list_t { #define FREELIST(p) do { alpm_list_free_inner(p, free); alpm_list_free(p); p = NULL; } while(0) /** item deallocation callback. - * @param the item to free + * @param item the item to free */ typedef void (*alpm_list_fn_free)(void * item); -- 2.30.0

2 1

[pacman-dev] [PATCH 1/2] makepkg: don't let the strip program mess up file attributes
by Eli Schwartz 08 Feb '21

08 Feb '21

It updates the stripped file by creating a temp file, chown/chmodding it, and replacing the original file. But upstream binutils has CVE-worthy issues with this if running strip as root, and some recent versions of strip don't play nicely with fakeroot. Also, this has always destroyed xattrs. :/ Sidestep the issue by telling strip to write to a temporary file, and manually dump the contents of that back into the original binary. Since the original binary is intact, albeit with different contents, it retains its correct attributes in fakeroot. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- scripts/libmakepkg/tidy/strip.sh.in | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/scripts/libmakepkg/tidy/strip.sh.in b/scripts/libmakepkg/tidy/strip.sh.in index 4d50f4475..f7238f813 100644 --- a/scripts/libmakepkg/tidy/strip.sh.in +++ b/scripts/libmakepkg/tidy/strip.sh.in @@ -93,7 +93,10 @@ strip_file() { fi fi - strip $@ "$binary" + if strip "$@" "$binary" -o "$binary.stripped"; then + cat "$binary.stripped" > "$binary" + fi + rm -f "$binary.stripped" } -- 2.30.0

2 6

[pacman-dev] [PATCH] libmakepkg: maintain file ownership while stripping
by Allan McRae 07 Feb '21

07 Feb '21

fakeroot can be flakey, as demonstrated by strip not maintaining file permissions under fakeroot with binutils-2.36.1 built against glibc-2.33. Instead of relying on fakeroot to be functional, ensure permissions are maintained across file stripping. Also set debug symbols to always have root:root ownership. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- build-aux/edit-script.sh.in | 1 + meson.build | 4 ++++ scripts/libmakepkg/tidy/strip.sh.in | 6 ++++++ 3 files changed, 11 insertions(+) diff --git a/build-aux/edit-script.sh.in b/build-aux/edit-script.sh.in index 661c22d5..fa5ebcf2 100644 --- a/build-aux/edit-script.sh.in +++ b/build-aux/edit-script.sh.in @@ -19,6 +19,7 @@ mode=$3 -e "s|@TEMPLATE_DIR[@]|@TEMPLATE_DIR@|g" \ -e "s|@DEBUGSUFFIX[@]|@DEBUGSUFFIX@|g" \ -e "s|@INODECMD[@]|@INODECMD@|g" \ + -e "s|@STATCMD[@]|@STATCMD@|g" \ -e "s|@FILECMD[@]|@FILECMD@|g" \ "$input" >"$output" diff --git a/meson.build b/meson.build index ea9bd2f7..edf988e9 100644 --- a/meson.build +++ b/meson.build @@ -224,6 +224,7 @@ add_project_arguments('-include', 'config.h', language : 'c') filecmd = 'file' inodecmd = 'stat -c \'%i %n\'' +statcmd = 'stat -c' strip_binaries = '--strip-all' strip_shared = '--strip-unneeded' strip_static = '--strip-debug' @@ -238,11 +239,13 @@ endif os = host_machine.system() if os.startswith('darwin') inodecmd = '/usr/bin/stat -f \'%i %N\'' + statcmd = '/usr/bin/stat -c' strip_binaries = '' strip_shared = '-s' strip_static = '-s' elif os.contains('bsd') or os == 'dragonfly' inodecmd = 'stat -f \'%i %N\'' + statcmd = 'stat -c' endif chost = run_command(cc, '-dumpmachine').stdout().strip() @@ -269,6 +272,7 @@ substs.set('BUILDSCRIPT', BUILDSCRIPT) substs.set('TEMPLATE_DIR', get_option('makepkg-template-dir')) substs.set('DEBUGSUFFIX', get_option('debug-suffix')) substs.set('INODECMD', inodecmd) +substs.set('STATCMD', statcmd) substs.set('FILECMD', filecmd) substs.set('LIBMAKEPKGDIR', LIBMAKEPKGDIR) substs.set('STRIP_BINARIES', strip_binaries) diff --git a/scripts/libmakepkg/tidy/strip.sh.in b/scripts/libmakepkg/tidy/strip.sh.in index 868b96f3..6676c012 100644 --- a/scripts/libmakepkg/tidy/strip.sh.in +++ b/scripts/libmakepkg/tidy/strip.sh.in @@ -66,10 +66,15 @@ strip_file() { fi done < <(source_files "$binary") + # save original file permissions + file_owner=$(@STATCMD@ '%u' "$binary") + file_group=$(@STATCMD@ '%g' "$binary") + # copy debug symbols to debug directory mkdir -p "$dbgdir/${binary%/*}" objcopy --only-keep-debug "$binary" "$dbgdir/$binary.debug" objcopy --add-gnu-debuglink="$dbgdir/${binary#/}.debug" "$binary" + chown root:root "$dbgdir/$binary.debug" # create any needed hardlinks while IFS= read -rd '' file ; do @@ -94,6 +99,7 @@ strip_file() { fi strip $@ "$binary" + chown $file_owner:$file_group "$binary" } -- 2.30.0

1 0

[pacman-dev] makepkg --packagelist while using --source or --allsource
by ＠RubenKelevra 03 Feb '21

03 Feb '21

Hey guys, I want to package some source files and discovered that --packagelist does not change it's output when using --source or --allsource. Since makepkg isn't generating the normal packages with these flags anyway, it's a bit confusing that the output isn't changing. It would be pretty helpful for my usecase to have makepkg printing the sourcepackage-filepaths instead of the regular filepaths. Example: [ruben@i3 brave-bin]$ makepkg --source --packagelist /home/ruben/tmp/makepkg_test/brave-bin/brave-bin-1:1.19.88-1-x86_64.pkg.tar.lz4 [ruben@i3 brave-bin]$ makepkg --packagelist /home/ruben/tmp/makepkg_test/brave-bin/brave-bin-1:1.19.88-1-x86_64.pkg.tar.lz4 Best regards, Ruben

2 2

[pacman-dev] makepkg default flags
by Christopher W. 03 Feb '21

03 Feb '21

A few months back there was a proposal to edit the default makepkg build flags for improved security: https://lists.archlinux.org/pipermail/arch-dev-public/2020-July/030031.html As far as I can tell, this was never committed. The show-stopper at the time was Clang 11 not being released yet. Now that it's been in the repos for a while, could we revisit this improvement and possibly get a mass rebuild with it applied?

2 1