- Pacman-dev - lists.archlinux.org

[PATCH] libmakepkg: Do not "strip" files when the format is not recognised
by Allan McRae 12 May '22

12 May '22

We use a multi step process during stripping to ensure permissions do not get changed. However, if the initial objcopy fails, the subsequent cat results in a blank file. Abandon early if objcopy fails. Fixes FS#74486 Signed-off-by: Allan McRae <allan(a)archlinux.org> --- scripts/libmakepkg/tidy/strip.sh.in | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/scripts/libmakepkg/tidy/strip.sh.in b/scripts/libmakepkg/tidy/strip.sh.in index 688bcf1b..4cf2107c 100644 --- a/scripts/libmakepkg/tidy/strip.sh.in +++ b/scripts/libmakepkg/tidy/strip.sh.in @@ -80,7 +80,12 @@ strip_file() { # copy debug symbols to debug directory mkdir -p "$dbgdir/${binary%/*}" - objcopy --only-keep-debug "$binary" "$dbgdir/$binary.debug" + + # abandon processing files that are not a recognised format + if ! objcopy --only-keep-debug "$binary" "$dbgdir/$binary.debug" 2>/dev/null; then + return + fi + local tempfile=$(mktemp "$binary.XXXXXX") objcopy --add-gnu-debuglink="$dbgdir/${binary#/}.debug" "$binary" "$tempfile" cat "$tempfile" > "$binary" -- 2.36.1

1 0

new Delta Update support
by Ruben Kelevra 08 May '22

08 May '22

Hey guys, previously pacman have had a delta update functionality which was (to my understanding) removed because of safety concerns and poor performance. Since the old delta version was written (and removed) zstd got the ability to create delta patches. I was running some tests on that, and it seems super promising in terms of performance for creation as well as applying those patches, and the deduplication ratio is phenomenal. I created the patches on an arch-vm running on an AMD EPYC 7702P with 10 unshared cores and 32 GB memory, the storage is an SSD RAID10. # Example: Widelands It got 731M version 1.0-1 and 4 successive updates within ~1 year with just pkgrel increasing. Total for the 4 additional versions is 2.9G. Creating all 4 delta patches: 1->5 2->5 3->5 4->5 It would take a total of real 0m24.383s usr 0m19.814s sys 0m5.859s and all patches combined would be 39M. I tested out applying them on a very old netbook (the slowest device I got on hand) 2 GB memory, Intel Atom x5-Z8300 and Arch on an MMC-SSD. The 4->5 patch for example takes real 1m32,850s user 0m2,325s sys 0m16,562s. On a more modern system (Intel Core i5-1135G7 / 16 GB memory / NVMe SSD) this take just real 0m1.208s user 0m0.255s sys 0m0.748s. Downloading the full package on the other hand over LTE takes 9m22s - downloading just the delta patch take 0m5s. # Other examples While I chose a fairly big package with (obviously) low amount of changes between the pkgrel versions, here are some other examples: libreoffice-fresh-7.2.5-5 to 7.3.0-1 saving 40% libreoffice-fresh-7.3.0-1 to 7.3.0-2 saving 45% 0ad-data-a25-1 to a25-2 saving 99.96% * 0ad-data-a25-2 to a25.b-1 saving 99.84% * glibc-2.35-3 to 2.35-4 saving 70% glibc-2.32-5 to 2.35-1 saving 51.8% opencv-4.5.5-4 to opencv-4.5.5-5 saving 93.3% opencv-4.5.4-9 to 4.5.5-1 saving 37% * had to split the tar archive after 1.6 GB and make patches for each part since zstd can only handle 2 GB files. # What would need to be done, to get this going? Well, the packages in the repo need to get a second signature, for the uncompressed tar package. Pacman could first try to fetch this tar signature. If it's on the server, the server supports delta updates – if not, the full update would be loaded. The database files would need to be extended to include the signatures for the uncompressed tar archives as well as the signature. Now pacman can fetch the patch file which fits for it's stored version to the latest version, decompressed the package stored locally and applies the patch file to it. Then pacman would check the signature/checksum of the resulting tar archive, read it and discard the uncompressed files afterwards. # Caveats This would obviously result in the pkg cache containing a full package file and 1 full then, 1->2 delta, 2->3 delta, 3->4 delta over time. This could be cleaned up by calculating the last version which should be stored and compress the file locally and store it with a dedicated extension to not clash with the regular packages (and their signatures). # Database files This could also work for database updates, obviously. But it would need a bit more work as the database files would need to be versioned (or maybe the timestamp is enough?). On a daily update of the community db, we could for example save 88.3% (2022-05-06 to 2022-05-07). On inter-daily updates, this would be down to just a couple of K: The last update of the community repo was just 40K as patch-file. That means saving 99.4% while applying only takes 0m0.075s. --- Hope you that's interesting and a thing you could look into :) Best regards, Ruben

2 2

[PATCH] cleanup: Flush cursor state to stdout before removing interrupt handler
by Chris Down 06 Apr '22

06 Apr '22

It's possible that the cursor does not reappear after pressing ^C during shutdown. In my case, I noticed this when pressing ^C after getting results from `pacman -F` -- this can reasonably reliably be triggered by issuing a file query and pressing ^C shortly after results are shown. There are two reasons for this issue: 1. The graceful SIGINT handler is removed at the start of cleanup(), but the window from entering cleanup() to reaching exit() is non trivial. The main offender is FREELIST(pm_targets), which on my T14s takes >0.1s to execute. This means that if you are unlucky enough to press ^C while there, the cursor isn't coming back, because we haven't issued any command to show the cursor again yet, and the userspace signal handler is already blown away. 2. Moving console_cursor_show() to earlier in cleanup() only half solves the issue. While it's fine not to flush after _hiding_ the cursor, since it will at least make itself apparent before any other text reaches the screen, _showing_ the cursor must be followed by flushing stdout, because once the graceful SIGINT handler is gone, if you press ^C, no flush will be triggered (and thus there will be no cursor). This fixes the issue by always starting out by showing the cursor again at cleanup() time. This means that no matter where we get caught at ^C, we will not end up leaving the terminal without its beloved ensign. Signed-off-by: Chris Down <chris(a)chrisdown.name> --- src/pacman/pacman.c | 2 +- src/pacman/util.c | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/src/pacman/pacman.c b/src/pacman/pacman.c index ea536a47..e5c6e420 100644 --- a/src/pacman/pacman.c +++ b/src/pacman/pacman.c @@ -289,6 +289,7 @@ static void setuseragent(void) */ static void cleanup(int ret) { + console_cursor_show(); remove_soft_interrupt_handler(); if(config) { /* free alpm library resources */ @@ -302,7 +303,6 @@ static void cleanup(int ret) /* free memory */ FREELIST(pm_targets); - console_cursor_show(); exit(ret); } diff --git a/src/pacman/util.c b/src/pacman/util.c index 53833d6f..d57867ab 100644 --- a/src/pacman/util.c +++ b/src/pacman/util.c @@ -1427,6 +1427,15 @@ void console_cursor_show(void) { if(isatty(fileno(stdout))) { printf(CURSOR_SHOW_ANSICODE); } + + /* We typically explicitly show the cursor either when we are getting + * input from stdin, or when we are in the process of exiting. In the + * former case, it's not guaranteed that the terminal will see the + * command before reading from stdin. In the latter case, we need to + * make sure that if we get a further TERM/INT after we return signal + * disposition to SIG_DFL, it doesn't leave the cursor invisible. + */ + fflush(stdout); } char *safe_fgets_stdin(char *s, int size) -- 2.35.1

1 1

Re: [PATCH] set bash env variables before running scripts
by Andrew Gregory 14 Mar '22

14 Mar '22

(accidentally moved off-list, moving back to pacman-dev) On 03/13/22 at 10:56am, Allan McRae wrote: > On 13/3/22 10:46, Andrew Gregory wrote: > > Bash increments it when it starts, so it'll be 2 in the scriptlet process. > > Can you just set it to 2? Saves me needing such knowledge! Or is there a > side effect I am missing? No other side effects that I'm aware of. Might just tweak the comment though.

1 0

[PATCH] set bash env variables before running scripts
by Andrew Gregory 13 Mar '22

13 Mar '22

Bash sources user configuration files under a number of conditions that can cause issues with scripts when bash is used as the scriptlet shell. Bash assumes it's being run under rsh/ssh if stdin is connected to a socket and sources the user bashrc unless the environment variable $SHLVL is >= 2. Commit 6a4c6a02de9b45abe4c0f78c4f5d14d92d3359d6 switched from pipes to sockets when communicating with child processes to work around SIGPIPE issues. Normally $SHLVL would be inherited from the shell running pacman, but operations involving scriptlets are generally run with sudo which does not let the $SHLVL variable through unless specifically configured to. Similarly $BASH_ENV can cause bash to source user-specified configuration files if set. https://lists.gnu.org/archive/html/help-bash/2022-02/msg00082.html Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> --- Not entirely sure we should actually be unsetting $BASH_ENV, but Chet recommended it. lib/libalpm/util.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/libalpm/util.c b/lib/libalpm/util.c index 06e0fe7f2..7340ce832 100644 --- a/lib/libalpm/util.c +++ b/lib/libalpm/util.c @@ -659,6 +659,11 @@ int _alpm_run_chroot(alpm_handle_t *handle, const char *cmd, char *const argv[], "/", strerror(errno)); exit(1); } + /* bash thinks it's being run under rsh/ssh if stdin is a socket and + * sources ~/.bashrc unles SHLVL is > 1 */ + setenv("SHLVL", "1", 0); + /* bash sources $BASH_ENV when run non-interactively */ + unsetenv("BASH_ENV"); umask(0022); _alpm_reset_signals(); execv(cmd, argv); -- 2.35.1

2 1

Fix typo in German translation
by Robert Löhning 12 Mar '22

12 Mar '22

Hello, I'd like to submit a one character translation fix. Please find it attached and inline below. Sorry for using Thunderbird, git send-email didn't seem to work for me. Best Regards, Robert From 6c1a2e034a1a98a7c88fcdb8f53a3615f709df0e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robert=20L=C3=B6hning?= <robert.loehning(a)qt.io> Date: Thu, 10 Mar 2022 23:26:19 +0100 Subject: [PATCH] Fix typo in German translation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Robert Löhning <robert.loehning(a)qt.io> --- src/pacman/po/de.po | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pacman/po/de.po b/src/pacman/po/de.po index 8a3dcfbe..ba1ae792 100644 --- a/src/pacman/po/de.po +++ b/src/pacman/po/de.po @@ -1293,7 +1293,7 @@ msgstr " --noconfirm fragt niemals nach einer Bestätigung\n" #: src/pacman/pacman.c:226 #, c-format msgid " --confirm always ask for confirmation\n" -msgstr " --confirm fraht immer nach einer Bestätigung\n" +msgstr " --confirm fragt immer nach einer Bestätigung\n" #: src/pacman/pacman.c:227 #, c-format -- 2.25.1

2 1

[PATCH] makepkg: use bsdtar --no-read-sparse for archive creation if available
by Jonas Witschel 11 Mar '22

11 Mar '22

bsdtar uses the "pax" TAR archive format by default, which has support for storing sparse file information in the archive. Unfortunately this is a source of unreproducibility because the sparse encoding is taken from the file system and different file systems handle sparse files differently: some file systems have no support for sparsely encoded files at all, and even file systems with sparse file support can report different file information for identical files due to differing implementations. As a real world example where this happens, consider the Arch Linux package "brotli-testdata 1.0.9-7", which contains a sparsely encoded all-zeros file "usr/share/brotli/testdata/zeros". Building this package on a btrfs file system yields a different package than building it on tmpfs or ext4 solely due to different sparse file information that gets recorded in the package tarball. To improve the reproducibility of archives containing sparsely encoded files, libarchive version 3.6.0 introduces a new --no-read-sparse option. This skips reading sparse file information from disk entirely and therefore stores files "expanded" in the archive, which is the only way to make them reliably reproducible across file systems. makepkg will use this option if libarchive is recent enough to support it, which is detected at build time. --- build-aux/edit-script.sh.in | 1 + meson.build | 6 ++++++ scripts/makepkg.sh.in | 4 ++-- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/build-aux/edit-script.sh.in b/build-aux/edit-script.sh.in index 85c56cfe..992033b2 100644 --- a/build-aux/edit-script.sh.in +++ b/build-aux/edit-script.sh.in @@ -20,6 +20,7 @@ sed \ -e "s|@DEBUGSUFFIX[@]|@DEBUGSUFFIX@|g" \ -e "s|@INODECMD[@]|@INODECMD@|g" \ -e "s|@FILECMD[@]|@FILECMD@|g" \ + -e "s|@BSDTAR_NO_READ_SPARSE[@]|@BSDTAR_NO_READ_SPARSE@|g" \ "$input" >"$output" if [[ $mode ]]; then diff --git a/meson.build b/meson.build index b7cca865..1519a2bb 100644 --- a/meson.build +++ b/meson.build @@ -91,6 +91,11 @@ endif libarchive = dependency('libarchive', version : '>=3.0.0', static : get_option('buildstatic')) +if libarchive.version().version_compare('>=3.6.0') + bsdtar_no_read_sparse = '--no-read-sparse' +else + bsdtar_no_read_sparse = '' +endif libcurl = dependency('libcurl', version : '>=7.55.0', @@ -274,6 +279,7 @@ substs.set('LIBMAKEPKGDIR', LIBMAKEPKGDIR) substs.set('STRIP_BINARIES', strip_binaries) substs.set('STRIP_SHARED', strip_shared) substs.set('STRIP_STATIC', strip_static) +substs.set('BSDTAR_NO_READ_SPARSE', bsdtar_no_read_sparse) subdir('lib/libalpm') subdir('src/common') diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 5aaabf63..69757d03 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -616,7 +616,7 @@ create_package() { msg2 "$(gettext "Compressing package...")" # TODO: Maybe this can be set globally for robustness shopt -s -o pipefail - list_package_files | LANG=C bsdtar --no-fflags -cnf - --null --files-from - | + list_package_files | LANG=C bsdtar --no-fflags @BSDTAR_NO_READ_SPARSE@ -cnf - --null --files-from - | compress_as "$PKGEXT" > "${pkg_file}" || ret=$? shopt -u -o pipefail @@ -714,7 +714,7 @@ create_srcpackage() { # TODO: Maybe this can be set globally for robustness shopt -s -o pipefail - LANG=C bsdtar --no-fflags -cLf - ${pkgbase} | compress_as "$SRCEXT" > "${pkg_file}" || ret=$? + LANG=C bsdtar --no-fflags @BSDTAR_NO_READ_SPARSE@ -cLf - ${pkgbase} | compress_as "$SRCEXT" > "${pkg_file}" || ret=$? shopt -u -o pipefail -- 2.35.1

2 1

[Git][pacman/pacman][master] 7 commits: Ensure WKD key lookup returns the correct key
by Allan McRae (＠allan) 08 Mar '22

08 Mar '22

Allan McRae pushed to branch master at Pacman / Pacman Commits: e1246bad by Allan McRae at 2022-03-06T19:57:09+10:00 Ensure WKD key lookup returns the correct key Looking up a key using WKD just ensures you have a key with the same email address, it does not ensure that a key with the correct fingerprint has been downloaded. Check a key with the relevant fingerprint is available after a WKD import. - - - - - 6b37d994 by Allan McRae at 2022-03-06T20:36:59+10:00 Fix segfault when failing to import keys Avoid a segfault when a search of the keyserver returns that the key is found but returns no primary IDs. We are then likely going to fail the import, but attempt anyway because no-one know what a keyserver will do! Fixes FS#73534. Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 632eb973 by Allan McRae at 2022-03-06T21:25:30+10:00 Do not use WKD to import keys for package installs from a file In order to use WKD in pacman -U/--upgrade operations, we need to get the packager information from the .PKGINFO within the package. That has obvious security implications. e.g. something like this could convince a user to download a different key to what they expect: packager = foo bar <>^[[2K^[[0G:: Import PGP key DEADBEEF, "foo <bar> While downloading an untrusted key has little impact due to the web-of-trust model used by pacman, this could be bad in combination with an exploit that allowed trust of keys in the keyring to be altered. To be safe, do not use WKD when installing using -U. Fixes FS#73703. Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 40583ebe by Allan McRae at 2022-03-06T21:49:56+10:00 Avoid information leakage with badly formed download header Parsing of Content-Disposition relies on well formed headers. A malformed header such as: Content-Disposition=""; will result in a strnduppayload->content_disp_name, -1, ptr), which will copy memory until it hits a \0. Prevent this by only copying the value if it exists. Fixes FS#73704. Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 293762ea by Andrew Gregory at 2022-03-07T20:37:01+10:00 query: only strip leading local/ for db packages Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 221905b5 by Andrew Gregory at 2022-03-07T20:37:45+10:00 query: allow querying extra info for package files Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - bddfcc3f by morganamilo at 2022-03-08T09:56:53+10:00 libalpm: add getter for handle on db and pkg db and pkg store a pointer to the handle for internal use but don't actually provide a way for a user to get it. Making this accessible is more convenient for front ends and FFI wrappers. For example, in other languages it's common to return the error value directly. To achieve this the python and rust wrappers also store their own pointer to the handle inside their own pkg/db wrappers. Exposing this would allow the wrappers to forgo the extra pointer and just return `pkg.get_handle().last_error()`. - - - - - 7 changed files: - lib/libalpm/alpm.h - lib/libalpm/be_package.c - lib/libalpm/db.c - lib/libalpm/dload.c - lib/libalpm/package.c - lib/libalpm/signing.c - src/pacman/query.c View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/7340fb9b2e070ac90f0746… -- View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/7340fb9b2e070ac90f0746… You're receiving this email because of your account on gitlab.archlinux.org.

1 0

[PATCH] Avoid information leakage with badly formed download header
by Allan McRae 06 Mar '22

06 Mar '22

Parsing of Content-Disposition relies on well formed headers. A malformed header such as: Content-Disposition=""; will result in a strnduppayload->content_disp_name, -1, ptr), which will copy memory until it hits a \0. Prevent this by only copying the value if it exists. Fixes FS#73704. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- lib/libalpm/dload.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index a64f405f..7c27c3ea 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -295,8 +295,11 @@ static size_t dload_parseheader_cb(void *ptr, size_t size, size_t nmemb, void *u endptr--; } - STRNDUP(payload->content_disp_name, fptr, endptr - fptr + 1, - RET_ERR(payload->handle, ALPM_ERR_MEMORY, realsize)); + /* avoid information leakage with badly formed headers */ + if(endptr > fptr) { + STRNDUP(payload->content_disp_name, fptr, endptr - fptr + 1, + RET_ERR(payload->handle, ALPM_ERR_MEMORY, realsize)); + } } } -- 2.35.1

1 0

[PATCH] Do not use WKD to import keys for package installs from a file
by Allan McRae 06 Mar '22

06 Mar '22

In order to use WKD in pacman -U/--upgrade operations, we need to get the packager information from the .PKGINFO within the package. That has obvious security implications. e.g. something like this could convince a user to download a different key to what they expect: packager = foo bar <>^[[2K^[[0G:: Import PGP key DEADBEEF, "foo <bar> While downloading an untrusted key has little impact due to the web-of-trust model used by pacman, this could be bad in combination with an exploit that allowed trust of keys in the keyring to be altered. To be safe, do not use WKD when installing using -U. Fixes FS#73703. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- lib/libalpm/be_package.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/lib/libalpm/be_package.c b/lib/libalpm/be_package.c index e837ffb8..203c98c1 100644 --- a/lib/libalpm/be_package.c +++ b/lib/libalpm/be_package.c @@ -721,7 +721,6 @@ int SYMEXPORT alpm_pkg_load(alpm_handle_t *handle, const char *filename, int ful int validation = 0; char *sigpath; alpm_pkg_t *pkg_temp; - char *packager; CHECK_HANDLE(handle, return -1); ASSERT(pkg != NULL, RET_ERR(handle, ALPM_ERR_WRONG_ARGS, -1)); @@ -747,13 +746,7 @@ int SYMEXPORT alpm_pkg_load(alpm_handle_t *handle, const char *filename, int ful char *key = k->data; if(_alpm_key_in_keychain(handle, key) == 0) { pkg_temp = _alpm_pkg_load_internal(handle, filename, full); - if(pkg_temp) { - packager = pkg_temp->packager; - - } else { - packager = NULL; - } - if(_alpm_key_import(handle, packager, key) == -1) { + if(_alpm_key_import(handle, NULL, key) == -1) { fail = 1; } _alpm_pkg_free(pkg_temp); -- 2.35.1

1 0