May 2022 - Pacman-dev - lists.archlinux.org

[PATCH 1/2] Diff color version strings
by David Gustavsson 18 Aug '22

18 Aug '22

In package table, with VerbosePkgLists and Color, color the part of package version strings that changes. Signed-off-by: David Gustavsson <david.e.gustavsson(a)gmail.com> --- src/pacman/conf.c | 4 ++++ src/pacman/conf.h | 2 ++ src/pacman/util.c | 35 +++++++++++++++++++++++++++++++---- src/pacman/util.h | 1 + 4 files changed, 38 insertions(+), 4 deletions(-) diff --git a/src/pacman/conf.c b/src/pacman/conf.c index f9edf75b..d5c598b9 100644 --- a/src/pacman/conf.c +++ b/src/pacman/conf.c @@ -80,6 +80,8 @@ void enable_colors(int colors) colstr->err = BOLDRED; colstr->faint = GREY46; colstr->nocolor = NOCOLOR; + colstr->diffrem = RED; + colstr->diffadd = GREEN; } else { colstr->colon = ":: "; colstr->title = ""; @@ -91,6 +93,8 @@ void enable_colors(int colors) colstr->err = ""; colstr->faint = ""; colstr->nocolor = ""; + colstr->diffrem = ""; + colstr->diffadd = ""; } } diff --git a/src/pacman/conf.h b/src/pacman/conf.h index f7916ca9..782b5758 100644 --- a/src/pacman/conf.h +++ b/src/pacman/conf.h @@ -33,6 +33,8 @@ typedef struct __colstr_t { const char *err; const char *faint; const char *nocolor; + const char *diffrem; + const char *diffadd; } colstr_t; typedef struct __config_repo_t { diff --git a/src/pacman/util.c b/src/pacman/util.c index 53833d6f..10d7c573 100644 --- a/src/pacman/util.c +++ b/src/pacman/util.c @@ -840,8 +840,10 @@ static alpm_list_t *create_verbose_row(pm_target_t *target) { char *str; off_t size = 0; + size_t i = 0; double human_size; const char *label; + const char *version; alpm_list_t *ret = NULL; /* a row consists of the package name, */ @@ -858,12 +860,28 @@ static alpm_list_t *create_verbose_row(pm_target_t *target) add_table_cell(&ret, str, CELL_NORMAL | CELL_FREE); /* old and new versions */ - pm_asprintf(&str, "%s", - target->remove != NULL ? alpm_pkg_get_version(target->remove) : ""); + version = alpm_pkg_get_version(target->remove); + char old_version[strlen(version) + strlen(config->colstr.diffrem) + strlen(config->colstr.nocolor) + 1]; + memcpy(old_version, (char *)version, strlen(version) + 1); + version = alpm_pkg_get_version(target->install); + char new_version[strlen(version) + strlen(config->colstr.diffadd) + strlen(config->colstr.nocolor) + 1]; + memcpy(new_version, (char *)version, strlen(version) + 1); + + i = 0; + while (old_version[i] != '\0' && new_version[i] != '\0' && old_version[i] == new_version[i]) { + i++; + } + insertstring(old_version, config->colstr.diffrem, i); + insertstring(new_version, config->colstr.diffadd, i); + snprintf(old_version + i + strlen(old_version + i), + strlen(config->colstr.nocolor) + 1, "%s", config->colstr.nocolor); + snprintf(new_version + i + strlen(new_version + i), + strlen(config->colstr.nocolor) + 1, "%s", config->colstr.nocolor); + + pm_asprintf(&str, "%s", old_version); add_table_cell(&ret, str, CELL_NORMAL | CELL_FREE); - pm_asprintf(&str, "%s", - target->install != NULL ? alpm_pkg_get_version(target->install) : ""); + pm_asprintf(&str, "%s", new_version); add_table_cell(&ret, str, CELL_NORMAL | CELL_FREE); /* and size */ @@ -1859,3 +1877,12 @@ void console_erase_line(void) { printf("\x1B[K"); } + +/* inserts string insert into string at index i, modifying string */ +void insertstring(char *string, const char *insert, size_t i) +{ + char temp[strlen(string) - i + 1]; + memcpy(temp, string + i, strlen(string + i) + 1); + memcpy(string + i, insert, strlen(insert) + 1); + memcpy(string + i + strlen(insert), temp, strlen(temp) + 1); +} diff --git a/src/pacman/util.h b/src/pacman/util.h index 52e79915..003961eb 100644 --- a/src/pacman/util.h +++ b/src/pacman/util.h @@ -88,6 +88,7 @@ void console_cursor_move_down(unsigned int lines); void console_cursor_move_end(void); /* Erases line from the current cursor position till the end of the line */ void console_erase_line(void); +void insertstring(char *string, const char *insert, size_t i); int pm_printf(alpm_loglevel_t level, const char *format, ...) __attribute__((format(printf,2,3))); int pm_asprintf(char **string, const char *format, ...) __attribute__((format(printf,2,3))); -- 2.35.1

2 3

[PATCH v2] util: Flush cursor state to stdout before removing signal handler
by Chris Down 22 Jul '22

22 Jul '22

It's possible that the cursor does not reappear after pressing ^C during shutdown. In my case, I noticed this when pressing ^C after getting results from `pacman -F` -- this can reasonably reliably be triggered by issuing a file query and pressing ^C shortly after results are shown. There are two reasons for this issue: 1. The graceful SIGINT handler is removed at the start of cleanup(), but the window from entering cleanup() to reaching exit() is non trivial. The main offender is FREELIST(pm_targets), which on my T14s takes >0.1s to execute. This means that if you are unlucky enough to press ^C while there, the cursor isn't coming back, because we haven't issued any command to show the cursor again yet, and the userspace signal handler is already blown away. 2. Moving console_cursor_show() to earlier in cleanup() only half solves the issue. While it's fine not to flush after _hiding_ the cursor, since it will at least make itself apparent before any other text reaches the screen, _showing_ the cursor must be followed by flushing stdout, because once the graceful SIGINT handler is gone, if you press ^C, no flush will be triggered (and thus there will be no cursor). This fixes the issue by always starting out by showing the cursor again at cleanup() time. This means that no matter where we get caught at ^C, we will not end up leaving the terminal without its beloved ensign. Signed-off-by: Chris Down <chris(a)chrisdown.name> --- src/pacman/pacman.c | 2 +- src/pacman/util.c | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/src/pacman/pacman.c b/src/pacman/pacman.c index ea536a47..e5c6e420 100644 --- a/src/pacman/pacman.c +++ b/src/pacman/pacman.c @@ -289,6 +289,7 @@ static void setuseragent(void) */ static void cleanup(int ret) { + console_cursor_show(); remove_soft_interrupt_handler(); if(config) { /* free alpm library resources */ @@ -302,7 +303,6 @@ static void cleanup(int ret) /* free memory */ FREELIST(pm_targets); - console_cursor_show(); exit(ret); } diff --git a/src/pacman/util.c b/src/pacman/util.c index 53833d6f..e92b2012 100644 --- a/src/pacman/util.c +++ b/src/pacman/util.c @@ -1426,6 +1426,16 @@ void console_cursor_hide(void) { void console_cursor_show(void) { if(isatty(fileno(stdout))) { printf(CURSOR_SHOW_ANSICODE); + + /* We typically explicitly show the cursor either when we are + * getting input from stdin, or when we are in the process of + * exiting. In the former case, it's not guaranteed that the + * terminal will see the command before reading from stdin. In + * the latter case, we need to make sure that if we get a + * further TERM/INT after we return signal disposition to + * SIG_DFL, it doesn't leave the cursor invisible. + */ + fflush(stdout); } } -- 2.35.1

2 4

[PATCH] Fix buffer overflow for 'Include' parameter in pacman.conf
by tim174 21 Jul '22

21 Jul '22

If the 'Include' parameter in the config file is set to a long string (for example 3000x '/') the pacman config parser will crash when calling glob in conf.c with this value. If the string is shorter than approx. 3000 symbols the normal error message is printed and the segfault does not occur. I was able to reproduce this on my own system and on the official Arch Linux iso image. The PATH_MAX variable is too large to prevent this bug, hence the new variable GLOB_LIMIT with a security buffer of 1000 since I don't know how persistent this limit is across systems. Its origin is unclear to me and I am not sure if it is a fixed value. This is why I would appreciate any help to make this a sustainable patch. Thanks Tim Signed-off-by: Tim <tim174 at protonmail.com> --- src/pacman/conf.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/pacman/conf.c b/src/pacman/conf.c index f9edf75b..d8222752 100644 --- a/src/pacman/conf.c +++ b/src/pacman/conf.c @@ -65,6 +65,9 @@ config_t *config = NULL; #define BOLDWHITE "\033[1;37m" #define GREY46 "\033[38;5;243m" +/* limit for glob input variable */ +#define GLOB_LIMIT 2000 + void enable_colors(int colors) { colstr_t *colstr = &config->colstr; @@ -1042,6 +1045,13 @@ static int process_include(const char *value, void *data, return 1; } + if(strlen(value) > GLOB_LIMIT) { + pm_printf(ALPM_LOG_ERROR, + ("config file %s, line %d, directive '%s': value too long\n"), + file, linenum, "Include"); + return 1; + } + if(section->depth >= config_max_recursion) { pm_printf(ALPM_LOG_ERROR, _("config parsing exceeded max recursion depth of %d.\n"), -- 2.34.1

2 1

[PATCH] dload: Remove unused ABORT_SIGINT
by Chris Down 21 Jul '22

21 Jul '22

The last user of ABORT_SIGINT was removed in commit 84723cab5dfc ("Cleanup the old sequential download code"), and this isn't exported as part of the public API. Signed-off-by: Chris Down <chris(a)chrisdown.name> --- lib/libalpm/dload.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 7c27c3ea..4fa17b35 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -153,8 +153,7 @@ static char *get_fullpath(const char *path, const char *filename, } enum { - ABORT_SIGINT = 1, - ABORT_OVER_MAXFILESIZE + ABORT_OVER_MAXFILESIZE = 1, }; static int dload_interrupted; -- 2.35.1

2 1

Extend print-format with expac options
by Jelle van der Waa 21 Jul '22

21 Jul '22

This patch series extends --print-format with a subset of the expac options. More options to be added in further patch series.

3 8

[RFC][WIP][PATCH 0/5] adding extended package data
by Andrew Gregory 21 Jul '22

21 Jul '22

ALPM is currently parsing several fields that are not particularly relevant to managing package installation (e.g. pkgbase, pkgtype, makedepends, checkdepends). pkgtype in particular is not even exposed in any way through the API, it has to be included in the parser with a "not actually used" comment. This patchset adds an extended data field to packages that can be used by packagers or package-building infrastructure to store arbitrary package data in a way that doesn't require altering alpm for every field and allows the data to actually be retrieved via alpm (though not as easily as first-class fields). Andrew Gregory (5): query: only strip leading local/ for db packages query: allow querying extra info for package files add data field for arbitrary package data query: print extended data when extra info is requested makepkg: store package type in extended data lib/libalpm/alpm.h | 7 +++++++ lib/libalpm/be_local.c | 25 +++++++++++++++++++++++++ lib/libalpm/be_package.c | 6 ++++++ lib/libalpm/be_sync.c | 16 ++++++++++++++++ lib/libalpm/package.c | 17 +++++++++++++++++ lib/libalpm/package.h | 6 ++++++ scripts/makepkg.sh.in | 2 +- src/pacman/package.c | 13 +++++++++++++ src/pacman/query.c | 16 ++++++---------- 9 files changed, 97 insertions(+), 11 deletions(-) -- 2.35.0

2 13

[PATCH] makepkg: Implement the verify function
by Morten Linderud 25 Jun '22

25 Jun '22

From: Morten Linderud <morten(a)linderud.pw> This patch implements a new verify function in makepkg. It allows us to do arbitrary authentication on sources before extraction. There are several new signing and validation methods being implemented and it would be hard to have `makepkg` implement support for things such as sequoia, cosign or minisign. This would allow us to distribute generic validation functions. This also implements a new `copy_` routine for our protocols as we need to have a separation between extracting sources and copying sources. Signed-off-by: Morten Linderud <morten(a)linderud.pw> --- doc/PKGBUILD.5.asciidoc | 4 ++++ doc/makepkg.8.asciidoc | 3 +++ scripts/libmakepkg/integrity.sh.in | 3 +++ scripts/libmakepkg/source.sh.in | 15 +++++++++++++++ scripts/libmakepkg/source/file.sh.in | 9 ++++++++- scripts/makepkg.sh.in | 17 ++++++++++++++++- 6 files changed, 49 insertions(+), 2 deletions(-) diff --git a/doc/PKGBUILD.5.asciidoc b/doc/PKGBUILD.5.asciidoc index 4ca8eb3b..e7743c88 100644 --- a/doc/PKGBUILD.5.asciidoc +++ b/doc/PKGBUILD.5.asciidoc @@ -344,6 +344,10 @@ function. fakeroot to ensure correct file permissions in the resulting package. All other functions will be run as the user calling makepkg. +*verify() Function*:: + An optional `verify()` function can be specified to implement arbiterary + source authentication. This function is run before sources are extracted. + *prepare() Function*:: An optional `prepare()` function can be specified in which operations to prepare the sources for building, such as patching, are performed. This diff --git a/doc/makepkg.8.asciidoc b/doc/makepkg.8.asciidoc index 38032e7b..75b2139f 100644 --- a/doc/makepkg.8.asciidoc +++ b/doc/makepkg.8.asciidoc @@ -168,6 +168,9 @@ Options *\--noprepare*:: Do not run the prepare() function in the PKGBUILD. +*\--noverify*:: + Do not run the verify() function in the PKGBUILD. + *\--sign*:: Sign the resulting package with gpg, overriding the setting in linkman:makepkg.conf[5]. diff --git a/scripts/libmakepkg/integrity.sh.in b/scripts/libmakepkg/integrity.sh.in index 070392fa..81f935df 100644 --- a/scripts/libmakepkg/integrity.sh.in +++ b/scripts/libmakepkg/integrity.sh.in @@ -42,4 +42,7 @@ check_source_integrity() { check_checksums "$@" check_pgpsigs "$@" fi + if (( VERIFYFUNC )); then + run_verify + fi } diff --git a/scripts/libmakepkg/source.sh.in b/scripts/libmakepkg/source.sh.in index e39dd16c..92dc71e4 100644 --- a/scripts/libmakepkg/source.sh.in +++ b/scripts/libmakepkg/source.sh.in @@ -69,6 +69,21 @@ download_sources() { done } +copy_sources(){ + msg "$(gettext "Copying sources...")" + local netfile all_sources + + get_all_sources_for_arch 'all_sources' + for netfile in "${all_sources[@]}"; do + local proto=$(get_protocol "$netfile") + if declare -f copy_$proto > /dev/null; then + copy_$proto "$netfile" + else + copy_file "$netfile" + fi + done +} + extract_sources() { msg "$(gettext "Extracting sources...")" local netfile all_sources diff --git a/scripts/libmakepkg/source/file.sh.in b/scripts/libmakepkg/source/file.sh.in index fa09d446..51452550 100644 --- a/scripts/libmakepkg/source/file.sh.in +++ b/scripts/libmakepkg/source/file.sh.in @@ -82,13 +82,20 @@ download_file() { fi } -extract_file() { +copy_file(){ local netfile=$1 local file=$(get_filename "$netfile") local filepath=$(get_filepath "$file") rm -f "$srcdir/${file}" ln -s "$filepath" "$srcdir/" +} + +extract_file() { + local netfile=$1 + + local file=$(get_filename "$netfile") + local filepath=$(get_filepath "$file") if in_array "$file" "${noextract[@]}"; then # skip source files in the noextract=() array diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 5aaabf63..b7b21af1 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -144,6 +144,9 @@ clean_up() { if (( PKGVERFUNC )); then rm -f "${pkgbase}-${fullver}-${CARCH}-pkgver.log"* fi + if (( VERIFYFUNC )); then + rm -f "${pkgbase}-${fullver}-${CARCH}-verify.log"* + fi if (( PREPAREFUNC )); then rm -f "${pkgbase}-${fullver}-${CARCH}-prepare.log"* fi @@ -447,6 +450,10 @@ run_function() { fi } +run_verify() { + run_function_safe "verify" +} + run_prepare() { run_function_safe "prepare" } @@ -973,6 +980,7 @@ while true; do -m|--nocolor) USE_COLOR='n'; PACMAN_OPTS+=("--color" "never") ;; --noarchive) NOARCHIVE=1 ;; --nocheck) RUN_CHECK='n' ;; + --noverify) RUN_VERIFY='n' ;; --noprepare) RUN_PREPARE='n' ;; --nosign) SIGNPKG='n' ;; -o|--nobuild) BUILDPKG=0 NOBUILD=1 ;; @@ -1093,7 +1101,7 @@ fi unset pkgname "${pkgbuild_schema_strings[@]}" "${pkgbuild_schema_arrays[@]}" unset "${known_hash_algos[@]/%/sums}" -unset -f pkgver prepare build check package "${!package_@}" +unset -f pkgver verify prepare build check package "${!package_@}" unset "${!makedepends_@}" "${!depends_@}" "${!source_@}" "${!checkdepends_@}" unset "${!optdepends_@}" "${!conflicts_@}" "${!provides_@}" "${!replaces_@}" unset "${!cksums_@}" "${!md5sums_@}" "${!sha1sums_@}" "${!sha224sums_@}" @@ -1165,6 +1173,12 @@ if (( ${#pkgname[@]} > 1 )) || have_function package_${pkgname}; then fi # test for available PKGBUILD functions +if have_function verify; then + # "Hide" verify() function if not going to be run + if [[ $RUN_VERIFY != "n" ]]; then + VERIFYFUNC=1 + fi +fi if have_function prepare; then # "Hide" prepare() function if not going to be run if [[ $RUN_PREPARE != "n" ]]; then @@ -1312,6 +1326,7 @@ if (( !REPKG )); then warning "$(gettext "Using existing %s tree")" "\$srcdir/" else download_sources + copy_sources check_source_integrity (( VERIFYSOURCE )) && exit $E_OK -- 2.36.1

2 3

[PATCH] pacman: send 'loading packages...' msg to stderr
by Jean Raby 25 Jun '22

25 Jun '22

This allows for parsing the output of: pacman --upgrade --print-format '<format>' pkg.zst without having to remove info messages from it. Signed-off-by: Jean Raby <jean(a)raby.sh> --- src/pacman/upgrade.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pacman/upgrade.c b/src/pacman/upgrade.c index 0691856f..6f70b115 100644 --- a/src/pacman/upgrade.c +++ b/src/pacman/upgrade.c @@ -99,7 +99,7 @@ int pacman_upgrade(alpm_list_t *targets) goto fail_free; } - printf(_("loading packages...\n")); + fprintf(stderr,_("loading packages...\n")); retval |= load_packages(local_targets, alpm_option_get_local_file_siglevel(config->handle)); retval |= load_packages(fetched_files, alpm_option_get_remote_file_siglevel(config->handle)); -- 2.35.1

2 3

[PATCH v2] strip: Use dwz to compress dwarf headers
by Morten Linderud 25 Jun '22

25 Jun '22

From: Morten Linderud <morten(a)linderud.pw> `dwz` allows us to compress the DWARF files when building debug packages. Potentially shaving off some of the sizes of the symbols we distribute. https://sourceware.org/dwz/ A sample of packages built with dwz; pacman: Original debug info size: 1520kB Size after compression: 1252kB systemd: Original debug info size: 46692kB Size after compression: 41036kB Signed-off-by: Morten Linderud <morten(a)linderud.pw> --- doc/makepkg.conf.5.asciidoc | 3 ++ scripts/libmakepkg/executable/dwz.sh.in | 38 ++++++++++++++++ .../executable/sepdebugcrcfix.sh.in | 38 ++++++++++++++++ scripts/libmakepkg/tidy/strip.sh.in | 44 +++++++++++++++++-- 4 files changed, 120 insertions(+), 3 deletions(-) create mode 100644 scripts/libmakepkg/executable/dwz.sh.in create mode 100644 scripts/libmakepkg/executable/sepdebugcrcfix.sh.in diff --git a/doc/makepkg.conf.5.asciidoc b/doc/makepkg.conf.5.asciidoc index a0d9a6d4..2e40f27f 100644 --- a/doc/makepkg.conf.5.asciidoc +++ b/doc/makepkg.conf.5.asciidoc @@ -194,6 +194,9 @@ Options DEBUG_CXXFLAGS to their counterpart buildflags. Creates a separate package containing the debug symbols when used with `strip'. + *dwz*;; + Compress dwarf files inside the debug package. + *lto*;; Enable building packages using link time optimization. Adds the flags specified in LTOFLAGS to CFLAGS, CXXFLAGS and LDFLAGS (or diff --git a/scripts/libmakepkg/executable/dwz.sh.in b/scripts/libmakepkg/executable/dwz.sh.in new file mode 100644 index 00000000..a540697c --- /dev/null +++ b/scripts/libmakepkg/executable/dwz.sh.in @@ -0,0 +1,38 @@ +#!/usr/bin/bash +# +# dwz.sh - Confirm presence of dwz binary +# +# Copyright (c) 2011-2022 Pacman Development Team <pacman-dev(a)lists.archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_EXECUTABLE_DWZ_SH" ]] && return +LIBMAKEPKG_EXECUTABLE_DWZ_SH=1 + +LIBRARY=${LIBRARY:-'@libmakepkgdir@'} + +source "$LIBRARY/util/message.sh" +source "$LIBRARY/util/option.sh" + +executable_functions+=('executable_dwz') + +executable_dwz() { + if check_option "dwz" "y"; then + if ! type -p dwz>/dev/null; then + error "$(gettext "Cannot find the %s binary required to compress dwarf files.")" "dwz" + return 1 + fi + fi +} diff --git a/scripts/libmakepkg/executable/sepdebugcrcfix.sh.in b/scripts/libmakepkg/executable/sepdebugcrcfix.sh.in new file mode 100644 index 00000000..6259ac32 --- /dev/null +++ b/scripts/libmakepkg/executable/sepdebugcrcfix.sh.in @@ -0,0 +1,38 @@ +#!/usr/bin/bash +# +# sepdebugcrcfix.sh - Confirm presence of sepdebugcrcfix binary +# +# Copyright (c) 2011-2022 Pacman Development Team <pacman-dev(a)lists.archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_EXECUTABLE_SEPDEBUGCRCFIX_SH" ]] && return +LIBMAKEPKG_EXECUTABLE_SEPDEBUGCRCFIX_SH=1 + +LIBRARY=${LIBRARY:-'@libmakepkgdir@'} + +source "$LIBRARY/util/message.sh" +source "$LIBRARY/util/option.sh" + +executable_functions+=('executable_sepdebugcrcfix') + +executable_sepdebugcrcfix() { + if check_option "dwz" "y"; then + if ! type -p sepdebugcrcfix>/dev/null; then + error "$(gettext "Cannot find the %s binary required to fix CRC.")" "sepdebugcrcfix" + return 1 + fi + fi +} diff --git a/scripts/libmakepkg/tidy/strip.sh.in b/scripts/libmakepkg/tidy/strip.sh.in index 688bcf1b..1365b77b 100644 --- a/scripts/libmakepkg/tidy/strip.sh.in +++ b/scripts/libmakepkg/tidy/strip.sh.in @@ -27,7 +27,7 @@ source "$LIBRARY/util/message.sh" source "$LIBRARY/util/option.sh" -packaging_options+=('strip' 'debug') +packaging_options+=('strip' 'debug' 'dwz') tidy_modify+=('tidy_strip') @@ -52,6 +52,37 @@ source_files() { | sort -zu | tr '\0' '\n' } +compress_dwarf_files(){ + binary_files="$@" + dwz_opts="" + readarray dwz_files < <(find "$dbgdir" -type f -name \*.debug | LC_ALL=C sort) + + # dwz multifile attempts to remove common symols found in all ELF files and + # throw them into one file. Replacing the references with .gnu_debugaltlink + local fullver=$(get_full_version) + dwz_multifile_name="${pkgbase}-${fullver}-${CARCH}" + if [ ${#dwz_files[@]} -gt 1 ]; then + dwz_opts="-m .dwz/${dwz_multifile_name}" + fi + mkdir -p "$dbgdir/.dwz" + + # The dwz options are taken from rpm/debugedit + (cd "$dbgdir" && + LANG=C dwz --hardlink \ + --quiet \ + --relative \ + --low-mem-die-limit 10000000 \ + --max-die-limit 50000000 \ + $dwz_opts \ + ${dwz_files[@]} 2> /dev/null) + + # Remove dir if empty + rmdir "$dbgdir/.dwz" 2> /dev/null + + # Some debuggers might use the CRC to find relevant debug files instead of the build-id + LANG=C sepdebugcrcfix "$dbgdir" $binary_files &>/dev/null +} + strip_file() { local binary=$1; shift @@ -142,7 +173,8 @@ tidy_strip() { fi local binary strip_flags - find . -type f -perm -u+w -print0 2>/dev/null | while IFS= read -rd '' binary ; do + declare -a binary_files + while IFS= read -rd '' binary ; do local STRIPLTO=0 case "$(LC_ALL=C readelf -h "$binary" 2>/dev/null)" in *Type:*'DYN (Shared object file)'*) # Libraries (.so) or Relocatable binaries @@ -166,6 +198,12 @@ tidy_strip() { esac strip_file "$binary" ${strip_flags} (( STRIPLTO )) && strip_lto "$binary" - done + binary_files+=("$binary") + done < <(find . -type f -perm -u+w -print0 2>/dev/null) + + if check_option "debug" "y" && check_option "dwz" "y"; then + msg2 "$(gettext "Compressing dwarf files...")" + compress_dwarf_files ${binary_files[@]} + fi fi } -- 2.36.1

3 3

[pacman-dev] [PATCH 1/3] alpm: add ALPM_TRANS_FLAG_NOKEEP
by morganamilo 25 Jun '22

25 Jun '22

this flag prevents backup files from being kept on package installation. This is useful for resetting a package's config files back to their original state. Implements FS#59908 although with it's own flag name instead of reusing nosave. This allows nokeep to optionally create a pacnew that you can then choose to disable by also setting nosave. --- I actually very dislike NOKEEP but it was the best I could come up with I would have prefered overwrite or nosave but they are taken. Better names are welcome. --- lib/libalpm/alpm.h | 3 ++- lib/libalpm/remove.c | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 8d8fe243..c6048b63 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -2704,7 +2704,8 @@ int alpm_pkg_mtree_close(const alpm_pkg_t *pkg, struct archive *archive); typedef enum _alpm_transflag_t { /** Ignore dependency checks. */ ALPM_TRANS_FLAG_NODEPS = 1, - /* (1 << 1) flag can go here */ + /** Don't keep backup files when installing packages. */ + ALPM_TRANS_FLAG_NOKEEP = (1 << 1), /** Delete files even if they are tagged as backup. */ ALPM_TRANS_FLAG_NOSAVE = (1 << 2), /** Ignore version numbers when checking dependencies. */ diff --git a/lib/libalpm/remove.c b/lib/libalpm/remove.c index de39724a..233fff0c 100644 --- a/lib/libalpm/remove.c +++ b/lib/libalpm/remove.c @@ -575,7 +575,9 @@ static int should_skip_file(alpm_handle_t *handle, { return _alpm_fnmatch_patterns(handle->noupgrade, path) == 0 || alpm_list_find_str(handle->trans->skip_remove, path) - || (newpkg && _alpm_needbackup(path, newpkg) + || (!(handle->trans->flags & ALPM_TRANS_FLAG_NOKEEP) + && newpkg + && _alpm_needbackup(path, newpkg) && alpm_filelist_contains(alpm_pkg_get_files(newpkg), path)); } -- 2.33.0

2 3

[PATCH] debugflags: Ensure we have unique source paths
by Morten Linderud 25 Jun '22

25 Jun '22

From: Morten Linderud <morten(a)linderud.pw> In some cases packages are built outside of a directory which contains pkgname-pkgver, this results in source listing in debug packages having a conflicting path like `/usr/src/debug/build/` which is not ideal. This patch ensures we always include the pkgbase to ensure the paths are unique. Signed-off-by: Morten Linderud <morten(a)linderud.pw> --- scripts/libmakepkg/buildenv/debugflags.sh.in | 6 +++--- scripts/libmakepkg/tidy/strip.sh.in | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/scripts/libmakepkg/buildenv/debugflags.sh.in b/scripts/libmakepkg/buildenv/debugflags.sh.in index 1cb58613..2bdb8b0b 100644 --- a/scripts/libmakepkg/buildenv/debugflags.sh.in +++ b/scripts/libmakepkg/buildenv/debugflags.sh.in @@ -30,9 +30,9 @@ buildenv_functions+=('buildenv_debugflags') buildenv_debugflags() { if check_option "debug" "y" && ! check_option "buildflags" "n"; then - DEBUG_CFLAGS+=" -ffile-prefix-map=$srcdir=${DBGSRCDIR:-/usr/src/debug}" - DEBUG_CXXFLAGS+=" -ffile-prefix-map=$srcdir=${DBGSRCDIR:-/usr/src/debug}" - DEBUG_RUSTFLAGS+=" --remap-path-prefix=$srcdir=${DBGSRCDIR:-/usr/src/debug}" + DEBUG_CFLAGS+=" -ffile-prefix-map=$srcdir=${DBGSRCDIR:-/usr/src/debug}/${pkgbase}" + DEBUG_CXXFLAGS+=" -ffile-prefix-map=$srcdir=${DBGSRCDIR:-/usr/src/debug}/${pkgbase}" + DEBUG_RUSTFLAGS+=" --remap-path-prefix=$srcdir=${DBGSRCDIR:-/usr/src/debug}/${pkgbase}" CFLAGS+=" $DEBUG_CFLAGS" CXXFLAGS+=" $DEBUG_CXXFLAGS" RUSTFLAGS+=" $DEBUG_RUSTFLAGS" diff --git a/scripts/libmakepkg/tidy/strip.sh.in b/scripts/libmakepkg/tidy/strip.sh.in index 688bcf1b..25bc649e 100644 --- a/scripts/libmakepkg/tidy/strip.sh.in +++ b/scripts/libmakepkg/tidy/strip.sh.in @@ -47,7 +47,7 @@ source_files() { LANG=C debugedit --no-recompute-build-id \ --base-dir "${srcdir}" \ - --dest-dir "${dbgsrcdir}" \ + --dest-dir "${dbgsrcdir}/${pkgbase}" \ --list-file /dev/stdout "$1" \ | sort -zu | tr '\0' '\n' } -- 2.36.1

2 1

[PATCH] makepkg: add source signing PGP keys to package if available
by Allan McRae 01 Jun '22

01 Jun '22

Arch Linux is adding source signing PGP keys to their package source tree alongside PKGBUILDs in the form keys/pgp/$fingerprint.asc. As the PGP keyserver infrastructure is a mess, this helps other people validate sources in a PKGBUILD. Add the keys to source packages if found alongside the PKGBUILD. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- I won't be committing this until the relevant Arch devtools patch is accepted so that the keys/pgp/ path is finalised. scripts/makepkg.sh.in | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 69757d03..bddcbe03 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -705,6 +705,16 @@ create_srcpackage() { done pkgname=(${pkgname_backup[@]}) + # add a copy of source PGP signing public keys if availabe in keys/pgp/<fingerprint>.asc + local key + for key in ${validpgpkeys[@]}; do + if [[ -f keys/pgp/$key.asc ]]; then + mkdir -p "${srclinks}/${pkgbase}/keys/pgp/" + ln -s "${startdir}/keys/pgp/$key.asc" "${srclinks}/${pkgbase}/keys/pgp/" + fi + done + + local fullver=$(get_full_version) local pkg_file="$SRCPKGDEST/${pkgbase}-${fullver}${SRCEXT}" -- 2.36.1

2 2

[PATCH] strip: Use dwz to compress dwarf headers
by Morten Linderud 27 May '22

27 May '22

From: Morten Linderud <morten(a)linderud.pw> `dwz` allows us to compress the DWARF files when building debug packages. Potentially shaving off some of the sizes of the symbols we distribute. https://sourceware.org/dwz/ A sample of packages built with dwz; pacman: Original debug info size: 1520kB Size after compression: 1252kB systemd: Original debug info size: 46692kB Size after compression: 41036kB Signed-off-by: Morten Linderud <morten(a)linderud.pw> --- doc/makepkg.conf.5.asciidoc | 3 ++ scripts/libmakepkg/executable/dwz.sh.in | 38 +++++++++++++++++++ .../executable/sepdebugcrcfix.sh.in | 38 +++++++++++++++++++ scripts/libmakepkg/tidy/strip.sh.in | 33 ++++++++++++++-- 4 files changed, 109 insertions(+), 3 deletions(-) create mode 100644 scripts/libmakepkg/executable/dwz.sh.in create mode 100644 scripts/libmakepkg/executable/sepdebugcrcfix.sh.in diff --git a/doc/makepkg.conf.5.asciidoc b/doc/makepkg.conf.5.asciidoc index a0d9a6d4..2e40f27f 100644 --- a/doc/makepkg.conf.5.asciidoc +++ b/doc/makepkg.conf.5.asciidoc @@ -194,6 +194,9 @@ Options DEBUG_CXXFLAGS to their counterpart buildflags. Creates a separate package containing the debug symbols when used with `strip'. + *dwz*;; + Compress dwarf files inside the debug package. + *lto*;; Enable building packages using link time optimization. Adds the flags specified in LTOFLAGS to CFLAGS, CXXFLAGS and LDFLAGS (or diff --git a/scripts/libmakepkg/executable/dwz.sh.in b/scripts/libmakepkg/executable/dwz.sh.in new file mode 100644 index 00000000..a540697c --- /dev/null +++ b/scripts/libmakepkg/executable/dwz.sh.in @@ -0,0 +1,38 @@ +#!/usr/bin/bash +# +# dwz.sh - Confirm presence of dwz binary +# +# Copyright (c) 2011-2022 Pacman Development Team <pacman-dev(a)lists.archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_EXECUTABLE_DWZ_SH" ]] && return +LIBMAKEPKG_EXECUTABLE_DWZ_SH=1 + +LIBRARY=${LIBRARY:-'@libmakepkgdir@'} + +source "$LIBRARY/util/message.sh" +source "$LIBRARY/util/option.sh" + +executable_functions+=('executable_dwz') + +executable_dwz() { + if check_option "dwz" "y"; then + if ! type -p dwz>/dev/null; then + error "$(gettext "Cannot find the %s binary required to compress dwarf files.")" "dwz" + return 1 + fi + fi +} diff --git a/scripts/libmakepkg/executable/sepdebugcrcfix.sh.in b/scripts/libmakepkg/executable/sepdebugcrcfix.sh.in new file mode 100644 index 00000000..6259ac32 --- /dev/null +++ b/scripts/libmakepkg/executable/sepdebugcrcfix.sh.in @@ -0,0 +1,38 @@ +#!/usr/bin/bash +# +# sepdebugcrcfix.sh - Confirm presence of sepdebugcrcfix binary +# +# Copyright (c) 2011-2022 Pacman Development Team <pacman-dev(a)lists.archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_EXECUTABLE_SEPDEBUGCRCFIX_SH" ]] && return +LIBMAKEPKG_EXECUTABLE_SEPDEBUGCRCFIX_SH=1 + +LIBRARY=${LIBRARY:-'@libmakepkgdir@'} + +source "$LIBRARY/util/message.sh" +source "$LIBRARY/util/option.sh" + +executable_functions+=('executable_sepdebugcrcfix') + +executable_sepdebugcrcfix() { + if check_option "dwz" "y"; then + if ! type -p sepdebugcrcfix>/dev/null; then + error "$(gettext "Cannot find the %s binary required to fix CRC.")" "sepdebugcrcfix" + return 1 + fi + fi +} diff --git a/scripts/libmakepkg/tidy/strip.sh.in b/scripts/libmakepkg/tidy/strip.sh.in index 688bcf1b..8504227d 100644 --- a/scripts/libmakepkg/tidy/strip.sh.in +++ b/scripts/libmakepkg/tidy/strip.sh.in @@ -27,7 +27,7 @@ source "$LIBRARY/util/message.sh" source "$LIBRARY/util/option.sh" -packaging_options+=('strip' 'debug') +packaging_options+=('strip' 'debug' 'dwz') tidy_modify+=('tidy_strip') @@ -52,6 +52,26 @@ source_files() { | sort -zu | tr '\0' '\n' } +compress_dwarf_files(){ + binary_files="$@" + readarray dwz_files < <(find "$dbgdir" -type f -name \*.debug | LC_ALL=C sort) + size_before=$(du -sk "$dbgdir" | cut -f1) + + # The dwz options are taken from rpm/debugedit + LANG=C dwz --hardlink \ + --quiet \ + --relative \ + --low-mem-die-limit 10000000 \ + --max-die-limit 50000000 \ + ${dwz_files[@]} + + size_after=$(du -sk "$dbgdir" | cut -f1) + msg2 "$(gettext "Original debug info size: ${size_before}kB, size after compression: ${size_after}kB")" + + # Some debuggers might use the CRC to find relevant debug files instead of the build-id + LANG=C sepdebugcrcfix "$dbgdir" $binary_files &>/dev/null +} + strip_file() { local binary=$1; shift @@ -142,7 +162,8 @@ tidy_strip() { fi local binary strip_flags - find . -type f -perm -u+w -print0 2>/dev/null | while IFS= read -rd '' binary ; do + declare -a binary_files + while IFS= read -rd '' binary ; do local STRIPLTO=0 case "$(LC_ALL=C readelf -h "$binary" 2>/dev/null)" in *Type:*'DYN (Shared object file)'*) # Libraries (.so) or Relocatable binaries @@ -166,6 +187,12 @@ tidy_strip() { esac strip_file "$binary" ${strip_flags} (( STRIPLTO )) && strip_lto "$binary" - done + binary_files+=("$binary") + done < <(find . -type f -perm -u+w -print0 2>/dev/null) + + if check_option "debug" "y" && check_option "dwz" "y"; then + msg2 "$(gettext "Compressign dwarf files...")" + compress_dwarf_files ${binary_files[@]} + fi fi } -- 2.36.1

1 0

[PATCH] doc/PKGBUILD: remove Arch Linux specific advise
by Allan McRae 18 May '22

18 May '22

The documentation for the license array was specific to Arch Linux. Remove it and some minor other Arch Linux specific references. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- doc/PKGBUILD.5.asciidoc | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/doc/PKGBUILD.5.asciidoc b/doc/PKGBUILD.5.asciidoc index 4ca8eb3b..1d70682c 100644 --- a/doc/PKGBUILD.5.asciidoc +++ b/doc/PKGBUILD.5.asciidoc @@ -3,7 +3,7 @@ PKGBUILD(5) Name ---- -PKGBUILD - Arch Linux package build description file +PKGBUILD - Package build description file Synopsis @@ -84,14 +84,7 @@ systems (see below). *license (array)*:: This field specifies the license(s) that apply to the package. - Commonly used licenses can be found in '/usr/share/licenses/common'. If you - see the package's license there, simply reference it in the license - field (e.g., `license=('GPL')`). If the package provides a license not - available in '/usr/share/licenses/common', then you should include it - in the package itself and set `license=('custom')` or - `license=('custom:LicenseName')`. The license should be placed in - '$pkgdir/usr/share/licenses/$pkgname/' when building the package. If - multiple licenses are applicable, list all of them: + If multiple licenses are applicable, list all of them: `license=('GPL' 'FDL')`. *install*:: @@ -515,8 +508,7 @@ The source URL is divided into four components: Example ------- The following is an example PKGBUILD for the 'patch' package. For more -examples, look through the build files of your distribution's packages. For -those using Arch Linux, consult the Arch Build System (ABS) tree. +examples, look through the build files of your distribution's packages. [source,sh] ------------------------------- -- 2.36.1

1 0

[PATCH] libmakepkg: Do not "strip" files when the format is not recognised
by Allan McRae 11 May '22

11 May '22

We use a multi step process during stripping to ensure permissions do not get changed. However, if the initial objcopy fails, the subsequent cat results in a blank file. Abandon early if objcopy fails. Fixes FS#74486 Signed-off-by: Allan McRae <allan(a)archlinux.org> --- scripts/libmakepkg/tidy/strip.sh.in | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/scripts/libmakepkg/tidy/strip.sh.in b/scripts/libmakepkg/tidy/strip.sh.in index 688bcf1b..4cf2107c 100644 --- a/scripts/libmakepkg/tidy/strip.sh.in +++ b/scripts/libmakepkg/tidy/strip.sh.in @@ -80,7 +80,12 @@ strip_file() { # copy debug symbols to debug directory mkdir -p "$dbgdir/${binary%/*}" - objcopy --only-keep-debug "$binary" "$dbgdir/$binary.debug" + + # abandon processing files that are not a recognised format + if ! objcopy --only-keep-debug "$binary" "$dbgdir/$binary.debug" 2>/dev/null; then + return + fi + local tempfile=$(mktemp "$binary.XXXXXX") objcopy --add-gnu-debuglink="$dbgdir/${binary#/}.debug" "$binary" "$tempfile" cat "$tempfile" > "$binary" -- 2.36.1

1 0

new Delta Update support
by Ruben Kelevra 08 May '22

08 May '22

Hey guys, previously pacman have had a delta update functionality which was (to my understanding) removed because of safety concerns and poor performance. Since the old delta version was written (and removed) zstd got the ability to create delta patches. I was running some tests on that, and it seems super promising in terms of performance for creation as well as applying those patches, and the deduplication ratio is phenomenal. I created the patches on an arch-vm running on an AMD EPYC 7702P with 10 unshared cores and 32 GB memory, the storage is an SSD RAID10. # Example: Widelands It got 731M version 1.0-1 and 4 successive updates within ~1 year with just pkgrel increasing. Total for the 4 additional versions is 2.9G. Creating all 4 delta patches: 1->5 2->5 3->5 4->5 It would take a total of real 0m24.383s usr 0m19.814s sys 0m5.859s and all patches combined would be 39M. I tested out applying them on a very old netbook (the slowest device I got on hand) 2 GB memory, Intel Atom x5-Z8300 and Arch on an MMC-SSD. The 4->5 patch for example takes real 1m32,850s user 0m2,325s sys 0m16,562s. On a more modern system (Intel Core i5-1135G7 / 16 GB memory / NVMe SSD) this take just real 0m1.208s user 0m0.255s sys 0m0.748s. Downloading the full package on the other hand over LTE takes 9m22s - downloading just the delta patch take 0m5s. # Other examples While I chose a fairly big package with (obviously) low amount of changes between the pkgrel versions, here are some other examples: libreoffice-fresh-7.2.5-5 to 7.3.0-1 saving 40% libreoffice-fresh-7.3.0-1 to 7.3.0-2 saving 45% 0ad-data-a25-1 to a25-2 saving 99.96% * 0ad-data-a25-2 to a25.b-1 saving 99.84% * glibc-2.35-3 to 2.35-4 saving 70% glibc-2.32-5 to 2.35-1 saving 51.8% opencv-4.5.5-4 to opencv-4.5.5-5 saving 93.3% opencv-4.5.4-9 to 4.5.5-1 saving 37% * had to split the tar archive after 1.6 GB and make patches for each part since zstd can only handle 2 GB files. # What would need to be done, to get this going? Well, the packages in the repo need to get a second signature, for the uncompressed tar package. Pacman could first try to fetch this tar signature. If it's on the server, the server supports delta updates – if not, the full update would be loaded. The database files would need to be extended to include the signatures for the uncompressed tar archives as well as the signature. Now pacman can fetch the patch file which fits for it's stored version to the latest version, decompressed the package stored locally and applies the patch file to it. Then pacman would check the signature/checksum of the resulting tar archive, read it and discard the uncompressed files afterwards. # Caveats This would obviously result in the pkg cache containing a full package file and 1 full then, 1->2 delta, 2->3 delta, 3->4 delta over time. This could be cleaned up by calculating the last version which should be stored and compress the file locally and store it with a dedicated extension to not clash with the regular packages (and their signatures). # Database files This could also work for database updates, obviously. But it would need a bit more work as the database files would need to be versioned (or maybe the timestamp is enough?). On a daily update of the community db, we could for example save 88.3% (2022-05-06 to 2022-05-07). On inter-daily updates, this would be down to just a couple of K: The last update of the community repo was just 40K as patch-file. That means saving 99.4% while applying only takes 0m0.075s. --- Hope you that's interesting and a thing you could look into :) Best regards, Ruben

2 2