May 2022 - Pacman-dev - lists.archlinux.org

[PATCH] debugflags: Ensure we have unique source paths
by Morten Linderud 25 Jun '22

25 Jun '22

From: Morten Linderud <morten(a)linderud.pw> In some cases packages are built outside of a directory which contains pkgname-pkgver, this results in source listing in debug packages having a conflicting path like `/usr/src/debug/build/` which is not ideal. This patch ensures we always include the pkgbase to ensure the paths are unique. Signed-off-by: Morten Linderud <morten(a)linderud.pw> --- scripts/libmakepkg/buildenv/debugflags.sh.in | 6 +++--- scripts/libmakepkg/tidy/strip.sh.in | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/scripts/libmakepkg/buildenv/debugflags.sh.in b/scripts/libmakepkg/buildenv/debugflags.sh.in index 1cb58613..2bdb8b0b 100644 --- a/scripts/libmakepkg/buildenv/debugflags.sh.in +++ b/scripts/libmakepkg/buildenv/debugflags.sh.in @@ -30,9 +30,9 @@ buildenv_functions+=('buildenv_debugflags') buildenv_debugflags() { if check_option "debug" "y" && ! check_option "buildflags" "n"; then - DEBUG_CFLAGS+=" -ffile-prefix-map=$srcdir=${DBGSRCDIR:-/usr/src/debug}" - DEBUG_CXXFLAGS+=" -ffile-prefix-map=$srcdir=${DBGSRCDIR:-/usr/src/debug}" - DEBUG_RUSTFLAGS+=" --remap-path-prefix=$srcdir=${DBGSRCDIR:-/usr/src/debug}" + DEBUG_CFLAGS+=" -ffile-prefix-map=$srcdir=${DBGSRCDIR:-/usr/src/debug}/${pkgbase}" + DEBUG_CXXFLAGS+=" -ffile-prefix-map=$srcdir=${DBGSRCDIR:-/usr/src/debug}/${pkgbase}" + DEBUG_RUSTFLAGS+=" --remap-path-prefix=$srcdir=${DBGSRCDIR:-/usr/src/debug}/${pkgbase}" CFLAGS+=" $DEBUG_CFLAGS" CXXFLAGS+=" $DEBUG_CXXFLAGS" RUSTFLAGS+=" $DEBUG_RUSTFLAGS" diff --git a/scripts/libmakepkg/tidy/strip.sh.in b/scripts/libmakepkg/tidy/strip.sh.in index 688bcf1b..25bc649e 100644 --- a/scripts/libmakepkg/tidy/strip.sh.in +++ b/scripts/libmakepkg/tidy/strip.sh.in @@ -47,7 +47,7 @@ source_files() { LANG=C debugedit --no-recompute-build-id \ --base-dir "${srcdir}" \ - --dest-dir "${dbgsrcdir}" \ + --dest-dir "${dbgsrcdir}/${pkgbase}" \ --list-file /dev/stdout "$1" \ | sort -zu | tr '\0' '\n' } -- 2.36.1

2 1

[PATCH] makepkg: add source signing PGP keys to package if available
by Allan McRae 01 Jun '22

01 Jun '22

Arch Linux is adding source signing PGP keys to their package source tree alongside PKGBUILDs in the form keys/pgp/$fingerprint.asc. As the PGP keyserver infrastructure is a mess, this helps other people validate sources in a PKGBUILD. Add the keys to source packages if found alongside the PKGBUILD. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- I won't be committing this until the relevant Arch devtools patch is accepted so that the keys/pgp/ path is finalised. scripts/makepkg.sh.in | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 69757d03..bddcbe03 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -705,6 +705,16 @@ create_srcpackage() { done pkgname=(${pkgname_backup[@]}) + # add a copy of source PGP signing public keys if availabe in keys/pgp/<fingerprint>.asc + local key + for key in ${validpgpkeys[@]}; do + if [[ -f keys/pgp/$key.asc ]]; then + mkdir -p "${srclinks}/${pkgbase}/keys/pgp/" + ln -s "${startdir}/keys/pgp/$key.asc" "${srclinks}/${pkgbase}/keys/pgp/" + fi + done + + local fullver=$(get_full_version) local pkg_file="$SRCPKGDEST/${pkgbase}-${fullver}${SRCEXT}" -- 2.36.1

2 2

[PATCH] strip: Use dwz to compress dwarf headers
by Morten Linderud 27 May '22

27 May '22

From: Morten Linderud <morten(a)linderud.pw> `dwz` allows us to compress the DWARF files when building debug packages. Potentially shaving off some of the sizes of the symbols we distribute. https://sourceware.org/dwz/ A sample of packages built with dwz; pacman: Original debug info size: 1520kB Size after compression: 1252kB systemd: Original debug info size: 46692kB Size after compression: 41036kB Signed-off-by: Morten Linderud <morten(a)linderud.pw> --- doc/makepkg.conf.5.asciidoc | 3 ++ scripts/libmakepkg/executable/dwz.sh.in | 38 +++++++++++++++++++ .../executable/sepdebugcrcfix.sh.in | 38 +++++++++++++++++++ scripts/libmakepkg/tidy/strip.sh.in | 33 ++++++++++++++-- 4 files changed, 109 insertions(+), 3 deletions(-) create mode 100644 scripts/libmakepkg/executable/dwz.sh.in create mode 100644 scripts/libmakepkg/executable/sepdebugcrcfix.sh.in diff --git a/doc/makepkg.conf.5.asciidoc b/doc/makepkg.conf.5.asciidoc index a0d9a6d4..2e40f27f 100644 --- a/doc/makepkg.conf.5.asciidoc +++ b/doc/makepkg.conf.5.asciidoc @@ -194,6 +194,9 @@ Options DEBUG_CXXFLAGS to their counterpart buildflags. Creates a separate package containing the debug symbols when used with `strip'. + *dwz*;; + Compress dwarf files inside the debug package. + *lto*;; Enable building packages using link time optimization. Adds the flags specified in LTOFLAGS to CFLAGS, CXXFLAGS and LDFLAGS (or diff --git a/scripts/libmakepkg/executable/dwz.sh.in b/scripts/libmakepkg/executable/dwz.sh.in new file mode 100644 index 00000000..a540697c --- /dev/null +++ b/scripts/libmakepkg/executable/dwz.sh.in @@ -0,0 +1,38 @@ +#!/usr/bin/bash +# +# dwz.sh - Confirm presence of dwz binary +# +# Copyright (c) 2011-2022 Pacman Development Team <pacman-dev(a)lists.archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_EXECUTABLE_DWZ_SH" ]] && return +LIBMAKEPKG_EXECUTABLE_DWZ_SH=1 + +LIBRARY=${LIBRARY:-'@libmakepkgdir@'} + +source "$LIBRARY/util/message.sh" +source "$LIBRARY/util/option.sh" + +executable_functions+=('executable_dwz') + +executable_dwz() { + if check_option "dwz" "y"; then + if ! type -p dwz>/dev/null; then + error "$(gettext "Cannot find the %s binary required to compress dwarf files.")" "dwz" + return 1 + fi + fi +} diff --git a/scripts/libmakepkg/executable/sepdebugcrcfix.sh.in b/scripts/libmakepkg/executable/sepdebugcrcfix.sh.in new file mode 100644 index 00000000..6259ac32 --- /dev/null +++ b/scripts/libmakepkg/executable/sepdebugcrcfix.sh.in @@ -0,0 +1,38 @@ +#!/usr/bin/bash +# +# sepdebugcrcfix.sh - Confirm presence of sepdebugcrcfix binary +# +# Copyright (c) 2011-2022 Pacman Development Team <pacman-dev(a)lists.archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_EXECUTABLE_SEPDEBUGCRCFIX_SH" ]] && return +LIBMAKEPKG_EXECUTABLE_SEPDEBUGCRCFIX_SH=1 + +LIBRARY=${LIBRARY:-'@libmakepkgdir@'} + +source "$LIBRARY/util/message.sh" +source "$LIBRARY/util/option.sh" + +executable_functions+=('executable_sepdebugcrcfix') + +executable_sepdebugcrcfix() { + if check_option "dwz" "y"; then + if ! type -p sepdebugcrcfix>/dev/null; then + error "$(gettext "Cannot find the %s binary required to fix CRC.")" "sepdebugcrcfix" + return 1 + fi + fi +} diff --git a/scripts/libmakepkg/tidy/strip.sh.in b/scripts/libmakepkg/tidy/strip.sh.in index 688bcf1b..8504227d 100644 --- a/scripts/libmakepkg/tidy/strip.sh.in +++ b/scripts/libmakepkg/tidy/strip.sh.in @@ -27,7 +27,7 @@ source "$LIBRARY/util/message.sh" source "$LIBRARY/util/option.sh" -packaging_options+=('strip' 'debug') +packaging_options+=('strip' 'debug' 'dwz') tidy_modify+=('tidy_strip') @@ -52,6 +52,26 @@ source_files() { | sort -zu | tr '\0' '\n' } +compress_dwarf_files(){ + binary_files="$@" + readarray dwz_files < <(find "$dbgdir" -type f -name \*.debug | LC_ALL=C sort) + size_before=$(du -sk "$dbgdir" | cut -f1) + + # The dwz options are taken from rpm/debugedit + LANG=C dwz --hardlink \ + --quiet \ + --relative \ + --low-mem-die-limit 10000000 \ + --max-die-limit 50000000 \ + ${dwz_files[@]} + + size_after=$(du -sk "$dbgdir" | cut -f1) + msg2 "$(gettext "Original debug info size: ${size_before}kB, size after compression: ${size_after}kB")" + + # Some debuggers might use the CRC to find relevant debug files instead of the build-id + LANG=C sepdebugcrcfix "$dbgdir" $binary_files &>/dev/null +} + strip_file() { local binary=$1; shift @@ -142,7 +162,8 @@ tidy_strip() { fi local binary strip_flags - find . -type f -perm -u+w -print0 2>/dev/null | while IFS= read -rd '' binary ; do + declare -a binary_files + while IFS= read -rd '' binary ; do local STRIPLTO=0 case "$(LC_ALL=C readelf -h "$binary" 2>/dev/null)" in *Type:*'DYN (Shared object file)'*) # Libraries (.so) or Relocatable binaries @@ -166,6 +187,12 @@ tidy_strip() { esac strip_file "$binary" ${strip_flags} (( STRIPLTO )) && strip_lto "$binary" - done + binary_files+=("$binary") + done < <(find . -type f -perm -u+w -print0 2>/dev/null) + + if check_option "debug" "y" && check_option "dwz" "y"; then + msg2 "$(gettext "Compressign dwarf files...")" + compress_dwarf_files ${binary_files[@]} + fi fi } -- 2.36.1

1 0

[PATCH] doc/PKGBUILD: remove Arch Linux specific advise
by Allan McRae 18 May '22

18 May '22

The documentation for the license array was specific to Arch Linux. Remove it and some minor other Arch Linux specific references. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- doc/PKGBUILD.5.asciidoc | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/doc/PKGBUILD.5.asciidoc b/doc/PKGBUILD.5.asciidoc index 4ca8eb3b..1d70682c 100644 --- a/doc/PKGBUILD.5.asciidoc +++ b/doc/PKGBUILD.5.asciidoc @@ -3,7 +3,7 @@ PKGBUILD(5) Name ---- -PKGBUILD - Arch Linux package build description file +PKGBUILD - Package build description file Synopsis @@ -84,14 +84,7 @@ systems (see below). *license (array)*:: This field specifies the license(s) that apply to the package. - Commonly used licenses can be found in '/usr/share/licenses/common'. If you - see the package's license there, simply reference it in the license - field (e.g., `license=('GPL')`). If the package provides a license not - available in '/usr/share/licenses/common', then you should include it - in the package itself and set `license=('custom')` or - `license=('custom:LicenseName')`. The license should be placed in - '$pkgdir/usr/share/licenses/$pkgname/' when building the package. If - multiple licenses are applicable, list all of them: + If multiple licenses are applicable, list all of them: `license=('GPL' 'FDL')`. *install*:: @@ -515,8 +508,7 @@ The source URL is divided into four components: Example ------- The following is an example PKGBUILD for the 'patch' package. For more -examples, look through the build files of your distribution's packages. For -those using Arch Linux, consult the Arch Build System (ABS) tree. +examples, look through the build files of your distribution's packages. [source,sh] ------------------------------- -- 2.36.1

1 0

[PATCH] libmakepkg: Do not "strip" files when the format is not recognised
by Allan McRae 11 May '22

11 May '22

We use a multi step process during stripping to ensure permissions do not get changed. However, if the initial objcopy fails, the subsequent cat results in a blank file. Abandon early if objcopy fails. Fixes FS#74486 Signed-off-by: Allan McRae <allan(a)archlinux.org> --- scripts/libmakepkg/tidy/strip.sh.in | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/scripts/libmakepkg/tidy/strip.sh.in b/scripts/libmakepkg/tidy/strip.sh.in index 688bcf1b..4cf2107c 100644 --- a/scripts/libmakepkg/tidy/strip.sh.in +++ b/scripts/libmakepkg/tidy/strip.sh.in @@ -80,7 +80,12 @@ strip_file() { # copy debug symbols to debug directory mkdir -p "$dbgdir/${binary%/*}" - objcopy --only-keep-debug "$binary" "$dbgdir/$binary.debug" + + # abandon processing files that are not a recognised format + if ! objcopy --only-keep-debug "$binary" "$dbgdir/$binary.debug" 2>/dev/null; then + return + fi + local tempfile=$(mktemp "$binary.XXXXXX") objcopy --add-gnu-debuglink="$dbgdir/${binary#/}.debug" "$binary" "$tempfile" cat "$tempfile" > "$binary" -- 2.36.1

1 0

new Delta Update support
by Ruben Kelevra 08 May '22

08 May '22

Hey guys, previously pacman have had a delta update functionality which was (to my understanding) removed because of safety concerns and poor performance. Since the old delta version was written (and removed) zstd got the ability to create delta patches. I was running some tests on that, and it seems super promising in terms of performance for creation as well as applying those patches, and the deduplication ratio is phenomenal. I created the patches on an arch-vm running on an AMD EPYC 7702P with 10 unshared cores and 32 GB memory, the storage is an SSD RAID10. # Example: Widelands It got 731M version 1.0-1 and 4 successive updates within ~1 year with just pkgrel increasing. Total for the 4 additional versions is 2.9G. Creating all 4 delta patches: 1->5 2->5 3->5 4->5 It would take a total of real 0m24.383s usr 0m19.814s sys 0m5.859s and all patches combined would be 39M. I tested out applying them on a very old netbook (the slowest device I got on hand) 2 GB memory, Intel Atom x5-Z8300 and Arch on an MMC-SSD. The 4->5 patch for example takes real 1m32,850s user 0m2,325s sys 0m16,562s. On a more modern system (Intel Core i5-1135G7 / 16 GB memory / NVMe SSD) this take just real 0m1.208s user 0m0.255s sys 0m0.748s. Downloading the full package on the other hand over LTE takes 9m22s - downloading just the delta patch take 0m5s. # Other examples While I chose a fairly big package with (obviously) low amount of changes between the pkgrel versions, here are some other examples: libreoffice-fresh-7.2.5-5 to 7.3.0-1 saving 40% libreoffice-fresh-7.3.0-1 to 7.3.0-2 saving 45% 0ad-data-a25-1 to a25-2 saving 99.96% * 0ad-data-a25-2 to a25.b-1 saving 99.84% * glibc-2.35-3 to 2.35-4 saving 70% glibc-2.32-5 to 2.35-1 saving 51.8% opencv-4.5.5-4 to opencv-4.5.5-5 saving 93.3% opencv-4.5.4-9 to 4.5.5-1 saving 37% * had to split the tar archive after 1.6 GB and make patches for each part since zstd can only handle 2 GB files. # What would need to be done, to get this going? Well, the packages in the repo need to get a second signature, for the uncompressed tar package. Pacman could first try to fetch this tar signature. If it's on the server, the server supports delta updates – if not, the full update would be loaded. The database files would need to be extended to include the signatures for the uncompressed tar archives as well as the signature. Now pacman can fetch the patch file which fits for it's stored version to the latest version, decompressed the package stored locally and applies the patch file to it. Then pacman would check the signature/checksum of the resulting tar archive, read it and discard the uncompressed files afterwards. # Caveats This would obviously result in the pkg cache containing a full package file and 1 full then, 1->2 delta, 2->3 delta, 3->4 delta over time. This could be cleaned up by calculating the last version which should be stored and compress the file locally and store it with a dedicated extension to not clash with the regular packages (and their signatures). # Database files This could also work for database updates, obviously. But it would need a bit more work as the database files would need to be versioned (or maybe the timestamp is enough?). On a daily update of the community db, we could for example save 88.3% (2022-05-06 to 2022-05-07). On inter-daily updates, this would be down to just a couple of K: The last update of the community repo was just 40K as patch-file. That means saving 99.4% while applying only takes 0m0.075s. --- Hope you that's interesting and a thing you could look into :) Best regards, Ruben

2 2