- Pacman-dev - lists.archlinux.org

[pacman-dev] Pacman database size study
by Anatol Pomozov 22 Jan '20

22 Jan '20

Hello folks. A few days ago there was a conversation at #pacman-dev IRC channel about the ways to reduce pacman database size. David and few other folks discussed how pgp signatures affect the final compressed size. I decided to use this question for a small personal study. I would like to present its results here. Here are the scripts that I used for the following experiments https://pkgbuild.com/~anatolik/db-size-experiments/ This directory also contains all the generated datafiles that I am going to discuss below. I use today’s community.db as the initial sample (file db.original). The first experiment is to parse db tarfile using the script and then write it back to a file: uncompressed size is 17757184 that is equal to original sample 'zstd -19' compressed size is 4366994 that is 1.0084540990896713 times better than original sample Tar *entries* content is identical to the original file. Uncompressed size is exactly the same. Compressed (zstd -19) size is 0.8% better. It comes from the fact that my script does not set entries user/group value and neither sets tar entries modification time. I am not sure if this information is actually used by pacman. Modification time contains a lot of entropy that compressor does not like. The next experiment is the same database but without ‘%MD5SUM% entries. md5sum is an old hash function algorithm that has known security issues and it cannot be qualified as a cryptographically strong hash function by modern day standards. Experiment 'plain_nomd5': uncompressed size is 17536365 that is 1.0125920622660398 times better than original sample 'zstd -19' compressed size is 4188019 that is 1.0515503869490563 times better than original sample Uncompressed size reduced by 1% and compressed size is 5% smaller. Md5 has a lot of entropy and compressor does not like it. The next experiment is to remove package pgp signatures. Arch repos have detached *.sig files for the packages so let’s see how removing %PGPSIG% affects database size: uncompressed size is 14085120 that is 1.2607051981097783 times better than original sample 'zstd -19' compressed size is 1160912 that is 3.7934942527943547 times better than original sample Uncompressed size is 1.26 times smaller and compressed size is 3.8 times smaller!!!! That’s a big difference. And now db without both md5 and pgpsig: uncompressed size is 13248000 that is 1.3403671497584542 times better than original sample 'zstd -19' compressed size is 1021667 that is 4.310517027563776 times better than original sample Compressed db size is down to 1MB that 4.3 times smaller than the original compressed db!!! Until now we were talking about current db format that is a set of files with plain text format. While this format is flexible and it is easy to add new fields it also introduces a lot of redundancy. I've been always curious how far we can go down with the db file size. What if we remove the repeating information and use some form of ‘stripped down’ format. Something similar to ‘packed’ C structs where only field values are stored. In addition to space effectiveness it is very easy to implement the parser (C, ruby or in any other language), see the scripts I shared above. The downside of such format its inflexibility and it’s harder to achieve forward compatibility. Okay let’s talk about the numbers: Experiment 'packed': uncompressed size is 7317679 that is 2.4266142310970458 times better than original sample 'zstd -19' compressed size is 4145790 that is 1.062261474893808 times better than original sample Packed uncompressed format is 2.42 times better than the tar file solution. That’s how much redundant data our current format contains. But what is really mind-blowing is that compressed format is only 6.2% better than for the plain text. Woah!!! Modern day compressors make wonders. zstd is very good in detecting and compressing all the redundancy in our files. Packed format without md5 field: uncompressed size is 7201183 that is 2.4658703993496625 times better than original sample 'zstd -19' compressed size is 3996814 that is 1.1018558782069918 times better than original sample Now without pgp signatures: uncompressed size is 2954238 that is 6.010749303204413 times better than original sample 'zstd -19' compressed size is 883681 that is 4.983600416892521 times better than original sample Packed format, no md5, no pgp: uncompressed size is 2837742 that is 6.257504734397982 times better than original sample 'zstd -19' compressed size is 763730 that is 5.7663218676757495 times better than original sample The smallest db size I was able to achieve is 764K (packed + compressed) that is 1.34 times better than plain+compressed format w/o md5 and pgp.

3 5

[pacman-dev] [PATCH] Increase maximum database size
by Allan McRae 19 Jan '20

19 Jan '20

We previously has the maximum database size as 25MB. This was set in the days before repos had as many packages as they do now, and before we started distributing files databases. Increase this limit to 128MB. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- So this has been hit in the wild. Manjaro patches their pacman package to allow databases of 32MB, because their [community] repo files database breaks the 25MB limit. But being Manjaro, the patch was never forwarded upstream, just like everything they have ever done. People in Arch are no better. A bug was reported, but some idiot (named Antonio Rojas) closed the bug as "not a bug", because it was not an Arch repo running into the issue. So I only discovered this by seeing a closed bug report. Now, onto the change... this is ~4x bigger than anything seen in the wild currently. Is that enough of an increasse. lib/libalpm/be_sync.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/libalpm/be_sync.c b/lib/libalpm/be_sync.c index 07d2b4ae..a7050290 100644 --- a/lib/libalpm/be_sync.c +++ b/lib/libalpm/be_sync.c @@ -224,7 +224,7 @@ int SYMEXPORT alpm_db_update(int force, alpm_db_t *db) int sig_ret = 0; /* set hard upper limit of 25MiB */ - payload.max_size = 25 * 1024 * 1024; + payload.max_size = 128 * 1024 * 1024; /* print server + filename into a buffer */ len = strlen(server) + strlen(db->treename) + strlen(dbext) + 2; -- 2.25.0

3 6

[pacman-dev] [PATCH] makepkg: make source tarballs work with per-package files containing '$pkgname'
by Eli Schwartz 19 Jan '20

19 Jan '20

Extracting function variables containing arbitrarily scoped variables of arbitrary nature is a disaster, but let's at least cover the common case of using the actual '$pkgname' in an install/changelog file. It's the odd case of actually being basically justified use of disambiguating between the same variable used in multiple different split packages and is even recommended in the PKGBUILD(5) documentation... ... and also, --printsrcinfo already uses and overwrites the variable 'pkgname' in pkgbuild_extract_to_srcinfo, so this "works" in .SRCINFO but doesn't work in .src.tar.gz Fixes FS#64932 Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- scripts/makepkg.sh.in | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index ca3e7459..674e0d87 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -784,13 +784,14 @@ create_srcpackage() { fi done - local i + local pkgname_backup=(${pkgname[@]}) + local i pkgname for i in 'changelog' 'install'; do local file files [[ ${!i} ]] && files+=("${!i}") - for name in "${pkgname[@]}"; do - if extract_function_variable "package_$name" "$i" 0 file; then + for pkgname in "${pkgname_backup[@]}"; do + if extract_function_variable "package_$pkgname" "$i" 0 file; then files+=("$file") fi done @@ -802,6 +803,7 @@ create_srcpackage() { fi done done + pkgname=(${pkgname_backup[@]}) local fullver=$(get_full_version) local pkg_file="$SRCPKGDEST/${pkgbase}-${fullver}${SRCEXT}" -- 2.24.1

2 3

[pacman-dev] [PATCH] Option to enable running makepkg in root
by Saul Tigh 17 Jan '20

17 Jan '20

Dear Pacman devs, Although running makepkg in root is frowned upon and considered dangerous, there are circumstances that one might need to do so. For example, during the bootstrap process of building my own toy gnu/linux distro (which uses pacman/makepkg), I have to run makepkg in root. I have added an option called "ALLOWROOT" to makepkg which when is set to 1 enables running makepkg in root. This option is disable by default and wouldn't cause any issue for the average user of pacman and makepkg. This is my first patch for pacman so criticism is welcome and please let me know how to improve it. >From def5a1cae3513e75aa6447a27f3f22a207ffdc61 Mon Sep 17 00:00:00 2001 From: Saul Tigh <jemzipx(a)gmail.com> Date: Thu, 16 Jan 2020 18:58:27 +0700 Subject: Signed-off-by: Saul Tigh <jemzipx(a)gmail.com> Although running makepkg in root is frowned upon, there are circumstances that one might need to do so. For example, during the bootstrap process of building his toy gnu/linux distro, the author of this patch needs to run makepkg in root. This option is disable by default. It can be enable by setting 'ALLOWROOT' in makepkg to 1. --- scripts/makepkg.sh.in | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index ca3e7459..b2cd8ee4 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -79,6 +79,7 @@ SIGNPKG='' SPLITPKG=0 SOURCEONLY=0 VERIFYSOURCE=0 +ALLOWROOT=0 if [[ -n $SOURCE_DATE_EPOCH ]]; then REPRODUCIBLE=1 @@ -1175,9 +1176,13 @@ fi if (( ! INFAKEROOT )); then if (( EUID == 0 )); then - error "$(gettext "Running %s as root is not allowed as it can cause permanent,\n\ -catastrophic damage to your system.")" "makepkg" - exit $E_ROOT + if (( ALLOWROOT )); then + plain "Running as root is not recommended. Proceed with caution." + else + error "$(gettext "Running %s as root is not allowed as it can cause permanent,\n\ + catastrophic damage to your system.")" "makepkg" + exit $E_ROOT + fi fi else if [[ -z $FAKEROOTKEY ]]; then @@ -1352,6 +1357,10 @@ if (( SOURCEONLY )); then check_source_integrity all cd_safe "$startdir" + # allow makepkg to run in root + if (( ALLOWROOT )); then + unset FAKEROOTKEY + fi enter_fakeroot if [[ $SIGNPKG = 'y' ]]; then -- 2.20.1

3 9

[pacman-dev] [PATCH v2] Option to enable running makepkg in
by Saul Tigh 16 Jan '20

16 Jan '20

Although running makepkg in root is frowned upon, there ae circumstances that one might need to do so. For example, during the bootstrap process of building my own toy gnu/linux distro, I need to run makepkg in root. This option is disabled by default. It can be enabled by setting 'ALLOWROOT' in makepkg to 1. --- scripts/makepkg.sh.in | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index ca3e7459..b2cd8ee4 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -79,6 +79,7 @@ SIGNPKG='' SPLITPKG=0 SOURCEONLY=0 VERIFYSOURCE=0 +ALLOWROOT=0 if [[ -n $SOURCE_DATE_EPOCH ]]; then REPRODUCIBLE=1 @@ -1175,9 +1176,13 @@ fi if (( ! INFAKEROOT )); then if (( EUID == 0 )); then - error "$(gettext "Running %s as root is not allowed as it can cause permanent,\n\ -catastrophic damage to your system.")" "makepkg" - exit $E_ROOT + if (( ALLOWROOT )); then + plain "Running as root is not recommended. Proceed with caution." + else + error "$(gettext "Running %s as root is not allowed as it can cause permanent,\n\ + catastrophic damage to your system.")" "makepkg" + exit $E_ROOT + fi fi else if [[ -z $FAKEROOTKEY ]]; then @@ -1352,6 +1357,10 @@ if (( SOURCEONLY )); then check_source_integrity all cd_safe "$startdir" + # allow makepkg to run in root + if (( ALLOWROOT )); then + unset FAKEROOTKEY + fi enter_fakeroot if [[ $SIGNPKG = 'y' ]]; then -- 2.20.1

1 0

[pacman-dev] [PATCH] Remove unneeded ltmain patch
by Allan McRae 16 Jan '20

16 Jan '20

Many moons ago, libtool was bad - I mean worse than today! It gobbled all --as-needed and we ended up with an overlinked libalpm. This was annoying, particularly when dealing with soname bumps in libraries pacman/libalpm had no business linking to. Luckily we had a fix, stolen from GNOME I believe. And with that fix, we lived in harmony with libtool for many years. Until one day, unbeknownst to us, libtool was "fixed". We kept applying our patch, because it still applied, but it did worse than nothing. It gobbled up our other LDFLAGS, and our libalpm started missing out on RELRO and BIND_NOW. This made the Arch Security Team unhappy. We will make them happy again by stopping the patch. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- autogen.sh | 6 ------ build-aux/ltmain-asneeded.patch | 33 --------------------------------- 2 files changed, 39 deletions(-) delete mode 100755 autogen.sh delete mode 100644 build-aux/ltmain-asneeded.patch diff --git a/autogen.sh b/autogen.sh deleted file mode 100755 index 18fa641d..00000000 --- a/autogen.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/sh -x - -autoreconf -i -patch -d build-aux -Np0 -i ltmain-asneeded.patch - -exit 0 diff --git a/build-aux/ltmain-asneeded.patch b/build-aux/ltmain-asneeded.patch deleted file mode 100644 index af5ae6b2..00000000 --- a/build-aux/ltmain-asneeded.patch +++ /dev/null @@ -1,33 +0,0 @@ ---- ltmain.sh.orig 2013-06-26 14:31:53.472627840 +1000 -+++ ltmain.sh 2013-06-26 14:30:56.137038936 +1000 -@@ -5800,6 +5800,14 @@ - arg=$func_stripname_result - ;; - -+ -Wl,*--as-needed*) -+ deplibs="$deplibs $wl--as-needed" -+ ;; -+ -+ -Wl,*--no-as-needed*) -+ deplibs="$deplibs $wl--no-as-needed" -+ ;; -+ - -Wl,*) - func_stripname '-Wl,' '' "$arg" - args=$func_stripname_result -@@ -6160,6 +6168,15 @@ - lib= - found=no - case $deplib in -+ -Wl,--as-needed|-Wl,--no-as-needed) -+ if test "$linkmode,$pass" = "prog,link"; then -+ compile_deplibs="$deplib $compile_deplibs" -+ finalize_deplibs="$deplib $finalize_deplibs" -+ else -+ deplibs="$deplib $deplibs" -+ fi -+ continue -+ ;; - -mt|-mthreads|-kthread|-Kthread|-pthread|-pthreads|--thread-safe \ - |-threads|-fopenmp|-openmp|-mp|-xopenmp|-omp|-qsmp=*) - if test "$linkmode,$pass" = "prog,link"; then -- 2.25.0

1 0

[pacman-dev] [PATCH] Increase meson test timeout
by Allan McRae 16 Jan '20

16 Jan '20

Running the testsuite using "PACTEST_VALGRIND=1 ninja test -C build", I ran into the following failure: 161/332 smoke001.py TIMEOUT 30.02 s I figure an i7 @ 3.10GHz should be enough to run our testsuite... so boost the meson test timeout to 120 seconds (which should be enough time for anyone...). Signed-off-by: Allan McRae <allan(a)archlinux.org> --- With this patch, I can run the test under valgrind. e.g. 161/332 smoke001.py OK 36.42 s test/pacman/meson.build | 1 + 1 file changed, 1 insertion(+) diff --git a/test/pacman/meson.build b/test/pacman/meson.build index 4e87b4f3..9de4f5a1 100644 --- a/test/pacman/meson.build +++ b/test/pacman/meson.build @@ -352,5 +352,6 @@ foreach input : pacman_tests PYTHON, protocol : 'tap', args : args, + timeout: 120, depends : [pacman_bin]) endforeach -- 2.25.0

1 0

[pacman-dev] [PATCH] autotools: preserve LDFLAGS when building libalpm
by anthraxx＠archlinux.org 16 Jan '20

16 Jan '20

From: Levente Polyak <anthraxx(a)archlinux.org> Use AM_LDFLAGS for libalpm_la so defined LDFLAGS will not be ignored. Signed-off-by: anthraxx <anthraxx(a)archlinux.org> --- lib/libalpm/Makefile.am | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/libalpm/Makefile.am b/lib/libalpm/Makefile.am index e4f83223..06dc0e43 100644 --- a/lib/libalpm/Makefile.am +++ b/lib/libalpm/Makefile.am @@ -59,7 +59,7 @@ libalpm_la_SOURCES = \ util-common.h util-common.c \ version.c -libalpm_la_LDFLAGS = -no-undefined -version-info $(LIB_VERSION_INFO) +libalpm_la_LDFLAGS = $(AM_LDFLAGS) -no-undefined -version-info $(LIB_VERSION_INFO) libalpm_la_CFLAGS = \ $(AM_CFLAGS) \ -- 2.25.0

2 2

[pacman-dev] [PATCH] makepkg: fix one more file-seccomp issue
by Eli Schwartz 10 Jan '20

10 Jan '20

When file is called via fakeroot, it doesn't matter whether you use -z or not, it is still incompatible with seccomp. Fix by configuring it with FILECMD when used in the fakeroot 'tidy' run. Fixes FS#65100 Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- For release/5.2.x only since the call site is gone in master. scripts/libmakepkg/tidy/strip.sh.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/libmakepkg/tidy/strip.sh.in b/scripts/libmakepkg/tidy/strip.sh.in index 1bd810f0..876f00f0 100644 --- a/scripts/libmakepkg/tidy/strip.sh.in +++ b/scripts/libmakepkg/tidy/strip.sh.in @@ -111,7 +111,7 @@ tidy_strip() { local binary strip_flags find . -type f -perm -u+w -print0 2>/dev/null | while IFS= read -rd '' binary ; do - case "$(file -bi "$binary")" in + case "$(@FILECMD@ -bi "$binary")" in *application/x-sharedlib*) # Libraries (.so) strip_flags="$STRIP_SHARED";; *application/x-archive*) # Libraries (.a) -- 2.24.1

1 0

[pacman-dev] [Usage] run pacman without root
by nly＠disroot.org 08 Jan '20

08 Jan '20

Is it possible to use pacman without root, and locally, say `~/pacman` instead of global `/usr'. I built and installed pacman from git sources. $ cd pacman && git checkout v5.2.1 $ autoreconf -vif $ ./configure --prefix=$HOME/pacman/usr/local --enable-static --with-libcurl --with-gpgme $ make $ make install Adding pacman to path, `.bashrc': source /home/nly/pacman/usr/local/etc/profile # (4.) $ pacman -Q nano warning: database file for 'core' does not exist (use '-Sy' to download) error: package 'nano' was not found $ pacman -Sy error: you cannot perform this operation unless you are root. I've linked to the etc/pacman.conf (3.) and etc/pacman.d/mirrorlist (2.). Thanks, Amar ---------------- 1. makepkg-conf: http://nly.info.tm/pacman/makepkg.conf 2. mirrorlist: http://nly.info.tm/pacman/mirrorlist 3. pacman-conf: http://nly.info.tm/pacman/pacman.conf 4. bash-profile: http://nly.info.tm/pacman/profile

1 0