- Pacman-dev - lists.archlinux.org

[pacman-dev] [MAKEPKG][PATCH] makepkg-do-not-count-hard-links-multiple-times
by la.luge＠free.fr 27 Oct '19

27 Oct '19

Uses stat -c '%i %s' and sort -u to count hardlink one time. Gives same result than cat | wc -c without harlinks and good result with hardlinks. Doesn't use filenames

2 2

[pacman-dev] [PATCH] makepkg: do not count hard links multiple times when calculating pkg size
by Eli Schwartz 27 Oct '19

27 Oct '19

Exclude files with hardlinks when cat'ing all the files, and do a second run to look at each file with hardlinks, keep track of the ones we've already operated on, and only cat each inode once. Then use "wc -c" to get the size of all (deduplicated) files the same way we were already doing. Original-patch-by: Ronan Pigott <rpigott(a)berkeley.edu> Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- scripts/Makefile.am | 1 + scripts/libmakepkg/util/dirsize.sh.in | 41 +++++++++++++++++++++++++++ scripts/libmakepkg/util/meson.build | 1 + scripts/makepkg.sh.in | 2 +- 4 files changed, 44 insertions(+), 1 deletion(-) create mode 100644 scripts/libmakepkg/util/dirsize.sh.in diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 88e9612d..63d09767 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -127,6 +127,7 @@ LIBMAKEPKG_IN = \ libmakepkg/util.sh \ libmakepkg/util/compress.sh \ libmakepkg/util/config.sh \ + libmakepkg/util/dirsize.sh \ libmakepkg/util/error.sh \ libmakepkg/util/message.sh \ libmakepkg/util/option.sh \ diff --git a/scripts/libmakepkg/util/dirsize.sh.in b/scripts/libmakepkg/util/dirsize.sh.in new file mode 100644 index 00000000..fce9f5b5 --- /dev/null +++ b/scripts/libmakepkg/util/dirsize.sh.in @@ -0,0 +1,41 @@ +#!/usr/bin/bash +# +# dirsize.sh - calculate size of all files in a directory +# +# Copyright (c) 2019 Pacman Development Team <pacman-dev(a)archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_UTIL_DIRSIZE_SH" ]] && return +LIBMAKEPKG_UTIL_DIRSIZE_SH=1 + + +# find the total filesize of all files in the current directory while only +# counting multiply hardlinked files once +dirsize() { + local file inode + declare -A files + + { + find . -type f -links 1 -exec cat {} + 2>/dev/null + while read -rd ' ' inode; do + IFS= read -r file + if [[ -z ${files[$inode]} ]]; then + files[$inode]=found + cat "$file" + fi + done < <(find . -type f -links +1 -exec @INODECMD@ {} + 2>/dev/null) + } | wc -c +} diff --git a/scripts/libmakepkg/util/meson.build b/scripts/libmakepkg/util/meson.build index c29503b7..6160d87e 100644 --- a/scripts/libmakepkg/util/meson.build +++ b/scripts/libmakepkg/util/meson.build @@ -3,6 +3,7 @@ libmakepkg_module = 'util' sources = [ 'compress.sh.in', 'config.sh.in', + 'dirsize.sh.in' 'error.sh.in', 'message.sh.in', 'option.sh.in', diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 947a30c5..d217d0a4 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -584,7 +584,7 @@ write_kv_pair() { } write_pkginfo() { - local size="$(find . -type f -exec cat {} + 2>/dev/null | wc -c)" + local size=$(dirsize) merge_arch_attrs -- 2.23.0

1 0

[pacman-dev] [PATCH v3 0/1] Correct package size calculation in makepkg
by Ronan Pigott 27 Oct '19

27 Oct '19

From: Ronan Pigott <rpigott(a)berkeley.edu> V3 addresses eschwartz comment that leading trailing whitespace are not handled correctly. With these changes it is no longer necessary to modify INODECMD and other files that use it, so those changes have been removed and another patch can address them if necessary. Filenames that coincide with a valid stat flag option are now handled as well with a -- argument to stat. Ronan Pigott (1): makepkg: do not count hard linked file sizes multiple times scripts/Makefile.am | 1 + scripts/libmakepkg/util/dirsize.sh.in | 45 +++++++++++++++++++++++++++ scripts/libmakepkg/util/meson.build | 1 + scripts/makepkg.sh.in | 2 +- 4 files changed, 48 insertions(+), 1 deletion(-) create mode 100644 scripts/libmakepkg/util/dirsize.sh.in -- 2.23.0

1 1

[pacman-dev] [PATCH] makepkg: protect against unexpected whitespace in filenames
by Eli Schwartz 27 Oct '19

27 Oct '19

zipman: read -r protects against those evil manpages whose filenames contain backslash escapes, (muahahaha?) IFS= read protects against filenames with: - leading whitespace (but no one is actually stupid enough to configure their MAN_DIRS=() in makepkg.conf with such silly directories, *right*?) - trailing whitespace (but likewise, no one should be stupid enough to write an uncompressed manpage for section '1 ' or something) Also fix several other cases where we read filenames without protecting against surrounding whitespace, or without using null-delimited filenames when we could trivially do so. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- unrelated to filesize counting issues, except that filesize counting should be inspired from the fixed code I show here, not from the released zipman scripts/libmakepkg/tidy/staticlibs.sh.in | 2 +- scripts/libmakepkg/tidy/strip.sh.in | 6 +++--- scripts/libmakepkg/tidy/zipman.sh.in | 8 ++++---- scripts/makepkg.sh.in | 6 +++--- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/scripts/libmakepkg/tidy/staticlibs.sh.in b/scripts/libmakepkg/tidy/staticlibs.sh.in index eaa6ac8a..bf7601d2 100644 --- a/scripts/libmakepkg/tidy/staticlibs.sh.in +++ b/scripts/libmakepkg/tidy/staticlibs.sh.in @@ -34,7 +34,7 @@ tidy_staticlibs() { if check_option "staticlibs" "n"; then msg2 "$(gettext "Removing static library files...")" local l - while read -rd '' l; do + while IFS= read -rd '' l; do if [[ -f "${l%.a}.so" || -h "${l%.a}.so" ]]; then rm "$l" fi diff --git a/scripts/libmakepkg/tidy/strip.sh.in b/scripts/libmakepkg/tidy/strip.sh.in index 99bfcada..1bd810f0 100644 --- a/scripts/libmakepkg/tidy/strip.sh.in +++ b/scripts/libmakepkg/tidy/strip.sh.in @@ -57,7 +57,7 @@ strip_file() { # copy source files to debug directory local f t - while read -r t; do + while IFS= read -r t; do f=${t/${dbgsrcdir}/"$srcdir"} mkdir -p "${dbgsrc/"$dbgsrcdir"/}${t%/*}" cp -- "$f" "${dbgsrc/"$dbgsrcdir"/}$t" @@ -69,7 +69,7 @@ strip_file() { objcopy --add-gnu-debuglink="$dbgdir/${binary#/}.debug" "$binary" # create any needed hardlinks - while read -rd '' file ; do + while IFS= read -rd '' file ; do if [[ "${binary}" -ef "${file}" && ! -f "$dbgdir/${file}.debug" ]]; then mkdir -p "$dbgdir/${file%/*}" ln "$dbgdir/${binary}.debug" "$dbgdir/${file}.debug" @@ -110,7 +110,7 @@ tidy_strip() { fi local binary strip_flags - find . -type f -perm -u+w -print0 2>/dev/null | while read -rd '' binary ; do + find . -type f -perm -u+w -print0 2>/dev/null | while IFS= read -rd '' binary ; do case "$(file -bi "$binary")" in *application/x-sharedlib*) # Libraries (.so) strip_flags="$STRIP_SHARED";; diff --git a/scripts/libmakepkg/tidy/zipman.sh.in b/scripts/libmakepkg/tidy/zipman.sh.in index 3c2e261e..ba2f006c 100644 --- a/scripts/libmakepkg/tidy/zipman.sh.in +++ b/scripts/libmakepkg/tidy/zipman.sh.in @@ -35,9 +35,9 @@ tidy_zipman() { msg2 "$(gettext "Compressing man and info pages...")" local file files inode link while read -rd ' ' inode; do - read file - find ${MAN_DIRS[@]} -type l 2>/dev/null | - while read -r link ; do + IFS= read -r file + find "${MAN_DIRS[@]}" -type l -print0 2>/dev/null | + while IFS= read -rd '' link ; do if [[ "${file}" -ef "${link}" ]] ; then rm -f "$link" "${link}.gz" if [[ ${file%/*} = "${link%/*}" ]]; then @@ -55,7 +55,7 @@ tidy_zipman() { ln "${files[$inode]}.gz" "${file}.gz" chmod 644 "${file}.gz" fi - done < <(find ${MAN_DIRS[@]} -type f \! -name "*.gz" \! -name "*.bz2" \ + done < <(find "${MAN_DIRS[@]}" -type f \! -name "*.gz" \! -name "*.bz2" \ -exec @INODECMD@ '{}' + 2>/dev/null) fi } diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 997c8668..947a30c5 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -474,7 +474,7 @@ find_libdepends() { local libdeps filename soarch sofile soname soversion declare -A libdeps - while read -r filename; do + while IFS= read -rd '' filename; do # get architecture of the file; if soarch is empty it's not an ELF binary soarch=$(LC_ALL=C readelf -h "$filename" 2>/dev/null | sed -n 's/.*Class.*ELF$32\|64$/\1/p') [[ -n "$soarch" ]] || continue @@ -495,7 +495,7 @@ find_libdepends() { libdeps[$soname]="${soversion}-${soarch}" fi done - done < <(find "$pkgdir" -type f -perm -u+x) + done < <(find "$pkgdir" -type f -perm -u+x -print0) local libdepends v for d in "${depends[@]}"; do @@ -1320,7 +1320,7 @@ if (( INFAKEROOT )); then else run_split_packaging fi - + create_debug_package msg "$(gettext "Leaving %s environment.")" "fakeroot" -- 2.23.0

1 0

[pacman-dev] [PATCH 0/2] Correct package size calculation in makepkg
by Ronan Pigott 27 Oct '19

27 Oct '19

From: Ronan Pigott <rpigott(a)berkeley.edu> This patchset is a fix for FS#64252 and is mostly the result of some discussion there. The goal is to fix the double counting of hard linked files and also to correct some errors counting the size of certain file names that coincide with valid cat flag options. To make supporting whitespace in filenames easier, I changed INODECMD to output only the inode and instead rely on find's -print0 to reliably parse complete filenames. This fixes some errors for files that begin or end in whitespace. It was noted in discussion that -print0 is not POSIX, but it seems portable regardless. Note it is possible to get a null-delimited list of file names in pure bash using a combination of bash's printf and globbing, if it is necessary. Ronan Pigott (2): makepkg: use null-delimited file lists when tracking inodes makepkg: do not count hard linked file sizes multiple times configure.ac | 6 +++--- meson.build | 6 +++--- scripts/libmakepkg/tidy/zipman.sh.in | 9 +++++---- scripts/makepkg.sh.in | 10 +++++++++- 4 files changed, 20 insertions(+), 11 deletions(-) -- 2.23.0

3 5

[pacman-dev] [PATCH] meson: fix inodecmd for darwin/bsd
by Eli Schwartz 27 Oct '19

27 Oct '19

The BSD stat command uses %N, not %n, and was incorrectly ported to meson. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- meson.build | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/meson.build b/meson.build index c5704efe..1e2a78c3 100644 --- a/meson.build +++ b/meson.build @@ -236,13 +236,13 @@ endif os = host_machine.system() if os.startswith('darwin') - inodecmd = '/usr/bin/stat -f \'%i %n\'' + inodecmd = '/usr/bin/stat -f \'%i %N\'' default_sedinplaceflags = ' -i \'\'' strip_binaries = '' strip_shared = '-s' strip_static = '-s' elif os.contains('bsd') or os == 'dragonfly' - inodecmd = 'stat -f \'%i %n\'' + inodecmd = 'stat -f \'%i %N\'' default_sedinplaceflags = ' -i \'\'' endif -- 2.23.0

1 0

[pacman-dev] [PATCH v2 0/2] Correct package size calculation in makepkg
by Ronan Pigott 27 Oct '19

27 Oct '19

From: Ronan Pigott <rpigott(a)berkeley.edu> V2 implements suggestions by Allan to separate dirsize calculation into a util library function and implement a two-pass approach for hard linked and non hard linked files. Ronan Pigott (2): makepkg: use null-delimited file lists when tracking inodes makepkg: do not count hard linked file sizes multiple times configure.ac | 6 ++-- meson.build | 6 ++-- scripts/Makefile.am | 1 + scripts/libmakepkg/tidy/zipman.sh.in | 9 +++--- scripts/libmakepkg/util/dirsize.sh.in | 45 +++++++++++++++++++++++++++ scripts/libmakepkg/util/meson.build | 1 + scripts/makepkg.sh.in | 2 +- 7 files changed, 59 insertions(+), 11 deletions(-) create mode 100644 scripts/libmakepkg/util/dirsize.sh.in -- 2.23.0

1 2

[pacman-dev] [PATCH] makepkg: do not count hard links multiple times when calculating pkg size
by Allan McRae 26 Oct '19

26 Oct '19

Keeps track of file inodes that we have already counted using the same strategy as we do for zipman. Still uses "cat | wc -c" to calculate file sizes as it is the only thing guaranteed to be consistent. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- scripts/Makefile.am | 1 + scripts/libmakepkg/util/dirsize.sh.in | 39 +++++++++++++++++++++++++++ scripts/libmakepkg/util/meson.build | 1 + scripts/makepkg.sh.in | 2 +- 4 files changed, 42 insertions(+), 1 deletion(-) create mode 100644 scripts/libmakepkg/util/dirsize.sh.in diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 88e9612d..63d09767 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -127,6 +127,7 @@ LIBMAKEPKG_IN = \ libmakepkg/util.sh \ libmakepkg/util/compress.sh \ libmakepkg/util/config.sh \ + libmakepkg/util/dirsize.sh \ libmakepkg/util/error.sh \ libmakepkg/util/message.sh \ libmakepkg/util/option.sh \ diff --git a/scripts/libmakepkg/util/dirsize.sh.in b/scripts/libmakepkg/util/dirsize.sh.in new file mode 100644 index 00000000..ba0783ca --- /dev/null +++ b/scripts/libmakepkg/util/dirsize.sh.in @@ -0,0 +1,39 @@ +#!/usr/bin/bash +# +# dirsize.sh - calculate size of all files in a directory +# +# Copyright (c) 2019 Pacman Development Team <pacman-dev(a)archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_UTIL_DIRSIZE_SH" ]] && return +LIBMAKEPKG_UTIL_DIRSIZE_SH=1 + +LIBRARY=${LIBRARY:-'/usr/share/makepkg'} + + +dirsize() { + local size=0 s file files inode + + while read -rd ' ' inode; do + read file + if [[ -z ${files[$inode]} ]]; then + files[$inode]=$file + size=$(( size + $(cat $file | wc -c) )) + fi + done < <(find . -type f -exec @INODECMD@ '{}' + 2>/dev/null) + + echo $size +} diff --git a/scripts/libmakepkg/util/meson.build b/scripts/libmakepkg/util/meson.build index c29503b7..6160d87e 100644 --- a/scripts/libmakepkg/util/meson.build +++ b/scripts/libmakepkg/util/meson.build @@ -3,6 +3,7 @@ libmakepkg_module = 'util' sources = [ 'compress.sh.in', 'config.sh.in', + 'dirsize.sh.in' 'error.sh.in', 'message.sh.in', 'option.sh.in', diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 997c8668..e75031c2 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -584,7 +584,7 @@ write_kv_pair() { } write_pkginfo() { - local size="$(find . -type f -exec cat {} + 2>/dev/null | wc -c)" + local size=$(dirsize) merge_arch_attrs -- 2.23.0

1 0

[pacman-dev] (no subject)
by arch＠eckner.net 24 Oct '19

24 Oct '19

Since the update to pacman 5.2.0, `makepkg --printsrcinfo` may emmit warnings about invalid packager format. However, all config variables in /etc/makepkg.conf, as well as directory permissions are not relevant if `--printsrcinfo` is given. So we simply skip the lint_conf and directory permisison checks then.

3 3

[pacman-dev] [PATCH] Fix segfault importing PGP key for pacman -U operations
by Allan McRae 24 Oct '19

24 Oct '19

Use after free. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- lib/libalpm/be_package.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/libalpm/be_package.c b/lib/libalpm/be_package.c index 0f45c331..73d466de 100644 --- a/lib/libalpm/be_package.c +++ b/lib/libalpm/be_package.c @@ -759,7 +759,7 @@ int SYMEXPORT alpm_pkg_load(alpm_handle_t *handle, const char *filename, int ful if(_alpm_key_in_keychain(handle, key) == 0) { pkg_temp = _alpm_pkg_load_internal(handle, filename, full); if(pkg_temp) { - packager = pkg_temp->packager; + packager = strdup(pkg_temp->packager); _alpm_pkg_free(pkg_temp); } else { packager = NULL; -- 2.23.0

1 0