March 2016 - Pacman-dev - lists.archlinux.org

[pacman-dev] [PATCH] libmakepkg: simplify removal of empty directories
by Allan McRae 20 Mar '16

20 Mar '16

The find command to remove empty directories can sometimes fail with newer versions of findutils (FS#48515). This can be replaced by a more simple command using the -empty and -delete flags. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- For pacman-5.0.2 scripts/libmakepkg/tidy/emptydirs.sh.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/libmakepkg/tidy/emptydirs.sh.in b/scripts/libmakepkg/tidy/emptydirs.sh.in index 32b8d8c..8afc9cc 100644 --- a/scripts/libmakepkg/tidy/emptydirs.sh.in +++ b/scripts/libmakepkg/tidy/emptydirs.sh.in @@ -33,6 +33,6 @@ tidy_remove+=('tidy_emptydirs') tidy_emptydirs() { if check_option "emptydirs" "n"; then msg2 "$(gettext "Removing empty directories...")" - find . -depth -type d -exec rmdir '{}' + 2>/dev/null + find . -type d -empty -delete fi } -- 2.7.3

2 1

[pacman-dev] [PATCH] Do not add root prefix twice when checking database files
by Allan McRae 20 Mar '16

20 Mar '16

When checking .INSTALL and .CHANGELOG files in the mtree file, we need to find the path they are stored in the local database. This was appending the root prefix twice as alpm_option_get_dbpath already includes the root path. While fixing that issue I added checks that the paths for the database files were not longer than PATH_MAX. Fixes FS#48563. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- For backport to 5.0.2. src/pacman/check.c | 58 ++++++++++++++++++++++++++++++++---------------------- 1 file changed, 34 insertions(+), 24 deletions(-) diff --git a/src/pacman/check.c b/src/pacman/check.c index 0fe74e8..86c25c8 100644 --- a/src/pacman/check.c +++ b/src/pacman/check.c @@ -254,7 +254,6 @@ int check_pkg_full(alpm_pkg_t *pkg) const char *root, *pkgname; size_t errors = 0; size_t rootlen; - char filepath[PATH_MAX]; struct archive *mtree; struct archive_entry *entry = NULL; size_t file_count = 0; @@ -267,7 +266,6 @@ int check_pkg_full(alpm_pkg_t *pkg) pm_printf(ALPM_LOG_ERROR, _("path too long: %s%s\n"), root, ""); return 1; } - strcpy(filepath, root); pkgname = alpm_pkg_get_name(pkg); mtree = alpm_pkg_mtree_open(pkg); @@ -282,6 +280,8 @@ int check_pkg_full(alpm_pkg_t *pkg) while(alpm_pkg_mtree_next(pkg, mtree, &entry) == ARCHIVE_OK) { struct stat st; const char *path = archive_entry_pathname(entry); + char filepath[PATH_MAX]; + int filepath_len; mode_t type; size_t file_errors = 0; int backup = 0; @@ -292,32 +292,42 @@ int check_pkg_full(alpm_pkg_t *pkg) path += 2; } - if(strcmp(path, ".INSTALL") == 0) { - char filename[PATH_MAX]; - snprintf(filename, PATH_MAX, "%slocal/%s-%s/install", - alpm_option_get_dbpath(config->handle) + 1, - pkgname, alpm_pkg_get_version(pkg)); - archive_entry_set_pathname(entry, filename); - path = archive_entry_pathname(entry); - } else if(strcmp(path, ".CHANGELOG") == 0) { - char filename[PATH_MAX]; - snprintf(filename, PATH_MAX, "%slocal/%s-%s/changelog", - alpm_option_get_dbpath(config->handle) + 1, - pkgname, alpm_pkg_get_version(pkg)); - archive_entry_set_pathname(entry, filename); - path = archive_entry_pathname(entry); - } else if(*path == '.') { - continue; + if(*path == '.') { + if(strcmp(path, ".INSTALL") == 0) { + char filename[PATH_MAX]; + snprintf(filename, PATH_MAX, "%slocal/%s-%s/install", + alpm_option_get_dbpath(config->handle) + 1, + pkgname, alpm_pkg_get_version(pkg)); + archive_entry_set_pathname(entry, filename); + path = archive_entry_pathname(entry); + } else if(strcmp(path, ".CHANGELOG") == 0) { + char filename[PATH_MAX]; + snprintf(filename, PATH_MAX, "%slocal/%s-%s/changelog", + alpm_option_get_dbpath(config->handle) + 1, + pkgname, alpm_pkg_get_version(pkg)); + archive_entry_set_pathname(entry, filename); + path = archive_entry_pathname(entry); + } else { + continue; + } + + /* Do not append root directory as alpm_option_get_dbpath already + * includes it */ + filepath_len = snprintf(filepath, PATH_MAX, "/%s", archive_entry_pathname(entry)); + if(filepath_len >= PATH_MAX) { + pm_printf(ALPM_LOG_WARNING, _("path too long: %s%s\n"), root, path); + continue; + } + } else { + filepath_len = snprintf(filepath, PATH_MAX, "%s%s", root, path); + if(filepath_len >= PATH_MAX) { + pm_printf(ALPM_LOG_WARNING, _("path too long: %s%s\n"), root, path); + continue; + } } file_count++; - if(rootlen + 1 + strlen(path) > PATH_MAX) { - pm_printf(ALPM_LOG_WARNING, _("path too long: %s%s\n"), root, path); - continue; - } - strcpy(filepath + rootlen, path); - exists = check_file_exists(pkgname, filepath, rootlen, &st); if(exists == 1) { errors++; -- 2.7.3

2 1

Re: [pacman-dev] LTO and PGO build options
by Que Quotion 20 Mar '16

20 Mar '16

>What is the advantage to all this beyond people adding "-flto" to their CFLAGS if they want LTO? The advantage is not having to do that. Theoretically, the compiler devs expect LTO to always work and want reports when it does not. We should be able to enable LTO for every build, but my experience is that--occasionally--the linker fails. Users so inclined could "set it and forget it" most of the time, and occasionally disable it if a package does not build. I find this more efficient than modifying the PKGBUILD of each package that could benefit from LTO. >All modern builds should handle the rest by themselves. Should, but don't last time I checked. A number of builds failed for not having {AR,RANLIB,NM}FLAGS set when attempting LTO. I see the complexity as another reason to have this built into makepkg instead of having to do all this tediousness on a per-package basis. The less anyone has to edit PKGBUILDs for generic things like optimization, the better I think. >I'm guessing this is untested... That's kinda like.. not testing it, yeah? I've been using this for quite some time. Nearly everything compiles successfully with pgo. There were a couple of wrapper leftovers to fix in my initial post, see below. The problem with only me doing the testing is no one will ever beleive my success. I have been successful, believe it--or try it for yourself. >Anyway, I don't think PGO has a place in makepkg itself Why not, specifically? We have other cflag stuff in there; besides, it should be disabled by default so users could opt-in when they feel ready. Do you know you can PGO just about any package in the repositories? It's still takes two build steps, but this macro saves a lot of tediousness over editing the individual PKGBUILDS. I don't see how it's any more or less worthy than ccache, upx, etc. As for the particular cflags I've used for PGO and LTO, these are *known to work* but may not be the only options, might be overkill, or there could even be a few more minor flags worth setting. I'd appreciate some commentary on this matter, and some tests to prove or disprove which flags are required. "-fwhopr" might be a safter idea than "-flto", to avoid the compiler running out of memory and crashing while compiling large programs. New patchset: In makepkg, "-Wl,-option" must be at the end of LDFLAGS: --- makepkg 2016-02-23 11:52:34.000000000 +0900 +++ makepkg 2016-03-20 16:57:07.260935565 +0900 @@ -866,6 +866,32 @@ export DISTCC_HOSTS fi + + # Use lto if it is requested (check buildenv and PKGBUILD opts) + if check_buildoption "lto" "y" && [[ -f "$(gcc -print-search-dirs | grep install | awk '{print $2 "liblto_plugin.so"}')" ]]; then + CFLAGS+=" -flto=$(getconf _NPROCESSORS_ONLN)" + CXXFLAGS+=" -flto=$(getconf _NPROCESSORS_ONLN)" + LDFLAGS+=" $CFLAGS $CXXFLAGS -Wl,-fuse-linker-plugin" + LTOPLUGIN="$(gcc -print-search-dirs | grep install | awk '{print $2 "liblto_plugin.so"}')" + ARFLAGS+=" --plugin $LTOPLUGIN" + RANLIBFLAGS+=" --plugin $LTOPLUGIN" + NMFLAGS+=" --plugin $LTOPLUGIN" + fi + + # Generate or utilize pgo if it is requested (check buildenv and PKGBUILD opts) + if check_buildoption "pgo" "y"; then + if [ ! -d "$PROFDEST/$pkgbase.gen" ]; then + mkdir "$PROFDEST/$pkgbase.gen" + CFLAGS+=" -fprofile-generate -fprofile-dir=$PROFDEST/$pkgbase.gen" + CXXFLAGS+=" -fprofile-generate -fprofile-dir=$PROFDEST/$pkgbase.gen" + LDFLAGS+=" -Wl,-lgcov" + elif [ ! -d "$PROFDEST/$pkgbase.used" ]; then + mv "$PROFDEST/$pkgbase.gen" "$PROFDEST/$pkgbase.used" + CFLAGS+=" -fprofile-correction -fprofile-use -fprofile-dir=$PROFDEST/$pkgbase.used" + CXXFLAGS+=" -fprofile-correction -fprofile-use -fprofile-dir=$PROFDEST/$pkgbase.used" + fi + fi + run_function_safe "build" } @@ -1865,6 +2028,7 @@ [[ -n ${PKGDEST} ]] && _PKGDEST=$(canonicalize_path ${PKGDEST}) [[ -n ${SRCDEST} ]] && _SRCDEST=$(canonicalize_path ${SRCDEST}) [[ -n ${SRCPKGDEST} ]] && _SRCPKGDEST=$(canonicalize_path ${SRCPKGDEST}) +[[ -n ${PROFDEST} ]] && _PROFDEST=$(canonicalize_path ${PROFDEST}) [[ -n ${LOGDEST} ]] && _LOGDEST=$(canonicalize_path ${LOGDEST}) [[ -n ${BUILDDIR} ]] && _BUILDDIR=$(canonicalize_path ${BUILDDIR}) [[ -n ${PKGEXT} ]] && _PKGEXT=${PKGEXT} @@ -1961,6 +2125,14 @@ IGNOREARCH=1 fi +PROFDEST=${_PROFDEST:-$PROFDEST} +PROFDEST=${PROFDEST:-$startdir} #default to $startdir if undefined +if [[ ! -w $PROFDEST ]] ; then + error "$(gettext "You do not have write permission to store profiles in %s.")" "$PROFDEST" + plain "$(gettext "Aborting...")" + exit 1 +fi + LOGDEST=${_LOGDEST:-$LOGDEST} LOGDEST=${LOGDEST:-$startdir} #default to $startdir if undefined if (( LOGGING )) && [[ ! -w $LOGDEST ]]; then In makepkg.conf, I should have specified lto and pgo as BUILDENV options, not package options: --- makepkg.conf 2016-03-20 17:07:12.708226549 +0900 +++ makepkg.conf 2016-03-20 17:12:09.358079792 +0900 @@ -83,6 +83,8 @@ #-- ccache: Use ccache to cache compilation #-- check: Run the check() function if present in the PKGBUILD #-- sign: Generate PGP signature file +#-- lto: Use link-time optimization to reduce compiled binary size and possibly improve performance +#-- pgo: Generate or use profile guided optimization to reduce compile binary size and improve performance # BUILDENV=(!distcc color !ccache !check !sign) # @@ -142,6 +144,8 @@ #SRCDEST=/home/sources #-- Source packages: specify a fixed directory where all src packages will be placed #SRCPKGDEST=/home/srcpackages +#-- Profile cache: specify a fixed directory where profiles will be stored +#PROFDEST=/home/sources #-- Log files: specify a fixed directory where all log files will be placed #LOGDEST=/home/makepkglogs #-- Packager: name/email of the person or organization building packages I have tested these patches. They work for me and have no ill side effects.

2 4

[pacman-dev] [WIP 0/4] Dynamic Users
by Andrew Gregory 20 Mar '16

20 Mar '16

Packages with files owned by dynamically created users/groups have to create the user and chown the files in a post_install/post_upgrade install script. This makes pacman's database out of sync with the filesystem. See FS#43484. This patchset makes alpm/pacman aware of symbolic usernames and uses fakeuser (https://github.com/progandy/fakeuser) to provide a method for PKGBUILDs to add fake users. alpm will still fall back to the raw uid if it has not already been assigned, so packages that add users in post_install/post_upgrade will continue to work correctly, but should be changed to use pre_install/pre_upgrade instead. The makepkg portions should be complete, other than documentation. The alpm/pacman portions still need to be optimized to use a cache so that we don't have to consult /etc/{passwd,group} for every single file. *NOTE*: there is a bug in libarchive that prevents the symlink check from working with this patchset. This will be fixed in the next release. PKGBUILD ======== pkgname=dynamic-users-test pkgver=1 pkgrel=1 arch=(any) install=dynamic-users-test.install # entries are in passwd(5)/group(5) format, empty trailing fields can be omitted sysusers=('myuser') sysgroups=('mygroup') package() { touch "$pkgdir/testfile" chown myuser:mygroup "$pkgdir/testfile" } dynamic-users-test.install ========================== pre_install() { getent passwd myuser &>/dev/null || useradd --system myuser getent group mygroup &>/dev/null || groupadd --system mygroup } pre_upgrade() { pre_install } Andrew Gregory (4): use custom uid/gid lookup functions makepkg add sysusers during package() makepkg: store symbolic user/group names in mtree check_file_permissions: use symbolic user/group names if available lib/libalpm/add.c | 32 ++++++++++++++++++- scripts/Makefile.am | 2 ++ scripts/libmakepkg/.gitignore | 2 ++ scripts/libmakepkg/sysgroups.sh.in | 60 +++++++++++++++++++++++++++++++++++ scripts/libmakepkg/sysusers.sh.in | 65 ++++++++++++++++++++++++++++++++++++++ scripts/makepkg.sh.in | 6 ++-- src/pacman/check.c | 46 +++++++++++++++++++++------ 7 files changed, 200 insertions(+), 13 deletions(-) create mode 100644 scripts/libmakepkg/sysgroups.sh.in create mode 100644 scripts/libmakepkg/sysusers.sh.in -- 2.7.2

2 6

[pacman-dev] [PATCH] makepkg: do not run prepare() with --noextract
by Evangelos Foutras 18 Mar '16

18 Mar '16

This is partial revert of 8454daa7feef (makepkg: run pkgver() and prepare() with --noextract). Reasoning for the reversion (copied from FS#43498): Running prepare() when --noextract is used no longer allows running 'makepkg -o && makepkg -e' with any PKGBUILD that applies patches in prepare(). [1] Sure there's --noprepare which restores the old behavior, but that's a lot of extra typing for what I believe is a much more common use of --noextract. For OP's use case of doing git bisects, you can specify the commit in the source array and thus skip --noextract since makepkg will checkout the correct commit each time. [1] I often extract the sources using 'makepkg -o', manually edit some source files, and then use 'makepkg -e' to package it (while possibly repeating the edit/package steps). --- doc/makepkg.8.txt | 10 +++++----- scripts/makepkg.sh.in | 6 +++--- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/makepkg.8.txt b/doc/makepkg.8.txt index 6f3f6b0..2dff1b1 100644 --- a/doc/makepkg.8.txt +++ b/doc/makepkg.8.txt @@ -55,11 +55,11 @@ Options installed. *-e, \--noextract*:: - Do not extract source files; use whatever source already exists in the - $srcdir/ directory. This is handy if you want to go into $srcdir/ and - manually patch or tweak code, then make a package out of the result. - Keep in mind that creating a patch may be a better solution to allow - others to use your PKGBUILD. + Do not extract source files or run the prepare() function (if present); + use whatever source already exists in the $srcdir/ directory. This is + handy if you want to go into $srcdir/ and manually patch or tweak code, + then make a package out of the result. Keep in mind that creating a + patch may be a better solution to allow others to use your PKGBUILD. *\--verifysource*:: For each source file in the source array of PKGBUILD, download the file diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 80b95f4..2fa3058 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -2206,6 +2206,9 @@ if (( !REPKG )); then fi extract_sources + if (( PREPAREFUNC )); then + run_prepare + fi fi if (( PKGVERFUNC )); then @@ -2213,9 +2216,6 @@ if (( !REPKG )); then basever=$(get_full_version) check_build_status fi - if (( PREPAREFUNC )); then - run_prepare - fi fi if (( NOBUILD )); then -- 2.7.1

4 8

[pacman-dev] LTO and PGO build options
by Que Quotion 18 Mar '16

18 Mar '16

This orignates from a utility I put together: https://bbs.archlinux.org/viewtopic.php?pid=1451263#p1451263 The script streamlines enabling some build options that makepkg does not yet offer, such as link-time optimization and profile-guided optimization. The above script works, but has limitations in that it is a wrapper script. I propose adding these as build options in makepkg itself. Most packages compile safely with both or either option and the benefits are enormous (especially with PGO). Here are the changes I propose to makepkg, for which I am open to suggestions and hope someone other than myself will test: --- /usr/bin/makepkg 2016-02-26 14:07:15.000000000 +0900 +++ ./makepkg.qq 2016-03-19 03:13:09.535977092 +0900 @@ -48,7 +48,7 @@ LIBRARY=${LIBRARY:-'/usr/share/makepkg'} -build_options=('ccache' 'distcc' 'buildflags' 'makeflags') +build_options=('ccache' 'distcc' 'buildflags' 'makeflags' 'pgo' 'lto') splitpkg_overrides=('pkgdesc' 'arch' 'url' 'license' 'groups' 'depends' 'optdepends' 'provides' 'conflicts' 'replaces' 'backup' 'options' 'install' 'changelog') @@ -866,6 +866,32 @@ export DISTCC_HOSTS fi + + # Use lto if it is requested (check buildenv and PKGBUILD opts) + if check_buildoption "lto" "y" && [[ -f "$(gcc -print-search-dirs | grep install | awk '{print $2 "liblto_plugin.so"}')" ]]; then + CFLAGS+=" -flto=$(getconf _NPROCESSORS_ONLN)" + CXXFLAGS+=" -flto=$(getconf _NPROCESSORS_ONLN)" + LDFLAGS+=" $CFLAGS -fuse-linker-plugin" + LTOPLUGIN="$(gcc -print-search-dirs | grep install | awk '{print $2 "liblto_plugin.so"}')" + ARFLAGS+=" --plugin $LTOPLUGIN" + RANLIBFLAGS+=" --plugin $LTOPLUGIN" + NMFLAGS+=" --plugin $LTOPLUGIN" + fi + + # Generate or utilize pgo if it is requested (check buildenv and PKGBUILD opts) + if check_buildoption "pgo" "y"; then + if [ ! -d "$PROFDEST/$pkgbase.gen" ]; then + mkdir "$PROFDEST/$pkgbase.gen" + CFLAGS+="-fprofile-generate -fprofile-dir=$PROFDEST/$pkgbase.gen" + CXXFLAGS+="-fprofile-generate -fprofile-dir=$PROFDEST/$pkgbase.gen" + LDFLAGS+="-lgcov" + elif [ ! -d "$PROFDEST/$pkgbase.used" ]; then + mv "$PROFDEST/$pkgbase.gen" "$PROFDEST/$pkgbase.used" + CFLAGS+="-fprofile-correction -fprofile-use -fprofile-dir=$PROFDEST/$pkgbase.used" + CXXFLAGS+="-fprofile-correction -fprofile-use -fprofile-dir=$PROFDEST/$pkgbase.used" + fi + fi + run_function_safe "build" } @@ -2002,6 +2028,7 @@ [[ -n ${PKGDEST} ]] && _PKGDEST=$(canonicalize_path ${PKGDEST}) [[ -n ${SRCDEST} ]] && _SRCDEST=$(canonicalize_path ${SRCDEST}) [[ -n ${SRCPKGDEST} ]] && _SRCPKGDEST=$(canonicalize_path ${SRCPKGDEST}) +[[ -n ${PROFDEST} ]] && _PROFDEST=$(canonicalize_path ${PROFDEST}) [[ -n ${LOGDEST} ]] && _LOGDEST=$(canonicalize_path ${LOGDEST}) [[ -n ${BUILDDIR} ]] && _BUILDDIR=$(canonicalize_path ${BUILDDIR}) [[ -n ${PKGEXT} ]] && _PKGEXT=${PKGEXT} @@ -2098,6 +2125,14 @@ IGNOREARCH=1 fi +PROFDEST=${_PROFDEST:-$PROFDEST} +PROFDEST=${PROFDEST:-$startdir} #default to $startdir if undefined +if [[ ! -w $PROFDEST ]] ; then + error "$(gettext "You do not have write permission to store profiles in %s.")" "$PROFDEST" + plain "$(gettext "Aborting...")" + exit 1 +fi + LOGDEST=${_LOGDEST:-$LOGDEST} LOGDEST=${LOGDEST:-$startdir} #default to $startdir if undefined if (( LOGGING )) && [[ ! -w $LOGDEST ]]; then

2 1

Re: [pacman-dev] LTO and PGO build options
by Que Quotion 18 Mar '16

18 Mar '16

The section @@ -48,7 +48,7 @@ above is probably unnecessary; I misunderstood the purpose of that line. Also, some changes to makepkg.conf to fit with the changes to makepkg: --- /etc/makepkg.conf 2016-01-09 02:34:08.007565594 +0900 +++ ./makepkg.conf.qq 2016-03-19 04:19:46.655963264 +0900 @@ -110,6 +110,8 @@ #-- purge: Remove files specified by PURGE_TARGETS #-- upx: Compress binary executable files using UPX #-- debug: Add debugging flags as specified in DEBUG_* variables +#-- lto: Use link-time optimization to reduce compiled binary size and possibly improve performance +#-- pgo: Generate or use profile guided optimization to reduce compile binary size and improve performance # OPTIONS=(strip docs !libtool !staticlibs emptydirs zipman purge upx !debug) @@ -142,6 +144,8 @@ #SRCDEST=/home/sources #-- Source packages: specify a fixed directory where all src packages will be placed #SRCPKGDEST=/home/srcpackages +#-- Profile cache: specify a fixed directory where profiles will be stored +#PROFDEST=/home/sources #-- Log files: specify a fixed directory where all log files will be placed #LOGDEST=/home/makepkglogs #-- Packager: name/email of the person or organization building packages

1 0

[pacman-dev] Fw: new message
by juergen＠hoetzel.info 09 Mar '16

09 Mar '16

Hey! Open message <http://sudarena.intelmedia.ro/sit.php?mejm> juergen(a)hoetzel.info

2 1