April 2013 - Pacman-dev - lists.archlinux.org

[pacman-dev] [PATCH][RFC] Unit tests!
by Andrew Gregory 15 Apr '13

15 Apr '13

Exposing and catching bugs deep in the internals of alpm or pacman can be rather difficult with our current test suite, so I'd like to add a unit test suite. Here's the setup I've come up with so far: * Tests use the check framework [1] (it's in the repos) * Tests should include the source file (not the header) for whatever is being tested. This is necessary because it seems like most of the low hanging fruit (ie functions that don't interact with the user or filesystem) are static. * Test files must define a function "build_suite" that assembles their tests into a suite. * Test files are built as dynamic libraries which are automatically discovered and loaded by the test runner. Check organizes individual tests into suites and then cases and allows individual suites/cases to be selected with an environment variable. Right now my example setup has exactly one case per suite because that's easier, I'm thinking it would be better though to have one case per test so that individual functions can be tested separately. For instance, `CK_RUN_SUITE=util.c ./runner` would test everything in util.c, and `CK_RUN_CASE=util.c/_alpm_strip_newline ./runner` would test only that one function. I've included a test runner and a couple stub tests with a minimal Makefile as an example. With check installed, running `make check` in test/alpm/unit gives: $ make check ./runner Running suite(s): alpm alpm_list util.c 75%: Checks: 4, Failures: 1, Errors: 0 util_test.c:22:F:Core:check_alpm_strip_newline:0: Assertion 'output==expected' failed: output=="test", expected=="this will fail" make: *** [check] Error 1 [1] http://check.sourceforge.net/doc/check_html/index.html Andrew Gregory (1): add unit tests test/alpm/unit/Makefile | 16 +++++++++ test/alpm/unit/alpm_list_test.c | 76 +++++++++++++++++++++++++++++++++++++++++ test/alpm/unit/runner.c | 47 +++++++++++++++++++++++++ test/alpm/unit/util_test.c | 34 ++++++++++++++++++ 4 files changed, 173 insertions(+) create mode 100644 test/alpm/unit/Makefile create mode 100644 test/alpm/unit/alpm_list_test.c create mode 100644 test/alpm/unit/runner.c create mode 100644 test/alpm/unit/util_test.c -- 1.8.2.1

2 2

[pacman-dev] [PATCH] pmpkg: set default mode for files in sync packages
by Andrew Gregory 14 Apr '13

14 Apr '13

TarInfo objects default to mode 0644 while mkfile in util.py uses 0755, causing pacman warnings about differing permissions on tests involving package updates. Set the mode on TarInfo objects to 0755 unless the test specifies a different mode. Bug referenced in FS#30723. Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> --- test/pacman/pmpkg.py | 2 ++ test/pacman/tests/mode001.py | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/test/pacman/pmpkg.py b/test/pacman/pmpkg.py index 988c73f..250d45b 100644 --- a/test/pacman/pmpkg.py +++ b/test/pacman/pmpkg.py @@ -160,6 +160,8 @@ def makepkg(self, path): info = tarfile.TarInfo(fileinfo["filename"]) if fileinfo["hasperms"]: info.mode = fileinfo["perms"] + else: + info.mode = 0755 if fileinfo["isdir"]: info.type = tarfile.DIRTYPE tar.addfile(info) diff --git a/test/pacman/tests/mode001.py b/test/pacman/tests/mode001.py index 4ec11e1..db886e7 100644 --- a/test/pacman/tests/mode001.py +++ b/test/pacman/tests/mode001.py @@ -9,4 +9,4 @@ self.addrule("PACMAN_RETCODE=0") for f in p.files: - self.addrule("FILE_MODE=%s|644" % f) + self.addrule("FILE_MODE=%s|755" % f) -- 1.8.2

2 3

[pacman-dev] code reuse for PKGBUILD
by Kent Fredric 12 Apr '13

12 Apr '13

So, I'm sort of new to Arch, but I've already found some things in how a few things work I have rather sizable miss-givings about, and see room for improvement. *~~~ BEGIN CONFUSING RANT HERE ~~~* *( or skip past it to the TL;DR )* The primary one at present is "No code re-use in PKGBUILD. Period" And this strikes me as a bad idea. I understand that having a PKGBUILD become not fully independent might be undesirable to some, but I'm hoping there is a pivotal middle-ground that can be useful. 1. Many PKGBUILD files are seriously outdated 2. Each and every Perl Module PKGBUILD requires the author to know *everything* about packaging Perl modules, and this is a somewhat hard thing to know: - version normalisation # I haven't seen anyone do it yet, and its the only sure-fire mechanism to get versions to play nicely together. - lots of boilerplate , cargo-cult copypasted around, see : https://aur.archlinux.org/packages/pe/perl-json-any/PKGBUILD Consider this for a moment: Gentoo, like Arch have been nuking '.packlist' files for some time, until recently it became apparent that some programs actually need those files to do their job. https://bugs.gentoo.org/show_bug.cgi?id=438660 This is "bad practice", and because the behaviour is not codified in some shared library that all perl module packages use, in order to "Fix" this problem on arch, it requires a considerable number of person-hours to vet the hundreds of perl PKGBUILDS to eliminate this behaviour. https://aur.archlinux.org/packages/pe/perl-devel-nytprof/PKGBUILD Another bad example of people reinventing all their own wheels, and doing all make/make test/make install during build. This behaviour should really be codified in a shared module. That way, when people realise the way the shared module has been doing things is for whatever reason, wrong, its simply a matter of fixing that, and every build past and present benefits from the change, with no additional effort on the PKGBUILD maintainers. What I hope to do is find some sort of consensus on how it would be best to add a little bit of code-reuse to pkgbuild, as this will, I feel, greatly improve maintenance of these files. Because *both* of those 2 PKGBUILD files I posted could have been easily refactored, and had *100%* of their independent code eliminated in favour of shared behaviour. extensions=('perl-module') _perl_distname=json-any _perl_ver='1.29' _perl_author=PERIGRIN pkgver='1.290.0' pkgrel='1' pkgdesc="Wrapper Class for the various JSON classes." arch=('any') options=('!emptydirs') depends=('perl-json>=2.02' 'perl-json-xs>=2.3' 'perl-yaml-syck') makedepends=() md5sums=('f7eea523d532668555456e4153334342') Notes: _perl_ver and pkgver are different due to my recommendation that upstream versions undergo a normalisation process prior to being used for arch versions. This is important, as perl versioning semantics are wildly different from the rest of the worlds. The average packager will not even realise how big a deal this is, but simple to say, perl versions are a nightmare: http://www.dagolden.com/index.php/369/version-numbers-should-be-boring/ On Gentoo, we've developed a tool to make it easier to convert perl versions to versions we can use in packages, and it basically "Just works". https://metacpan.org/module/Gentoo::PerlMod::Version license= is omitted, as this way, it can default to the most common license on CPAN, "The Perl5 License" -> license=('PerlArtistic' 'GPL') and can be overridden as needed. url= is omitted, as this can easily be generated from the _perl_distname value, and overridden only when necessary. This means you can either have the underlying module refer to pages on search.cpan.org, or metacpan.org source=() is also omitted, as the primary src URI can also be divined from combining _perl_distname , _perl_author and _perl_ver and you can usually leave out build(), check() and package(), as these phases are all very well known and have to meet a standard behaviour to operate with 'cpan' clients, so its quite straightforward to have a set of default behaviours. Implementation wise, my biggest question is "Where is 'perl-module' stored". The simplest option is a pacman distribution, such as 'pacman-extensions', that installs files in /usr/lib/pacman/extension/ , and makepkg/pacman can be made aware of this feature and load the modules on demand. For those who don't like the inherent issue where "PKGBUILD depends on code outside tar.gz", it could be engineered so that `makepkg -S` copies files from /usr/lib/pacman/extension/ into the built tree at /extensions/ , and then have some sort of mechanic that chooses between using the "system" version of the extension or the "bundled" version of the extension. With built packages, the behaviours from the extension would probably have to be cooked in to the dist ( somehow ), but thats an accepted caveat of how binary distribution usually works. *========= TL;DR TL;DR TL;DR TL;DR TL;DR =========* 1. I'm a noob to Arch 2. ... But I have substantial experience packaging perl modules for gentoo ( https://github.com/gentoo-perl/perl-experimental/graphs/contributors , https://www.ohloh.net/p/gentoo-perl- overlay/contributors/summary ) 3. And I strongly feel that in regard to packaging things, the PKGBUILD spec at present seems a little immature 4. And I strongly feel, that PKGBUILD could gain a great deal of improvement utility by having /some/ shared code mechanism 5. ... But I'm not quite sure what the best way to do this is 6. And I'm strongly against any ideas that involve developing a *Second* platform that simply emits files in the PKGBUILD format, or implementing competing products to `makepkg` 7. And I'd rather *not* have hard-inlining behaviour , I'm fine with the external behaviour being shipped independently, but if thats not universal, then I'd like to find a middleground In advance, thanks for your consideration =). -- Kent perl -e "print substr( \"edrgmaM SPA NOcomil.ic\\@tfrken\", \$_ * 3, 3 ) for ( 9,8,0,7,1,6,5,4,3,2 );" http://kent-fredric.fox.geek.nz

6 18

[pacman-dev] [PATCH] Do not use checkout directory for SVN config
by Allan McRae 11 Apr '13

11 Apr '13

Using the checkout directory for the SVN config can result in clashes between config files and files from the SVN checkout. Instead, use a ".makepkg" directory within the checkout. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- Is the ".makepkg" name unique enough? We need to use the --config-dir flag to avoid affects of the users SVN config. Also, the config directory needs to be inside the source directory to avoid issues with multiple SVN checkouts in the same directory. scripts/makepkg.sh.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 67ee22a..2bafb24 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -705,7 +705,7 @@ download_svn() { if [[ ! -d "$dir" ]] || dir_is_empty "$dir" ; then msg2 "$(gettext "Cloning %s %s repo...")" "${repo}" "svn" - if ! svn checkout --config-dir "$dir" "$url" "$dir"; then + if ! svn checkout --config-dir "$dir/.makepkg" "$url" "$dir"; then error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "svn" plain "$(gettext "Aborting...")" exit 1 -- 1.8.2.1

1 1

[pacman-dev] [PATCH] paccache: avoid subshell in calling runcmd
by Dave Reisner 11 Apr '13

11 Apr '13

Avoids problems with one of the worst CLI tools ever created, su. Fixes FS#34656. Signed-off-by: Dave Reisner <dreisner(a)archlinux.org> --- contrib/paccache.sh.in | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contrib/paccache.sh.in b/contrib/paccache.sh.in index c331273..64c3c53 100644 --- a/contrib/paccache.sh.in +++ b/contrib/paccache.sh.in @@ -308,9 +308,9 @@ totalsaved=$(@SIZECMD@ "${candidates[@]}" | awk '{ sum += $1 } END { print sum } # crush. kill. destroy. (( verbose )) && cmdopts+=(-v) if (( delete )); then - printf '%s\0' "${candidates[@]}" | runcmd xargs -0 rm "${cmdopts[@]}" + runcmd xargs -0a <(printf '%s\0' "${candidates[@]}") rm "${cmdopts[@]}" elif (( move )); then - printf '%s\0' "${candidates[@]}" | runcmd xargs -0 mv "${cmdopts[@]}" -t "$movedir" + runcmd xargs -0a <(printf '%s\0' "${candidates[@]}") mv "${cmdopts[@]}" -t "$movedir" fi summarize "$pkgcount" "${candidates[@]}" -- 1.8.2.1

2 1

[pacman-dev] [PATCH 2/2] Accept all bzr URLs in the source array
by Maxime Gauduin 11 Apr '13

11 Apr '13

From: Alucryd <alucryd(a)gmail.com> This check will run 'bzr info' on the distant URL provided in the source array and compare the branch root URL with the output of 'bzr config parent_location' run inside the local repo to make sure we are building from the right sources. Previously, the check was only run locally, as a result the local parent_location had to be used in the source array. makepkg will fallback to this behavior for offline builds. Signed-off-by: Maxime Gauduin <alucryd(a)gmail.com> --- scripts/makepkg.sh.in | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index d5b9077..0ac4975 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -474,10 +474,21 @@ download_bzr() { fi elif (( ! HOLDVER )); then # Make sure we are fetching the right repo - if [[ "$url" != "$(bzr config parent_location -d $dir)" ]] ; then - error "$(gettext "%s is not a branch of %s")" "$dir" "$url" - plain "$(gettext "Aborting...")" - exit 1 + local distant_url="$(bzr info "$url" 2> /dev/null | grep 'branch root' | sed 's| branch root: ||')" + local local_url="$(bzr config parent_location -d "$dir")" + if [[ -n "$distant_url" ]]; then + if [[ "$distant_url" != "$local_url" ]]; then + error "$(gettext "%s is not a branch of %s")" "$dir" "$url" + plain "$(gettext "Aborting...")" + exit 1 + fi + else + if [[ "$url" != "$local_url" ]] ; then + error "$(gettext "%s is not a branch of %s")" "$dir" "$url" + error "$(gettext "The local URL is %s")" "$local_url" + plain "$(gettext "Aborting...")" + exit 1 + fi fi msg2 "$(gettext "Pulling %s ...")" "${displaylocation}" cd_safe "$dir" -- 1.8.2

5 9

[pacman-dev] [PATCH] makepkg: fix svn repo extraction
by Allan McRae 11 Apr '13

11 Apr '13

Copy SVN repos rather than using "svn export" to keep all anotation files in the repo for build scripts that use (e.g.) "svin info". Signed-off-by: Allan McRae <allan(a)archlinux.org> --- scripts/makepkg.sh.in | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 2bafb24..143ce5f 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -742,7 +742,7 @@ extract_svn() { if [[ -n $fragment ]]; then case ${fragment%%=*} in revision) - ref=('-r' "${fragment##*=}") + ref="${fragment##*=}" ;; *) error "$(gettext "Unrecognized reference: %s")" "${fragment}" @@ -751,9 +751,14 @@ extract_svn() { esac fi - if ! svn export ${ref[@]} "$dir"; then - error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "svn" - plain "$(gettext "Aborting...")" + cp -a "$dir" . + + if [[ -n ${ref} ]]; then + cd_safe "$(get_filename "$netfile")" + if ! svn update -r ${ref}; then + error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "svn" + plain "$(gettext "Aborting...")" + fi fi popd &>/dev/null -- 1.8.2.1

1 0

[pacman-dev] [PATCH] makepkg: don't run remove_deps twice when unneeded
by William Giokas 11 Apr '13

11 Apr '13

remove_deps already has a check and won't run unless -r is specified, so if this was meant to remove dependencies of a failure no matter what, then it's not doing it, and with -r it is run twice on a failure for no real reason. Signed-off-by: William Giokas <1007380(a)gmail.com> --- scripts/makepkg.sh.in | 1 - 1 file changed, 1 deletion(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 73f45d4..3bcc527 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -1356,7 +1356,6 @@ error_function() { if (( ! BASH_SUBSHELL )); then error "$(gettext "A failure occurred in %s().")" "$1" plain "$(gettext "Aborting...")" - remove_deps fi exit 2 # $E_BUILD_FAILED } -- 1.8.2

2 1

[pacman-dev] [PATCH 2/2] Accept all bzr URLs in the source array
by Maxime Gauduin 09 Apr '13

09 Apr '13

From: Alucryd <alucryd(a)gmail.com> This check will run 'bzr info' on the distant URL provided in the source array and compare the branch root URL with the output of 'bzr config parent_location' run inside the local repo to make sure we are building from the right sources. Previously, the check was only run locally, as a result the local parent_location had to be used in the source array. makepkg will fallback to this behavior for offline builds. Signed-off-by: Maxime Gauduin <alucryd(a)gmail.com> --- scripts/makepkg.sh.in | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index d5b9077..0ac4975 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -474,10 +474,21 @@ download_bzr() { fi elif (( ! HOLDVER )); then # Make sure we are fetching the right repo - if [[ "$url" != "$(bzr config parent_location -d $dir)" ]] ; then - error "$(gettext "%s is not a branch of %s")" "$dir" "$url" - plain "$(gettext "Aborting...")" - exit 1 + local distant_url="$(bzr info $url 2> /dev/null | grep root | sed 's| branch root: ||')" + local local_url="$(bzr config parent_location -d $dir)" + if [[ ! -z "$distant_url" ]]; then + if [[ "$distant_url" != "$local_url" ]]; then + error "$(gettext "%s is not a branch of %s")" "$dir" "$url" + plain "$(gettext "Aborting...")" + exit 1 + fi + else + if [[ "$url" != "$local_url" ]] ; then + error "$(gettext "%s is not a branch of %s")" "$dir" "$url" + error "$(gettext "The local URL is %s")" "$local_url" + plain "$(gettext "Aborting...")" + exit 1 + fi fi msg2 "$(gettext "Pulling %s ...")" "${displaylocation}" cd_safe "$dir" -- 1.8.2

4 13

[pacman-dev] [PATCH] Fix dest dir when using lp: bzr URLs in the source array
by Maxime Gauduin 09 Apr '13

09 Apr '13

From: Alucryd <alucryd(a)gmail.com> Signed-off-by: Maxime Gauduin <alucryd(a)gmail.com> --- scripts/makepkg.sh.in | 3 +++ 1 file changed, 3 insertions(+) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 0ac4975..69001d3 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -245,6 +245,9 @@ get_filename() { filename=${netfile%%#*} filename=${filename%/} filename=${filename##*/} + if [[ $proto = bzr* ]]; then + filename=${filename#*lp:} + fi if [[ $proto = git* ]]; then filename=${filename%%.git*} fi -- 1.8.2

1 0