December 2010 - Pacman-dev - lists.archlinux.org

[pacman-dev] [PATCH] Add script to update pacman database format
by Allan McRae 08 Dec '10

08 Dec '10

The pacman-db-upgrade script was added to detect old pacman database formats and upgrade them. Currently performs the merging of depends files into desc files in the local database. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- scripts/Makefile.am | 7 +++++-- scripts/pacman-db-upgrade.sh.in | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 2 deletions(-) create mode 100644 scripts/pacman-db-upgrade.sh.in diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 78deb0b..7448691 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -10,14 +10,16 @@ OURSCRIPTS = \ pacman-optimize \ pkgdelta \ rankmirrors \ - repo-add + repo-add \ + pacman-db-upgrade EXTRA_DIST = \ makepkg.sh.in \ pacman-optimize.sh.in \ pkgdelta.sh.in \ rankmirrors.sh.in \ - repo-add.sh.in + repo-add.sh.in \ + pacman-db-upgrade.sh.in # Files that should be removed, but which Automake does not know. MOSTLYCLEANFILES = $(bin_SCRIPTS) *.tmp @@ -68,5 +70,6 @@ repo-add: $(srcdir)/repo-add.sh.in repo-remove: $(srcdir)/repo-add.sh.in rm -f repo-remove $(LN_S) repo-add repo-remove +pacman-db-update: $(srcdir)/pacman-db-update.sh.in # vim:set ts=2 sw=2 noet: diff --git a/scripts/pacman-db-upgrade.sh.in b/scripts/pacman-db-upgrade.sh.in new file mode 100644 index 0000000..e28b1d2 --- /dev/null +++ b/scripts/pacman-db-upgrade.sh.in @@ -0,0 +1,34 @@ +#!@BASH_SHELL@ -e +# +# pacman-db-upgrade - upgrade the local pacman db to a newer format +# @configure_input@ +# +# Copyright (c) 2010 Pacman Development Team <pacman-dev(a)archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +eval $(awk '/DBPath/ {print $1$2$3}' @sysconfdir@/pacman.conf) +DBPath="${DBPath:-@localstatedir@/lib/pacman/}/local" + +# pacman-3.4 to 3.5 upgrade - merge depends into desc +if [[ $(find $DBPath -name depends) ]]; then + echo "==> pacman-3.4 database format dectected - upgrading" + for i in $DBPath/*; do + if [[ -f $i/depends ]]; then + cat $i/depends >> $i/desc + rm $i/depends + fi + done +fi -- 1.7.3.3

2 3

[pacman-dev] [PATCH] Use size_t for alpm_list sizes
by Allan McRae 08 Dec '10

08 Dec '10

There is a lot of swtiching between size_t and int for alpm_list sizes in the codebase. Start converting these to all be size_t by adjusting the return type of alpm_list_count and fixing all additional warnings given by -Wconversion that are generated by this change. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- lib/libalpm/add.c | 4 ++-- lib/libalpm/alpm.h | 2 +- lib/libalpm/alpm_list.c | 8 ++++---- lib/libalpm/alpm_list.h | 6 +++--- lib/libalpm/be_local.c | 2 +- lib/libalpm/be_sync.c | 2 +- lib/libalpm/conflict.c | 4 ++-- lib/libalpm/diskspace.c | 6 +++--- lib/libalpm/remove.c | 6 +++--- lib/libalpm/sync.c | 2 +- src/pacman/callback.c | 5 +++-- src/pacman/callback.h | 2 +- 12 files changed, 25 insertions(+), 24 deletions(-) diff --git a/lib/libalpm/add.c b/lib/libalpm/add.c index b0b5a53..31a7928 100644 --- a/lib/libalpm/add.c +++ b/lib/libalpm/add.c @@ -477,7 +477,7 @@ static int extract_single_file(struct archive *archive, return(errors); } -static int commit_single_pkg(pmpkg_t *newpkg, int pkg_current, int pkg_count, +static int commit_single_pkg(pmpkg_t *newpkg, size_t pkg_current, size_t pkg_count, pmtrans_t *trans, pmdb_t *db) { int i, ret = 0, errors = 0; @@ -715,7 +715,7 @@ cleanup: int _alpm_upgrade_packages(pmtrans_t *trans, pmdb_t *db) { - int pkg_count, pkg_current; + size_t pkg_count, pkg_current; int skip_ldconfig = 0, ret = 0; alpm_list_t *targ; diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 08d0269..86212a4 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -405,7 +405,7 @@ typedef void (*alpm_trans_cb_conv)(pmtransconv_t, void *, void *, void *, int *); /* Transaction Progress callback */ -typedef void (*alpm_trans_cb_progress)(pmtransprog_t, const char *, int, int, int); +typedef void (*alpm_trans_cb_progress)(pmtransprog_t, const char *, int, size_t, size_t); int alpm_trans_get_flags(); alpm_list_t * alpm_trans_get_add(); diff --git a/lib/libalpm/alpm_list.c b/lib/libalpm/alpm_list.c index 80ba1ee..185295c 100644 --- a/lib/libalpm/alpm_list.c +++ b/lib/libalpm/alpm_list.c @@ -269,7 +269,7 @@ alpm_list_t SYMEXPORT *alpm_list_mmerge(alpm_list_t *left, alpm_list_t *right, a * * @return the resultant list */ -alpm_list_t SYMEXPORT *alpm_list_msort(alpm_list_t *list, int n, alpm_list_fn_cmp fn) +alpm_list_t SYMEXPORT *alpm_list_msort(alpm_list_t *list, size_t n, alpm_list_fn_cmp fn) { if (n > 1) { alpm_list_t *left = list; @@ -511,7 +511,7 @@ inline alpm_list_t SYMEXPORT *alpm_list_first(const alpm_list_t *list) * * @return an alpm_list_t node for index `n` */ -alpm_list_t SYMEXPORT *alpm_list_nth(const alpm_list_t *list, int n) +alpm_list_t SYMEXPORT *alpm_list_nth(const alpm_list_t *list, size_t n) { const alpm_list_t *i = list; while(n--) { @@ -574,9 +574,9 @@ void SYMEXPORT *alpm_list_getdata(const alpm_list_t *node) * * @return the number of list items */ -int SYMEXPORT alpm_list_count(const alpm_list_t *list) +size_t SYMEXPORT alpm_list_count(const alpm_list_t *list) { - unsigned int i = 0; + size_t i = 0; const alpm_list_t *lp = list; while(lp) { ++i; diff --git a/lib/libalpm/alpm_list.h b/lib/libalpm/alpm_list.h index bd639f7..19fe548 100644 --- a/lib/libalpm/alpm_list.h +++ b/lib/libalpm/alpm_list.h @@ -56,7 +56,7 @@ alpm_list_t *alpm_list_add(alpm_list_t *list, void *data); alpm_list_t *alpm_list_add_sorted(alpm_list_t *list, void *data, alpm_list_fn_cmp fn); alpm_list_t *alpm_list_join(alpm_list_t *first, alpm_list_t *second); alpm_list_t *alpm_list_mmerge(alpm_list_t *left, alpm_list_t *right, alpm_list_fn_cmp fn); -alpm_list_t *alpm_list_msort(alpm_list_t *list, int n, alpm_list_fn_cmp fn); +alpm_list_t *alpm_list_msort(alpm_list_t *list, size_t n, alpm_list_fn_cmp fn); alpm_list_t *alpm_list_remove(alpm_list_t *haystack, const void *needle, alpm_list_fn_cmp fn, void **data); alpm_list_t *alpm_list_remove_str(alpm_list_t *haystack, const char *needle, char **data); alpm_list_t *alpm_list_remove_dupes(const alpm_list_t *list); @@ -67,13 +67,13 @@ alpm_list_t *alpm_list_reverse(alpm_list_t *list); /* item accessors */ alpm_list_t *alpm_list_first(const alpm_list_t *list); -alpm_list_t *alpm_list_nth(const alpm_list_t *list, int n); +alpm_list_t *alpm_list_nth(const alpm_list_t *list, size_t n); alpm_list_t *alpm_list_next(const alpm_list_t *list); alpm_list_t *alpm_list_last(const alpm_list_t *list); void *alpm_list_getdata(const alpm_list_t *entry); /* misc */ -int alpm_list_count(const alpm_list_t *list); +size_t alpm_list_count(const alpm_list_t *list); void *alpm_list_find(const alpm_list_t *haystack, const void *needle, alpm_list_fn_cmp fn); void *alpm_list_find_ptr(const alpm_list_t *haystack, const void *needle); char *alpm_list_find_str(const alpm_list_t *haystack, const char *needle); diff --git a/lib/libalpm/be_local.c b/lib/libalpm/be_local.c index 9dce66d..fec8979 100644 --- a/lib/libalpm/be_local.c +++ b/lib/libalpm/be_local.c @@ -422,7 +422,7 @@ int _alpm_local_db_populate(pmdb_t *db) } closedir(dbdir); - db->pkgcache = alpm_list_msort(db->pkgcache, count, _alpm_pkg_cmp); + db->pkgcache = alpm_list_msort(db->pkgcache, (size_t)count, _alpm_pkg_cmp); return(count); } diff --git a/lib/libalpm/be_sync.c b/lib/libalpm/be_sync.c index b6c9937..5951f2f 100644 --- a/lib/libalpm/be_sync.c +++ b/lib/libalpm/be_sync.c @@ -191,7 +191,7 @@ int _alpm_sync_db_populate(pmdb_t *db) } } - db->pkgcache = alpm_list_msort(db->pkgcache, count, _alpm_pkg_cmp); + db->pkgcache = alpm_list_msort(db->pkgcache, (size_t)count, _alpm_pkg_cmp); archive_read_finish(archive); return(count); diff --git a/lib/libalpm/conflict.c b/lib/libalpm/conflict.c index 2b0efa5..e64a00b 100644 --- a/lib/libalpm/conflict.c +++ b/lib/libalpm/conflict.c @@ -402,8 +402,8 @@ alpm_list_t *_alpm_db_find_fileconflicts(pmdb_t *db, pmtrans_t *trans, alpm_list_t *upgrade, alpm_list_t *remove) { alpm_list_t *i, *j, *conflicts = NULL; - int numtargs = alpm_list_count(upgrade); - int current; + size_t numtargs = alpm_list_count(upgrade); + size_t current; ALPM_LOG_FUNC; diff --git a/lib/libalpm/diskspace.c b/lib/libalpm/diskspace.c index 5ed8e0b..728fd2c 100644 --- a/lib/libalpm/diskspace.c +++ b/lib/libalpm/diskspace.c @@ -243,11 +243,11 @@ cleanup: int _alpm_check_diskspace(pmtrans_t *trans, pmdb_t *db_local) { alpm_list_t *mount_points, *i; - int replaces = 0, abort = 0; + size_t replaces = 0, current = 0; + int abort = 0; alpm_list_t *targ; pmpkg_t *pkg; - int numtargs = alpm_list_count(trans->add); - int current = 0; + size_t numtargs = alpm_list_count(trans->add); mount_points = mount_point_list(); if(mount_points == NULL) { diff --git a/lib/libalpm/remove.c b/lib/libalpm/remove.c index 90848dc..fa870c1 100644 --- a/lib/libalpm/remove.c +++ b/lib/libalpm/remove.c @@ -368,7 +368,7 @@ int _alpm_remove_packages(pmtrans_t *trans, pmdb_t *db) { pmpkg_t *info; alpm_list_t *targ, *lp; - int pkg_count; + size_t pkg_count; ALPM_LOG_FUNC; @@ -382,7 +382,7 @@ int _alpm_remove_packages(pmtrans_t *trans, pmdb_t *db) char scriptlet[PATH_MAX]; info = (pmpkg_t*)targ->data; const char *pkgname = NULL; - int targcount = alpm_list_count(targ); + size_t targcount = alpm_list_count(targ); if(handle->trans->state == STATE_INTERRUPTED) { return(0); @@ -413,7 +413,7 @@ int _alpm_remove_packages(pmtrans_t *trans, pmdb_t *db) } } - int filenum = alpm_list_count(files); + size_t filenum = alpm_list_count(files); alpm_list_t *newfiles; _alpm_log(PM_LOG_DEBUG, "removing %d files\n", filenum); diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index 8af32e4..2d0a899 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -801,7 +801,7 @@ int _alpm_sync_commit(pmtrans_t *trans, pmdb_t *db_local, alpm_list_t **data) { alpm_list_t *i, *j, *files = NULL; alpm_list_t *deltas = NULL; - int replaces = 0; + size_t replaces = 0; int errors = 0; const char *cachedir = NULL; int ret = -1; diff --git a/src/pacman/callback.c b/src/pacman/callback.c index c8f604f..ec4a683 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -325,13 +325,14 @@ void cb_trans_conv(pmtransconv_t event, void *data1, void *data2, /* callback to handle display of transaction progress */ void cb_trans_progress(pmtransprog_t event, const char *pkgname, int percent, - int howmany, int remain) + size_t howmany, size_t remain) { float timediff; /* size of line to allocate for text printing (e.g. not progressbar) */ int infolen; - int tmp, digits, textlen; + int digits, textlen; + size_t tmp; char *opr = NULL; /* used for wide character width determination and printing */ int len, wclen, wcwid, padwid; diff --git a/src/pacman/callback.h b/src/pacman/callback.h index c56eaa7..09fe8bd 100644 --- a/src/pacman/callback.h +++ b/src/pacman/callback.h @@ -33,7 +33,7 @@ void cb_trans_conv(pmtransconv_t event, void *data1, void *data2, /* callback to handle display of transaction progress */ void cb_trans_progress(pmtransprog_t event, const char *pkgname, int percent, - int howmany, int remain); + size_t howmany, size_t remain); /* callback to handle receipt of total download value */ void cb_dl_total(off_t total); -- 1.7.3.3

1 1

[pacman-dev] [PATCH] Explicitly test time difference is greater than zero
by Allan McRae 08 Dec '10

08 Dec '10

We are comparing a floating point number so should use an inequality rather than implicitly testing != 0. Prevents warning given by -Wfloat-equal. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- src/pacman/callback.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/src/pacman/callback.c b/src/pacman/callback.c index f585b28..c8f604f 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -352,7 +352,7 @@ void cb_trans_progress(pmtransprog_t event, const char *pkgname, int percent, timediff = get_update_timediff(0); } - if(percent > 0 && percent < 100 && !timediff) { + if(percent > 0 && percent < 100 && timediff > 0) { /* only update the progress bar when * a) we first start * b) we end the progress -- 1.7.3.3

1 0

[pacman-dev] [PATCH] makepkg: perform sanity checks on variables in package functions
by Allan McRae 08 Dec '10

08 Dec '10

Check the over-ridden entries for provides, backup, optdepends and options for illegal entries. Partially fixes FS#16004. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- Resubmit of patch: - Added check for valid overridden options as it uses the same technique as the other checks that were added. - Fixed PKGBUILD vs $BUILDFILE usage scripts/makepkg.sh.in | 19 +++++++++++++------ 1 files changed, 13 insertions(+), 6 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 4e321d1..ac10ad3 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -27,7 +27,7 @@ # makepkg uses quite a few external programs during its execution. You # need to have at least the following installed for makepkg to function: -# bsdtar (libarchive), bzip2, coreutils, fakeroot, file, find (findutils), +# awk, bsdtar (libarchive), bzip2, coreutils, fakeroot, file, find (findutils), # gettext, grep, gzip, openssl, sed, tput (ncurses), xz # gettext initialization @@ -1222,21 +1222,27 @@ check_sanity() { fi fi - for i in ${provides[@]}; do + local provides_list + eval $(awk '/^[[:space:]]*provides=/,/)/' "$BUILDFILE" | sed "s/provides=/provides_list+=/") + for i in ${provides_list[@]}; do if [[ $i != ${i//</} || $i != ${i//>/} ]]; then error "$(gettext "Provides array cannot contain comparison (< or >) operators.")" return 1 fi done - for i in "${backup[@]}"; do + local backup_list + eval $(awk '/^[[:space:]]*backup=/,/)/' "$BUILDFILE" | sed "s/backup=/backup_list+=/") + for i in "${backup_list[@]}"; do if [[ ${i:0:1} = "/" ]]; then error "$(gettext "Backup entry should not contain leading slash : %s")" "$i" return 1 fi done - for i in "${optdepends[@]}"; do + local optdepends_list + eval $(awk '/^[[:space:]]*optdepends=/,/)/' "$BUILDFILE" | sed "s/optdepends=/optdepends_list+=/") + for i in "${optdepends_list[@]}"; do local pkg=${i%%:*} if [[ ! $pkg =~ ^[[:alnum:]\>\<\=\.\+\_\-]+$ ]]; then error "$(gettext "Invalid syntax for optdepend : '%s'")" "$i" @@ -1257,8 +1263,9 @@ check_sanity() { done local valid_options=1 - local known kopt - for i in ${options[@]}; do + local known kopt options_list + eval $(awk '/^[[:space:]]*options=/,/)/' "$BUILDFILE" | sed "s/options=/options_list+=/") + for i in ${options_list[@]}; do known=0 # check if option matches a known option or its inverse for kopt in ${packaging_options[@]} ${other_options[@]}; do -- 1.7.3.3

1 0

[pacman-dev] [PATCH] makepkg: perform sanity checks on variables in package functions
by Allan McRae 07 Dec '10

07 Dec '10

Check the over-ridden entries for provides, backup and optdepends for illegal entries. Fixes FS#16004. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- scripts/makepkg.sh.in | 14 ++++++++++---- 1 files changed, 10 insertions(+), 4 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 4e321d1..d5ccdbf 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -27,7 +27,7 @@ # makepkg uses quite a few external programs during its execution. You # need to have at least the following installed for makepkg to function: -# bsdtar (libarchive), bzip2, coreutils, fakeroot, file, find (findutils), +# awk, bsdtar (libarchive), bzip2, coreutils, fakeroot, file, find (findutils), # gettext, grep, gzip, openssl, sed, tput (ncurses), xz # gettext initialization @@ -1222,21 +1222,27 @@ check_sanity() { fi fi - for i in ${provides[@]}; do + local provides_list + eval $(awk '/^[[:space:]]*provides=/,/)/' PKGBUILD | sed "s/provides=/provides_list+=/") + for i in ${provides_list[@]}; do if [[ $i != ${i//</} || $i != ${i//>/} ]]; then error "$(gettext "Provides array cannot contain comparison (< or >) operators.")" return 1 fi done - for i in "${backup[@]}"; do + local backup_list + eval $(awk '/^[[:space:]]*backup=/,/)/' PKGBUILD | sed "s/backup=/backup_list+=/") + for i in "${backup_list[@]}"; do if [[ ${i:0:1} = "/" ]]; then error "$(gettext "Backup entry should not contain leading slash : %s")" "$i" return 1 fi done - for i in "${optdepends[@]}"; do + local optdepends_list + eval $(awk '/^[[:space:]]*optdepends=/,/)/' PKGBUILD | sed "s/optdepends=/optdepends_list+=/") + for i in "${optdepends_list[@]}"; do local pkg=${i%%:*} if [[ ! $pkg =~ ^[[:alnum:]\>\<\=\.\+\_\-]+$ ]]; then error "$(gettext "Invalid syntax for optdepend : '%s'")" "$i" -- 1.7.3.3

2 2

[pacman-dev] Changing how repo dbs are updated
by Allan McRae 04 Dec '10

04 Dec '10

Hi, This message is more of a sounding board for me to get the issues surrounding this sorted and point out what I am planning to do. But any comments on this would also be appreciated. Especially #3 below. Issue I see currently and in the future with signed databases: 1) Currently the repo dbs are updated just like downloading a package file. If the update is started and canceled part way though, you get a repo.db.part file which pacman attempts to continue downloading. However, unlike package files, this file is not static content and so we should never continue the download. See https://bugs.archlinux.org/task/15657 . This can be handled by just deleting the repo.db.part file if present, but it might be better just never create .part files in the first place for repo dbs by downloading to a temporary location and moving/deleting based on successful completion. That would mean having a different download function for repo dbs and packages. See #2 for additional reasons to split this... 2) Database signing. Currently the code downloads the database, deletes the old now invalid signature, then downloads the new signature. If the signature is valid, then all is fine. However, if it fails to download or is invalid, pacman issues an error about failing to update the database. The database on your system is now not correctly signed (which is bad given its signature is only checked on update...). I think that the old database and signature should only be overwritten if the new database download is successful _and_ its signature is valid. This requires downloading the database and its signature to a temporary location and then moving the files only once they are confirmed valid. That would require a different download interface for package and database downloads, but that is a good thing as we can get rid of the force crap from the one used for packages. 3) pacman -Syy behavior. Instead of adding a "force" flag to overwrite the old database, would it be better to just delete the old database first? Currently, if you use pacman -Syy and a database download fails, you are left with the old sync database you told pacman to get rid of. Is leaving pacman with no database for that repo a better solution? I'm not sure about #3... But to fix #1 and #2, I think we need to split the download handling for dbs and packages slightly unless someone has a better idea of how to deal with those? Allan

4 7