- Pacman-dev - lists.archlinux.org

[pacman-dev] [PATCH 1/5] Fix several issues with xdelta
by Xavier Chantry 01 Mar '09

01 Mar '09

1) The changes to sync.c look big but there are mostly caused by the indentation. Fix a bug where download_size == 0 because the packages and deltas are already in the cache, but we still need to build the deltas list and apply the deltas to create the final package. 2) Fix the gzip / md5sum issue by switching to xdelta3, disabling external recompression and using gzip -n in pacman, and disable bsdtar compression and using gzip -n in makepkg. Signed-off-by: Xavier Chantry <shiningxc(a)gmail.com> --- doc/pacman.conf.5.txt | 2 +- lib/libalpm/delta.c | 4 +- lib/libalpm/sync.c | 98 +++++++++++++++++++++++-------------------------- scripts/makepkg.sh.in | 28 +++----------- 4 files changed, 55 insertions(+), 77 deletions(-) diff --git a/doc/pacman.conf.5.txt b/doc/pacman.conf.5.txt index fa21294..2f1fe3d 100644 --- a/doc/pacman.conf.5.txt +++ b/doc/pacman.conf.5.txt @@ -144,7 +144,7 @@ Options *UseDelta*:: Download delta files instead of complete packages if possible. Requires - the xdelta program to be installed. + the xdelta3 program to be installed. *TotalDownload*:: When downloading, display the amount downloaded, download rate, ETA, diff --git a/lib/libalpm/delta.c b/lib/libalpm/delta.c index ff77501..337eb66 100644 --- a/lib/libalpm/delta.c +++ b/lib/libalpm/delta.c @@ -232,13 +232,13 @@ off_t _alpm_shortest_delta_path(alpm_list_t *deltas, return(bestsize); } - _alpm_log(PM_LOG_DEBUG, "started delta shortest-path search\n"); + _alpm_log(PM_LOG_DEBUG, "started delta shortest-path search for '%s'\n", to); vertices = delta_graph_init(deltas); bestsize = delta_vert(vertices, to, to_md5, &bestpath); - _alpm_log(PM_LOG_DEBUG, "delta shortest-path search complete\n"); + _alpm_log(PM_LOG_DEBUG, "delta shortest-path search complete : '%lld'\n", (long long)bestsize); alpm_list_free_inner(vertices, _alpm_graph_free); alpm_list_free(vertices); diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index 709a36d..71d8771 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -725,14 +725,9 @@ static int apply_deltas(pmtrans_t *trans) CALLOC(to, len, sizeof(char), RET_ERR(PM_ERR_MEMORY, 1)); snprintf(to, len, "%s/%s", cachedir, d->to); - /* an example of the patch command: (using /cache for cachedir) - * xdelta patch /path/to/pacman_3.0.0-1_to_3.0.1-1-i686.delta \ - * /path/to/pacman-3.0.0-1-i686.pkg.tar.gz \ - * /cache/pacman-3.0.1-1-i686.pkg.tar.gz - */ - /* build the patch command */ - snprintf(command, PATH_MAX, "xdelta patch %s %s %s", delta, from, to); + /* -R for disabling external recompression with gzip, -c for sending to stdout */ + snprintf(command, PATH_MAX, "xdelta3 -d -q -R -c -s %s %s | gzip -n > %s", from, delta, to); _alpm_log(PM_LOG_DEBUG, _("command: %s\n"), command); @@ -848,29 +843,31 @@ int _alpm_sync_commit(pmtrans_t *trans, pmdb_t *db_local, alpm_list_t **data) fname = alpm_pkg_get_filename(spkg); ASSERT(fname != NULL, RET_ERR(PM_ERR_PKG_INVALID_NAME, -1)); - if(spkg->download_size != 0) { - alpm_list_t *delta_path = spkg->delta_path; - if(delta_path) { - alpm_list_t *dlts = NULL; - - for(dlts = delta_path; dlts; dlts = dlts->next) { - pmdelta_t *d = dlts->data; - - if(d->download_size != 0) { - /* add the delta filename to the download list if - * it's not in the cache */ - files = alpm_list_add(files, strdup(d->delta)); - } - - /* keep a list of the delta files for md5sums */ - deltas = alpm_list_add(deltas, d); + alpm_list_t *delta_path = spkg->delta_path; + if(delta_path) { + /* using deltas */ + alpm_list_t *dlts = NULL; + + for(dlts = delta_path; dlts; dlts = dlts->next) { + pmdelta_t *d = dlts->data; + + if(d->download_size != 0) { + /* add the delta filename to the download list if needed */ + files = alpm_list_add(files, strdup(d->delta)); } - } else { - /* not using deltas, so add the file to the download list */ + /* keep a list of all the delta files for md5sums */ + deltas = alpm_list_add(deltas, d); + } + + } else { + /* not using deltas */ + if(spkg->download_size != 0) { + /* add the filename to the download list if needed */ files = alpm_list_add(files, strdup(fname)); } } + } } @@ -891,37 +888,34 @@ int _alpm_sync_commit(pmtrans_t *trans, pmdb_t *db_local, alpm_list_t **data) handle->totaldlcb(0); } - if(handle->usedelta) { + /* if we have deltas to work with */ + if(handle->usedelta && deltas) { int ret = 0; - - /* only output if there are deltas to work with */ - if(deltas) { - errors = 0; - /* Check integrity of deltas */ - EVENT(trans, PM_TRANS_EVT_DELTA_INTEGRITY_START, NULL, NULL); - - for(i = deltas; i; i = i->next) { - pmdelta_t *d = alpm_list_getdata(i); - const char *filename = alpm_delta_get_filename(d); - const char *md5sum = alpm_delta_get_md5sum(d); - - if(test_md5sum(trans, filename, md5sum) != 0) { - errors++; - *data = alpm_list_add(*data, strdup(filename)); - } - } - if(errors) { - pm_errno = PM_ERR_DLT_INVALID; - goto error; + errors = 0; + /* Check integrity of deltas */ + EVENT(trans, PM_TRANS_EVT_DELTA_INTEGRITY_START, NULL, NULL); + + for(i = deltas; i; i = i->next) { + pmdelta_t *d = alpm_list_getdata(i); + const char *filename = alpm_delta_get_filename(d); + const char *md5sum = alpm_delta_get_md5sum(d); + + if(test_md5sum(trans, filename, md5sum) != 0) { + errors++; + *data = alpm_list_add(*data, strdup(filename)); } - EVENT(trans, PM_TRANS_EVT_DELTA_INTEGRITY_DONE, NULL, NULL); + } + if(errors) { + pm_errno = PM_ERR_DLT_INVALID; + goto error; + } + EVENT(trans, PM_TRANS_EVT_DELTA_INTEGRITY_DONE, NULL, NULL); - /* Use the deltas to generate the packages */ - EVENT(trans, PM_TRANS_EVT_DELTA_PATCHES_START, NULL, NULL); - ret = apply_deltas(trans); - EVENT(trans, PM_TRANS_EVT_DELTA_PATCHES_DONE, NULL, NULL); + /* Use the deltas to generate the packages */ + EVENT(trans, PM_TRANS_EVT_DELTA_PATCHES_START, NULL, NULL); + ret = apply_deltas(trans); + EVENT(trans, PM_TRANS_EVT_DELTA_PATCHES_DONE, NULL, NULL); - } if(ret) { pm_errno = PM_ERR_DLT_PATCHFAILED; goto error; diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 52e80d1..9e0f249 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -975,7 +975,7 @@ create_package() { # the null string rather than itself shopt -s nullglob - if ! bsdtar -c${TAR_OPT}f "$pkg_file" $comp_files *; then + if ! bsdtar -c${TAR_OPT}f - $comp_files * | gzip -n > "$pkg_file"; then error "$(gettext "Failed to create package file.")" exit 1 # TODO: error code fi @@ -985,8 +985,8 @@ create_package() { create_xdelta() { if [ "$(check_buildenv xdelta)" != "y" ]; then return - elif [ ! "$(type -p xdelta)" ]; then - error "$(gettext "Cannot find the xdelta binary! Is xdelta installed?")" + elif [ ! "$(type -p xdelta3)" ]; then + error "$(gettext "Cannot find the xdelta3 binary! Is xdelta3 installed?")" return fi @@ -1020,25 +1020,9 @@ create_xdelta() { local delta_file="$PKGDEST/$pkgname-${latest_version}_to_$pkgver-$pkgrel-$CARCH.delta" local ret=0 - # xdelta will decompress base_file & pkg_file into TMP_DIR (or /tmp if - # TMP_DIR is unset) then perform the delta on the resulting tars - xdelta delta "$base_file" "$pkg_file" "$delta_file" || ret=$? - - if [ $ret -eq 0 -o $ret -eq 1 ]; then - # Generate the final gz using xdelta for compression. xdelta will be our - # common denominator compression utility between the packager and the - # users. makepkg and pacman must use the same compression algorithm or - # the delta generated package may not match, producing md5 checksum - # errors. - msg2 "$(gettext "Recreating package tarball from delta to match md5 signatures")" - msg2 "$(gettext "NOTE: the delta should ONLY be distributed with this tarball")" - ret=0 - xdelta patch "$delta_file" "$base_file" "$pkg_file" || ret=$? - if [ $ret -ne 0 ]; then - error "$(gettext "Could not generate the package from the delta.")" - exit 1 - fi - else + xdelta3 -s "$base_file" "$pkg_file" "$delta_file" || ret=$? + + if [ $ret -ne 0 ]; then warning "$(gettext "Delta was not able to be created.")" fi else -- 1.6.1.3

5 20

[pacman-dev] [PATCH 1/6] repo-add: print warning if same version already exists
by Xavier Chantry 01 Mar '09

01 Mar '09

Simple fix for FS#13414 Signed-off-by: Xavier Chantry <shiningxc(a)gmail.com> --- scripts/repo-add.sh.in | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-) diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index c6d25aa..b12188c 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -172,12 +172,16 @@ db_write_entry() return 1 fi - # remove an existing entry if it exists, ignore failures - db_remove_entry "$pkgname" - startdir=$(pwd) pushd "$gstmpdir" 2>&1 >/dev/null + if [ -d "$pkgname-$pkgver" ]; then + warning "$(gettext "An entry for '%s' already existed")" "$pkgname-$pkgver" + fi + + # remove an existing entry if it exists, ignore failures + db_remove_entry "$pkgname" + # create package directory mkdir "$pkgname-$pkgver" cd "$pkgname-$pkgver" -- 1.6.1.3

4 13

[pacman-dev] [GIT] The official pacman repository branch, maint, updated. v3.2.2-10-gb3c67a1
by dan＠archlinux.org 28 Feb '09

28 Feb '09

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "The official pacman repository". The branch, maint has been updated via b3c67a195f254fe456003559ceac0ae0771d93e6 (commit) from a309a016bfed9f67452f2a6556fb4391fd55b34e (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- ----------------------------------------------------------------------- Summary of changes: lib/libalpm/po/LINGUAS | 1 + lib/libalpm/po/kk.po | 557 +++++++++++++++++ po/LINGUAS | 1 + po/kk.po | 1575 ++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 2134 insertions(+), 0 deletions(-) create mode 100644 lib/libalpm/po/kk.po create mode 100644 po/kk.po hooks/post-receive -- The official pacman repository

1 0

[pacman-dev] [GIT] The official pacman repository branch, master, updated. v3.2.2-99-gd8d9ab8
by dan＠archlinux.org 28 Feb '09

28 Feb '09

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "The official pacman repository". The branch, master has been updated via d8d9ab8c870eb9f2df5e6e607b1da34e6e40e29e (commit) via 4458f71851e6056957e6fafebec286765bbbcc39 (commit) via b4e1365657f477f4e27933ccc3d408b7e19ac6da (commit) via 8c09c19139b47dab3bf2282f87fe289e1282f361 (commit) via f09f82ee595319917c794dff69486bca851a5601 (commit) via b3c67a195f254fe456003559ceac0ae0771d93e6 (commit) from 2268981ebf562a7dda1e788dbd931e48adf9fabb (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit d8d9ab8c870eb9f2df5e6e607b1da34e6e40e29e Author: Dan McGee <dan(a)archlinux.org> Date: Sat Feb 28 15:46:51 2009 -0600 Update checks for python in configure Signed-off-by: Dan McGee <dan(a)archlinux.org> commit 4458f71851e6056957e6fafebec286765bbbcc39 Merge: b4e1365657f477f4e27933ccc3d408b7e19ac6da b3c67a195f254fe456003559ceac0ae0771d93e6 Author: Dan McGee <dan(a)archlinux.org> Date: Sat Feb 28 15:43:36 2009 -0600 Merge branch 'maint' commit b4e1365657f477f4e27933ccc3d408b7e19ac6da Author: Xavier Chantry <shiningxc(a)gmail.com> Date: Wed Feb 25 19:40:57 2009 +0100 makepkg : remove xdelta support The pkgdelta script can be used instead. Signed-off-by: Xavier Chantry <shiningxc(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> Signed-off-by: Dan McGee <dan(a)archlinux.org> commit 8c09c19139b47dab3bf2282f87fe289e1282f361 Author: Xavier Chantry <shiningxc(a)gmail.com> Date: Wed Feb 25 18:24:38 2009 +0100 libalpm: remove from_md5 and to_md5 from pmdelta_t The from_md5 and to_md5 fields were a nice extra safety, which would avoid trying to apply deltas on corrupted package files. However, they are not strictly necessary, since xdelta should be able to detect that on its own. The main problem is that it is impossible to compute these informations from the delta only. So repo-add would not be able to compute the delta entry based on just the delta file. Signed-off-by: Xavier Chantry <shiningxc(a)gmail.com> Signed-off-by: Dan McGee <dan(a)archlinux.org> commit f09f82ee595319917c794dff69486bca851a5601 Author: Xavier Chantry <shiningxc(a)gmail.com> Date: Thu Feb 26 18:00:15 2009 +0100 repo-add: print warning if same version already exists Simple fix for FS#13414. Signed-off-by: Xavier Chantry <shiningxc(a)gmail.com> Signed-off-by: Dan McGee <dan(a)archlinux.org> ----------------------------------------------------------------------- Summary of changes: configure.ac | 2 +- doc/makepkg.conf.5.txt | 7 +- etc/makepkg.conf.in | 5 +- lib/libalpm/alpm.h | 2 - lib/libalpm/delta.c | 57 +-- lib/libalpm/delta.h | 14 +- lib/libalpm/po/LINGUAS | 1 + lib/libalpm/po/kk.po | 557 +++++++++++++++++ lib/libalpm/sync.c | 1 - po/LINGUAS | 1 + po/kk.po | 1575 ++++++++++++++++++++++++++++++++++++++++++++++++ scripts/makepkg.sh.in | 66 -- scripts/repo-add.sh.in | 10 +- 13 files changed, 2165 insertions(+), 133 deletions(-) create mode 100644 lib/libalpm/po/kk.po create mode 100644 po/kk.po hooks/post-receive -- The official pacman repository

1 0

[pacman-dev] [PATCH 11/11] repo-add : fix package variables declaration
by Xavier Chantry 27 Feb '09

27 Feb '09

- arch was missing - backup is not used by repo-add. However makepkg still needs to put it in PKGINFO because pacman uses it - startdir is no longer used after the new delta implementation - the declaration of group, depend, backup, etc is not needed because these variables are always declared before being used : declare $var="$val" case "$var" in group) _groups="$_groups$group\n" ;; - reorder the variables declaration to follow the same order than they are written to the depends and desc file, for making future checks easier Signed-off-by: Xavier Chantry <shiningxc(a)gmail.com> --- scripts/repo-add.sh.in | 12 ++++-------- 1 files changed, 4 insertions(+), 8 deletions(-) diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index 504adfd..4097a11 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -180,10 +180,8 @@ db_write_entry() { # blank out all variables local pkgfile="$1" - local pkgname pkgver pkgdesc url builddate packager csize size \ - group depend backup license replaces provides conflict force \ - _groups _depends _backups _licenses _replaces _provides _conflicts \ - startdir optdepend _optdepends md5sum + local pkgname pkgver pkgdesc csize size md5sum url arch builddate packager force \ + _groups _licenses _replaces _depends _conflicts _provides _optdepends local OLDIFS="$IFS" # IFS (field separator) is only the newline character @@ -200,12 +198,11 @@ db_write_entry() declare $var="$val" case "$var" in group) _groups="$_groups$group\n" ;; - depend) _depends="$_depends$depend\n" ;; - backup) _backups="$_backups$backup\n" ;; license) _licenses="$_licenses$license\n" ;; replaces) _replaces="$_replaces$replaces\n" ;; - provides) _provides="$_provides$provides\n" ;; + depend) _depends="$_depends$depend\n" ;; conflict) _conflicts="$_conflicts$conflict\n" ;; + provides) _provides="$_provides$provides\n" ;; optdepend) _optdepends="$_optdepends$optdepend\n" ;; esac done @@ -222,7 +219,6 @@ db_write_entry() return 1 fi - startdir=$(pwd) pushd "$gstmpdir" 2>&1 >/dev/null if [ -d "$pkgname-$pkgver" ]; then -- 1.6.1.3

1 0

[pacman-dev] repo-add sanity check
by Allan McRae 27 Feb '09

27 Feb '09

Hi, I took repo-add from the pacman package and one built from the latest git will all Xavier's patches and made a repo db from all packages in my cache (all of testing, core and extra, some community). There are a few differences (see http://dev.archlinux.org/~allan/repoadd-diff.txt) A summary of the differences: A bunch of packages have the ARCH field removed: -%ARCH% -i686 A few differences to do with quotes. e.g. -custom:INTEL537 +custom:"INTEL537" and +%DESC% +Efficient bit vector, set of integers and "big int" math library This change is not so good... (k3b-i18n package) %DEPENDS% -k3b>=1.0.5-1 +k3b>=$pkgver Allan

3 5

[pacman-dev] Disk Space
by Matthew Wood 26 Feb '09

26 Feb '09

I'm a little new to package management coding, so forgive my stupid question, but I'm wondering if there is a way to get the current package size. For example, like Aarons example, if foobar-1.0 is installed, and we got the size of foobar then we should have a reasonable size estimate of foobar-1.1. If we took that estimate, multiply by 10% and compare that number to the disk space remaining, would that catch 99% of the errors? If there isn't a current mechanism we could have a function that adds the total size of all the installed apps files and return that to the upgrade function. What about that coupled with a prompt if disk space seems low, say below 100-200 mb? I'd be willing to help with whatever patches are necessary for this.

2 1

[pacman-dev] Dan's and Xavier's git home
by Xavier 26 Feb '09

26 Feb '09

On Thu, Feb 26, 2009 at 10:25 AM, Allan McRae <allan(a)archlinux.org> wrote: > > OK. I will pull your patches into a git branch to remind myself to go > through these in a lot of detail sometime in the next week or so. Do you > have a git home now that Dan moved his server or do I need to apply them > from the mailing list? shining.toofishes.net does not seem to exist > anymore. > My repo has moved to Dan's new server, and they can now all be found on the same page, with a cgit interface : http://code.toofishes.net/cgit/ Thanks Dan :)

2 2

[pacman-dev] Disk Space
by Matthew Wood 24 Feb '09

24 Feb '09

What if we checked for disk space on all volumes, prior to the processing of the users command. If the function detects that disk space is lower than a certain amount, say 100 mb, to at least warn the user? This would of course be a temporary solution, for a more permanent possibility we could implement two functions that work in conjunction with each other. The first function would check the downloads to ensure they completed successfully, if they did then it would pass the work to a second function that checks the extracted package size against all mounted volumes, or at least the size of the volumes of the major installation location (/usr, /etc, /var, etc) Does this make any sense or am I barking up the wrong tree? Matt

5 5

[pacman-dev] Possible bug in pacman
by Matías De la Puente 24 Feb '09

24 Feb '09

hi all! i wanted to ignore a pkg in a group but i can with this: pacman -S gnome-extra --ignore orca Pacman after i say yes to install all the packages but then ask me if i want to install orcad that's insidce ignorePkg/ignoreGroup, i say no but then pacman exit saying that it can't found orca pkg or group Thanks matias

2 1