[pacman-dev] Add delta information to libalpm.
This patchset adds delta information to repository databases and libalpm, allowing libalpm to download multiple delta files when upgrading a package. I still have some work to do, but I thought I would post this now to get any suggestions and criticisms that anyone has. Some things I still need to do: Add error checking/md5sum to the download function. Remove some hardcoded stuff (like ".pkg.tar.gz"). Run through valgrind to make sure everything is freed. Delete delta files and intermediate packages after they are not needed. Test... The patches in action: $ ls -lh wbox*.pkg.tar.gz wbox*.delta -rw-r--r-- 1 nathanj users 55K 2007-10-15 17:11 wbox-3-1-i686.pkg.tar.gz -rw-r--r-- 1 nathanj users 7.9K 2007-10-15 17:11 wbox-3-1_to_4-1-i686.delta -rw-r--r-- 1 nathanj users 56K 2007-10-15 17:11 wbox-4-1-i686.pkg.tar.gz -rw-r--r-- 1 nathanj users 398 2007-10-15 17:11 wbox-4-1_to_4-2-i686.delta -rw-r--r-- 1 nathanj users 56K 2007-10-15 17:11 wbox-4-2-i686.pkg.tar.gz $ repo-add test.db.tar.gz wbox-4-2-i686.pkg.tar.gz ==> Extracting database to a temporary location... ==> Adding package 'wbox-4-2-i686.pkg.tar.gz' -> Removing existing package 'wbox-3-1'... -> Creating 'desc' db entry... -> Computing md5 checksums... -> Creating 'depends' db entry... -> Creating 'deltas' db entry... -> Added delta 'wbox-3-1_to_4-1-i686.delta' -> Added delta 'wbox-4-1_to_4-2-i686.delta' ==> Creating updated database file /home/nathanj/pkgs/pkgs/test.db.tar.gz $ pacman -Q wbox wbox 3-1 $ sudo pacman -Sy wbox :: Synchronizing package databases... test 3.4K 8.2M/s 00:00:00 [#####################] 100% resolving dependencies... done. looking for inter-conflicts... done. Targets: wbox-4-2 Total Package Size: 0.05 MB Total Download Size 0.01 MB Total Installed Size: 0.08 MB Proceed with installation? [Y/n] :: Retrieving packages from test... wbox-3-1_to_4-1-i686... 7.9K 23.3M/s 00:00:00 [#####################] 100% wbox-4-1_to_4-2-i686... 0.4K 5.3M/s 00:00:00 [#####################] 100% Generating wbox-4-1-i686.pkg.tar.gz with wbox-3-1_to_4-1-i686.delta... done. Generating wbox-4-2-i686.pkg.tar.gz with wbox-4-1_to_4-2-i686.delta... done. checking package integrity... done. (1/1) checking for file conflicts [#####################] 100% (1/1) upgrading wbox [#####################] 100%
Signed-off-by: Nathan Jones <nathanj@insightbb.com> --- scripts/repo-add.sh.in | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 45 insertions(+), 1 deletions(-) diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index cb741d7..67fa4f1 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -97,6 +97,34 @@ write_list_entry() { fi } +# write a delta entry to the pacman database +# arg1 - path to delta +db_write_delta() +{ + # blank out all variables and set deltafile + local deltafile=$(readlink -f "$1") + local filename=$(basename "$deltafile") + local pkgname fromver tover arch csize md5sum + + pkgname="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\1/')" + fromver="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\2/')" + tover="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\3/')" + arch="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\4/')" + md5sum="$(md5sum "$deltafile" | cut -d ' ' -f 1)" + + # get size of delta + csize=$(du -b -L "$deltafile" | cut -f 1) + + # ensure $pkgname and $pkgver variables were found + if [ -z "$pkgname" -o -z "$fromver" ]; then + return 1 + fi + + # add the entry for this delta file + echo -e "$fromver $tover $csize $filename $md5sum" >>deltas +} # end db_write_delta + + # write an entry to the pacman database # arg1 - path to package db_write_entry() @@ -106,6 +134,7 @@ db_write_entry() local pkgname pkgver pkgdesc url builddate packager csize size \ group depend backup license replaces provides conflict \ _groups _depends _backups _licenses _replaces _provides _conflicts + local startdir local OLDIFS="$IFS" # IFS (field seperator) is only the newline character @@ -133,6 +162,7 @@ db_write_entry() # get compressed size of package csize=$(du -b -L "$pkgfile" | cut -f 1) + startdir=`pwd` pushd "$gstmpdir" 2>&1 >/dev/null # ensure $pkgname and $pkgver variables were found @@ -183,8 +213,22 @@ db_write_entry() write_list_entry "CONFLICTS" "$_conflicts" "depends" write_list_entry "PROVIDES" "$_provides" "depends" + # create deltas entry + msg2 "$(gettext "Creating 'deltas' db entry...")" + echo -e "%DELTAS%" >>deltas + for delta in $startdir/$pkgname-*-*_to_*-*-$arch.delta; do + if [ -e "$delta" ]; then + if db_write_delta "$delta"; then + msg2 "$(gettext "Added delta '%s'")" "$(basename "$delta")" + else + msg2 "$(gettext "Could not add delta '%s'")" "$(basename "$delta")" + fi + fi + done + echo -e "" >>deltas + # preserve the modification time - touch -r "$pkgfile" desc depends + touch -r "$pkgfile" desc depends deltas popd 2>&1 >/dev/null } # end db_write_entry -- 1.5.3.4
I'll just go through one patch at a time and make some inline comments. Overall they look very good though. On 10/15/07, Nathan Jones <nathanj@insightbb.com> wrote:
Signed-off-by: Nathan Jones <nathanj@insightbb.com> --- scripts/repo-add.sh.in | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 45 insertions(+), 1 deletions(-)
diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index cb741d7..67fa4f1 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -97,6 +97,34 @@ write_list_entry() { fi }
+# write a delta entry to the pacman database +# arg1 - path to delta +db_write_delta() +{ + # blank out all variables and set deltafile + local deltafile=$(readlink -f "$1") + local filename=$(basename "$deltafile") + local pkgname fromver tover arch csize md5sum + + pkgname="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\1/')" + fromver="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\2/')" + tover="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\3/')" + arch="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\4/')" + md5sum="$(md5sum "$deltafile" | cut -d ' ' -f 1)"
Is there any way to do it without so much sed-magic, or at least make it look slightly cleaner? If not, then we'll have to live with it. Can you add some comments above it explaining the regex in case someone else ever has to change it though?
+ + # get size of delta + csize=$(du -b -L "$deltafile" | cut -f 1) + + # ensure $pkgname and $pkgver variables were found + if [ -z "$pkgname" -o -z "$fromver" ]; then + return 1 + fi + + # add the entry for this delta file + echo -e "$fromver $tover $csize $filename $md5sum" >>deltas +} # end db_write_delta + + # write an entry to the pacman database # arg1 - path to package db_write_entry() @@ -106,6 +134,7 @@ db_write_entry() local pkgname pkgver pkgdesc url builddate packager csize size \ group depend backup license replaces provides conflict \ _groups _depends _backups _licenses _replaces _provides _conflicts + local startdir
Might as well omit the 'local' and stick it in somewhere in the big list that makes sense.
local OLDIFS="$IFS" # IFS (field seperator) is only the newline character @@ -133,6 +162,7 @@ db_write_entry() # get compressed size of package csize=$(du -b -L "$pkgfile" | cut -f 1)
+ startdir=`pwd`
Please use the $() format instead of ``, we've removed all `` escapes throughout the pacman toolset.
pushd "$gstmpdir" 2>&1 >/dev/null
# ensure $pkgname and $pkgver variables were found @@ -183,8 +213,22 @@ db_write_entry() write_list_entry "CONFLICTS" "$_conflicts" "depends" write_list_entry "PROVIDES" "$_provides" "depends"
+ # create deltas entry + msg2 "$(gettext "Creating 'deltas' db entry...")" + echo -e "%DELTAS%" >>deltas + for delta in $startdir/$pkgname-*-*_to_*-*-$arch.delta; do + if [ -e "$delta" ]; then + if db_write_delta "$delta"; then + msg2 "$(gettext "Added delta '%s'")" "$(basename "$delta")" + else + msg2 "$(gettext "Could not add delta '%s'")" "$(basename "$delta")" + fi + fi + done + echo -e "" >>deltas + # preserve the modification time - touch -r "$pkgfile" desc depends + touch -r "$pkgfile" desc depends deltas
Will this create an empty deltas file if it doesn't exist? We probably don't want that (just db clutter).
popd 2>&1 >/dev/null } # end db_write_entry -- 1.5.3.4
-Dan
On Mon, Oct 15, 2007 at 09:24:19PM -0500, Dan McGee wrote:
+ pkgname="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\1/')" + fromver="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\2/')" + tover="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\3/')" + arch="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\4/')" + md5sum="$(md5sum "$deltafile" | cut -d ' ' -f 1)"
Is there any way to do it without so much sed-magic, or at least make it look slightly cleaner? If not, then we'll have to live with it. Can you add some comments above it explaining the regex in case someone else ever has to change it though?
This was the only method I could think of, aside from creating the delta as a tar file with a .DELTAINFO like packages. I'll add the comment.
+ # create deltas entry + msg2 "$(gettext "Creating 'deltas' db entry...")" + echo -e "%DELTAS%" >>deltas + for delta in $startdir/$pkgname-*-*_to_*-*-$arch.delta; do + if [ -e "$delta" ]; then + if db_write_delta "$delta"; then + msg2 "$(gettext "Added delta '%s'")" "$(basename "$delta")" + else + msg2 "$(gettext "Could not add delta '%s'")" "$(basename "$delta")" + fi + fi + done + echo -e "" >>deltas + # preserve the modification time - touch -r "$pkgfile" desc depends + touch -r "$pkgfile" desc depends deltas
Will this create an empty deltas file if it doesn't exist? We probably don't want that (just db clutter).
The deltas file will always have at least '%DELTAS%\n' in it. I will change it so it is only created if needed.
On Tue, 2007-10-16 at 09:03 -0400, Nathan Jones wrote:
On Mon, Oct 15, 2007 at 09:24:19PM -0500, Dan McGee wrote:
+ pkgname="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\1/')" + fromver="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\2/')" + tover="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\3/')" + arch="$(echo "$filename" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\4/')" + md5sum="$(md5sum "$deltafile" | cut -d ' ' -f 1)"
Is there any way to do it without so much sed-magic, or at least make it look slightly cleaner? If not, then we'll have to live with it. Can you add some comments above it explaining the regex in case someone else ever has to change it though?
This was the only method I could think of, aside from creating the delta as a tar file with a .DELTAINFO like packages. I'll add the comment.
You could remove the extra parentheses: echo ${filename} | sed -e 's/^\(.*\)-.*-.*_to_.*$/\1/' echo ${filename} | sed -e 's/^.*-\(.*-.*\)_to_.*$/\1/' echo ${filename} | sed -e 's/^.*_to_\(.*-.*\)-.*$/\1/' echo ${filename} | sed -e 's/^.*_to_.*-.*-\(.*\)\..*$/\1/' or dump the parts to an array: arr=(`echo ${filename} | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.* \).delta/\1 \2 \3 \4/'`) To bad there is no "non-greedy" modifier in sed.
+ # create deltas entry + msg2 "$(gettext "Creating 'deltas' db entry...")" + echo -e "%DELTAS%" >>deltas + for delta in $startdir/$pkgname-*-*_to_*-*-$arch.delta; do + if [ -e "$delta" ]; then + if db_write_delta "$delta"; then + msg2 "$(gettext "Added delta '%s'")" "$(basename "$delta")" + else + msg2 "$(gettext "Could not add delta '%s'")" "$(basename "$delta")" + fi + fi + done + echo -e "" >>deltas + # preserve the modification time - touch -r "$pkgfile" desc depends + touch -r "$pkgfile" desc depends deltas
Will this create an empty deltas file if it doesn't exist? We probably don't want that (just db clutter).
The deltas file will always have at least '%DELTAS%\n' in it. I will change it so it is only created if needed.
_______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://archlinux.org/mailman/listinfo/pacman-dev
Signed-off-by: Nathan Jones <nathanj@insightbb.com> --- lib/libalpm/Makefile.am | 1 + lib/libalpm/alpm.h | 13 +++ lib/libalpm/alpm_list.c | 18 ++++ lib/libalpm/alpm_list.h | 1 + lib/libalpm/be_files.c | 19 ++++ lib/libalpm/db.h | 3 +- lib/libalpm/delta.c | 249 +++++++++++++++++++++++++++++++++++++++++++++++ lib/libalpm/delta.h | 44 ++++++++ lib/libalpm/package.c | 15 +++ lib/libalpm/package.h | 1 + 10 files changed, 363 insertions(+), 1 deletions(-) create mode 100644 lib/libalpm/delta.c create mode 100644 lib/libalpm/delta.h diff --git a/lib/libalpm/Makefile.am b/lib/libalpm/Makefile.am index 6948239..49ce14c 100644 --- a/lib/libalpm/Makefile.am +++ b/lib/libalpm/Makefile.am @@ -18,6 +18,7 @@ libalpm_la_SOURCES = \ cache.h cache.c \ conflict.h conflict.c \ db.h db.c \ + delta.h delta.c \ deps.h deps.c \ error.h error.c \ group.h group.c \ diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 6eda210..527822d 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -45,6 +45,7 @@ extern "C" { typedef struct __pmdb_t pmdb_t; typedef struct __pmpkg_t pmpkg_t; +typedef struct __pmdelta_t pmdelta_t; typedef struct __pmgrp_t pmgrp_t; typedef struct __pmserver_t pmserver_t; typedef struct __pmtrans_t pmtrans_t; @@ -207,12 +208,24 @@ alpm_list_t *alpm_pkg_get_optdepends(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_requiredby(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_conflicts(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_provides(pmpkg_t *pkg); +alpm_list_t *alpm_pkg_get_deltas(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_replaces(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_files(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_backup(pmpkg_t *pkg); unsigned short alpm_pkg_has_scriptlet(pmpkg_t *pkg); /* + * Deltas + */ + +char *alpm_delta_get_from(pmdelta_t *delta); +char *alpm_delta_get_to(pmdelta_t *delta); +unsigned long alpm_delta_get_size(pmdelta_t *delta); +char *alpm_delta_get_filename(pmdelta_t *delta); +char *alpm_delta_get_md5sum(pmdelta_t *delta); +alpm_list_t *alpm_shortest_delta_path(alpm_list_t *deltas, const char *from, const char *to); + +/* * Groups */ const char *alpm_grp_get_name(const pmgrp_t *grp); diff --git a/lib/libalpm/alpm_list.c b/lib/libalpm/alpm_list.c index 5671b4a..2301411 100644 --- a/lib/libalpm/alpm_list.c +++ b/lib/libalpm/alpm_list.c @@ -379,6 +379,24 @@ alpm_list_t SYMEXPORT *alpm_list_strdup(const alpm_list_t *list) } /** + * @brief Copy a list, without copying data. + * + * @param list the list to copy + * + * @return a copy of the original list + */ +alpm_list_t SYMEXPORT *alpm_list_copy(const alpm_list_t *list) +{ + const alpm_list_t *lp = list; + alpm_list_t *newlist = NULL; + while(lp) { + newlist = alpm_list_add(newlist, lp->data); + lp = lp->next; + } + return(newlist); +} + +/** * @brief Create a new list in reverse order. * * @param list the list to copy diff --git a/lib/libalpm/alpm_list.h b/lib/libalpm/alpm_list.h index a2a06f4..8fce280 100644 --- a/lib/libalpm/alpm_list.h +++ b/lib/libalpm/alpm_list.h @@ -60,6 +60,7 @@ alpm_list_t *alpm_list_remove(alpm_list_t *haystack, const void *needle, alpm_li alpm_list_t *alpm_list_remove_node(alpm_list_t *node); alpm_list_t *alpm_list_remove_dupes(const alpm_list_t *list); alpm_list_t *alpm_list_strdup(const alpm_list_t *list); +alpm_list_t *alpm_list_copy(const alpm_list_t *list); alpm_list_t *alpm_list_reverse(alpm_list_t *list); /* item accessors */ diff --git a/lib/libalpm/be_files.c b/lib/libalpm/be_files.c index 2bfdb95..59155cf 100644 --- a/lib/libalpm/be_files.c +++ b/lib/libalpm/be_files.c @@ -44,6 +44,7 @@ #include "error.h" #include "handle.h" #include "package.h" +#include "delta.h" /* This function is used to convert the downloaded db file to the proper backend @@ -484,6 +485,24 @@ int _alpm_db_read(pmdb_t *db, pmpkg_t *info, pmdbinfrq_t inforeq) fp = NULL; } + /* DELTAS */ + if(inforeq & INFRQ_DELTAS) { + snprintf(path, PATH_MAX, "%s/%s-%s/deltas", db->path, info->name, info->version); + if((fp = fopen(path, "r"))) { + while(!feof(fp)) { + fgets(line, 255, fp); + _alpm_strtrim(line); + if(!strcmp(line, "%DELTAS%")) { + while(fgets(line, 512, fp) && strlen(_alpm_strtrim(line))) { + info->deltas = alpm_list_add(info->deltas, _alpm_delta_parse(line)); + } + } + } + fclose(fp); + fp = NULL; + } + } + /* INSTALL */ if(inforeq & INFRQ_SCRIPTLET) { snprintf(path, PATH_MAX, "%s/%s-%s/install", db->path, info->name, info->version); diff --git a/lib/libalpm/db.h b/lib/libalpm/db.h index 3ee4977..16e1298 100644 --- a/lib/libalpm/db.h +++ b/lib/libalpm/db.h @@ -33,8 +33,9 @@ typedef enum _pmdbinfrq_t { INFRQ_DEPENDS = 0x04, INFRQ_FILES = 0x08, INFRQ_SCRIPTLET = 0x10, + INFRQ_DELTAS = 0x20, /* ALL should be sum of all above */ - INFRQ_ALL = 0x1F + INFRQ_ALL = 0x3F } pmdbinfrq_t; /* Database */ diff --git a/lib/libalpm/delta.c b/lib/libalpm/delta.c new file mode 100644 index 0000000..1760a3c --- /dev/null +++ b/lib/libalpm/delta.c @@ -0,0 +1,249 @@ +/* + * delta.c + * + * Copyright (c) 2002-2006 by Judd Vinet <jvinet@zeroflux.org> + * Copyright (c) 2005 by Aurelien Foret <orelien@chez.com> + * Copyright (c) 2005, 2006 by Christian Hamar <krics@linuxforum.hu> + * Copyright (c) 2005, 2006 by Miklos Vajna <vmiklos@frugalware.org> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, + * USA. + */ + +#include "config.h" + +#include <stdlib.h> +#include <string.h> + +/* libalpm */ +#include "delta.h" +#include "util.h" +#include "log.h" +#include "alpm_list.h" +#include "alpm.h" + +/** \addtogroup alpm_deltas Delta Functions + * @brief Functions to manipulate libalpm deltas + * @{ + */ + +char SYMEXPORT *alpm_delta_get_from(pmdelta_t *delta) +{ + ALPM_LOG_FUNC; + + return delta->from; +} + +char SYMEXPORT *alpm_delta_get_to(pmdelta_t *delta) +{ + ALPM_LOG_FUNC; + + return delta->to; +} + +unsigned long SYMEXPORT alpm_delta_get_size(pmdelta_t *delta) +{ + ALPM_LOG_FUNC; + + return delta->size; +} + +char SYMEXPORT *alpm_delta_get_filename(pmdelta_t *delta) +{ + ALPM_LOG_FUNC; + + return delta->filename; +} + +char SYMEXPORT *alpm_delta_get_md5sum(pmdelta_t *delta) +{ + ALPM_LOG_FUNC; + + return delta->md5sum; +} + +/** @} */ + +/** Calculates the combined size of a list of delta files. + * + * @param deltas the list of pmdelta_t * objects + * + * @return the combined size + */ +unsigned long _alpm_delta_path_size(alpm_list_t *deltas) +{ + unsigned long sum = 0; + alpm_list_t *dlts = deltas; + + while(dlts) { + pmdelta_t *d = (pmdelta_t *)alpm_list_getdata(dlts); + sum += d->size; + + dlts = alpm_list_next(dlts); + } + + return sum; +} + +/** Calculates the combined size of a list of delta files that are not + * in the cache. + * + * @param deltas the list of pmdelta_t * objects + * + * @return the combined size + */ +unsigned long _alpm_delta_path_size_uncached(alpm_list_t *deltas) +{ + unsigned long sum = 0; + alpm_list_t *dlts = deltas; + + while(dlts) { + pmdelta_t *d = (pmdelta_t *)alpm_list_getdata(dlts); + char *fname = _alpm_filecache_find(d->filename); + + if(!fname) { + sum += d->size; + } + + FREE(fname); + + dlts = alpm_list_next(dlts); + } + + return sum; +} + +/** Calculates the shortest path from one version to another. + * + * The shortest path is defined as the path with the smallest combined + * size, not the length of the path. + * + * The algorithm used is Dijkstra's shortest path algorithm. + * + * @param deltas the list of pmdelta_t * objects that a package has + * @param from the version to start from + * @param to the version to end at + * @param path the current path + * + * @return the list of pmdelta_t * objects that has the smallest size. + * NULL (the empty list) is returned if there is no path between the + * versions. + */ +static alpm_list_t *shortest_delta_path(alpm_list_t *deltas, + const char *from, const char *to, alpm_list_t *path) +{ + alpm_list_t *d; + alpm_list_t *shortest = NULL; + + /* Found the 'to' version, this is a good path so return it. */ + if(strcmp(from, to) == 0) + return path; + + for(d = deltas; d; d = alpm_list_next(d)) { + pmdelta_t *v = alpm_list_getdata(d); + + /* If this vertex has already been visited in the path, go to the + * next vertex. */ + if(alpm_list_find(path, v)) + continue; + + /* Once we find a vertex that starts at the 'from' version, + * recursively find the shortest path using the 'to' version of this + * current vertex as the 'from' version in the function call. */ + if(strcmp(v->from, from) == 0) { + alpm_list_t *newpath = alpm_list_copy(path); + newpath = alpm_list_add(newpath, v); + + if((newpath = shortest_delta_path(deltas, v->to, to, newpath))) { + /* The path returned works, now use it unless there is already a + * shorter path found. */ + if(shortest == NULL) { + shortest = newpath; + } else if(_alpm_delta_path_size(shortest) > _alpm_delta_path_size(newpath)) { + alpm_list_free(shortest); + shortest = newpath; + } else { + alpm_list_free(newpath); + } + } + } + } + + return shortest; +} + +/** Calculates the shortest path from one version to another. + * + * The shortest path is defined as the path with the smallest combined + * size, not the length of the path. + * + * @param deltas the list of pmdelta_t * objects that a package has + * @param from the version to start from + * @param to the version to end at + * + * @return the list of pmdelta_t * objects that has the smallest size. + * NULL (the empty list) is returned if there is no path between the + * versions. + */ +alpm_list_t SYMEXPORT *alpm_shortest_delta_path(alpm_list_t *deltas, const char *from, const char *to) +{ + alpm_list_t *path = NULL; + + path = shortest_delta_path(deltas, from, to, path); + + return path; +} + +/** Parses the string representation of a pmdelta_t object. + * + * This function assumes that the string is in the correct format. + * + * @param line the string to parse + * + * @return A pointer to the new pmdelta_t object + */ +pmdelta_t *_alpm_delta_parse(char *line) +{ + pmdelta_t *delta; + char *tmp = line, *tmp2; + + delta = malloc(sizeof(pmdelta_t)); + + tmp2 = tmp; + tmp = strchr(tmp, ' '); + *(tmp++) = '\0'; + delta->from = strdup(tmp2); + + tmp2 = tmp; + tmp = strchr(tmp, ' '); + *(tmp++) = '\0'; + delta->to = strdup(tmp2); + + tmp2 = tmp; + tmp = strchr(tmp, ' '); + *(tmp++) = '\0'; + delta->size = atol(tmp2); + + tmp2 = tmp; + tmp = strchr(tmp, ' '); + *(tmp++) = '\0'; + delta->filename = strdup(tmp2); + + delta->md5sum = strdup(tmp); + + return delta; +} + +/* vim: set ts=2 sw=2 noet: */ diff --git a/lib/libalpm/delta.h b/lib/libalpm/delta.h new file mode 100644 index 0000000..4655d7a --- /dev/null +++ b/lib/libalpm/delta.h @@ -0,0 +1,44 @@ +/* + * delta.h + * + * Copyright (c) 2002-2006 by Judd Vinet <jvinet@zeroflux.org> + * Copyright (c) 2005 by Aurelien Foret <orelien@chez.com> + * Copyright (c) 2006 by David Kimpe <dnaku@frugalware.org> + * Copyright (c) 2005, 2006 by Christian Hamar <krics@linuxforum.hu> + * Copyright (c) 2005, 2006 by Miklos Vajna <vmiklos@frugalware.org> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, + * USA. + */ +#ifndef _ALPM_DELTA_H +#define _ALPM_DELTA_H + +#include "alpm.h" + +struct __pmdelta_t { + char *from; + char *to; + unsigned long size; + char *filename; + char *md5sum; +}; + +unsigned long _alpm_delta_path_size(alpm_list_t *deltas); +unsigned long _alpm_delta_path_size_uncached(alpm_list_t *deltas); +pmdelta_t *_alpm_delta_parse(char *line); + +#endif /* _ALPM_DELTA_H */ + +/* vim: set ts=2 sw=2 noet: */ diff --git a/lib/libalpm/package.c b/lib/libalpm/package.c index 38e6e4c..6a8e0d8 100644 --- a/lib/libalpm/package.c +++ b/lib/libalpm/package.c @@ -446,6 +446,20 @@ alpm_list_t SYMEXPORT *alpm_pkg_get_provides(pmpkg_t *pkg) return pkg->provides; } +alpm_list_t SYMEXPORT *alpm_pkg_get_deltas(pmpkg_t *pkg) +{ + ALPM_LOG_FUNC; + + /* Sanity checks */ + ASSERT(handle != NULL, return(NULL)); + ASSERT(pkg != NULL, return(NULL)); + + if(pkg->origin == PKG_FROM_CACHE && !(pkg->infolevel & INFRQ_DELTAS)) { + _alpm_db_read(pkg->origin_data.db, pkg, INFRQ_DELTAS); + } + return pkg->deltas; +} + alpm_list_t SYMEXPORT *alpm_pkg_get_replaces(pmpkg_t *pkg) { ALPM_LOG_FUNC; @@ -725,6 +739,7 @@ void _alpm_pkg_free(pmpkg_t *pkg) FREELIST(pkg->groups); FREELIST(pkg->provides); FREELIST(pkg->replaces); + FREELIST(pkg->deltas); if(pkg->origin == PKG_FROM_FILE) { FREE(pkg->origin_data.file); } diff --git a/lib/libalpm/package.h b/lib/libalpm/package.h index 42ebe0e..daaeb36 100644 --- a/lib/libalpm/package.h +++ b/lib/libalpm/package.h @@ -75,6 +75,7 @@ struct __pmpkg_t { alpm_list_t *requiredby; alpm_list_t *conflicts; alpm_list_t *provides; + alpm_list_t *deltas; /* internal */ pmpkgfrom_t origin; /* Replaced 'void *data' with this union as follows: -- 1.5.3.4
On 10/15/07, Nathan Jones <nathanj@insightbb.com> wrote:
Signed-off-by: Nathan Jones <nathanj@insightbb.com> --- lib/libalpm/Makefile.am | 1 + lib/libalpm/alpm.h | 13 +++ lib/libalpm/alpm_list.c | 18 ++++ lib/libalpm/alpm_list.h | 1 + lib/libalpm/be_files.c | 19 ++++ lib/libalpm/db.h | 3 +- lib/libalpm/delta.c | 249 +++++++++++++++++++++++++++++++++++++++++++++++ lib/libalpm/delta.h | 44 ++++++++ lib/libalpm/package.c | 15 +++ lib/libalpm/package.h | 1 + 10 files changed, 363 insertions(+), 1 deletions(-) create mode 100644 lib/libalpm/delta.c create mode 100644 lib/libalpm/delta.h
diff --git a/lib/libalpm/Makefile.am b/lib/libalpm/Makefile.am index 6948239..49ce14c 100644 --- a/lib/libalpm/Makefile.am +++ b/lib/libalpm/Makefile.am @@ -18,6 +18,7 @@ libalpm_la_SOURCES = \ cache.h cache.c \ conflict.h conflict.c \ db.h db.c \ + delta.h delta.c \ deps.h deps.c \ error.h error.c \ group.h group.c \
Glad you caught this, but can you add the two files to libalpm/po/POTFILES.in as well?
diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 6eda210..527822d 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -45,6 +45,7 @@ extern "C" {
typedef struct __pmdb_t pmdb_t; typedef struct __pmpkg_t pmpkg_t; +typedef struct __pmdelta_t pmdelta_t; typedef struct __pmgrp_t pmgrp_t; typedef struct __pmserver_t pmserver_t; typedef struct __pmtrans_t pmtrans_t; @@ -207,12 +208,24 @@ alpm_list_t *alpm_pkg_get_optdepends(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_requiredby(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_conflicts(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_provides(pmpkg_t *pkg); +alpm_list_t *alpm_pkg_get_deltas(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_replaces(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_files(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_backup(pmpkg_t *pkg); unsigned short alpm_pkg_has_scriptlet(pmpkg_t *pkg);
/* + * Deltas + */ + +char *alpm_delta_get_from(pmdelta_t *delta); +char *alpm_delta_get_to(pmdelta_t *delta); +unsigned long alpm_delta_get_size(pmdelta_t *delta); +char *alpm_delta_get_filename(pmdelta_t *delta); +char *alpm_delta_get_md5sum(pmdelta_t *delta); +alpm_list_t *alpm_shortest_delta_path(alpm_list_t *deltas, const char *from, const char *to); + +/* * Groups */ const char *alpm_grp_get_name(const pmgrp_t *grp); diff --git a/lib/libalpm/alpm_list.c b/lib/libalpm/alpm_list.c index 5671b4a..2301411 100644 --- a/lib/libalpm/alpm_list.c +++ b/lib/libalpm/alpm_list.c @@ -379,6 +379,24 @@ alpm_list_t SYMEXPORT *alpm_list_strdup(const alpm_list_t *list) }
/** + * @brief Copy a list, without copying data. + * + * @param list the list to copy + * + * @return a copy of the original list + */ +alpm_list_t SYMEXPORT *alpm_list_copy(const alpm_list_t *list) +{ + const alpm_list_t *lp = list; + alpm_list_t *newlist = NULL; + while(lp) { + newlist = alpm_list_add(newlist, lp->data); + lp = lp->next; + } + return(newlist); +} + +/** * @brief Create a new list in reverse order. * * @param list the list to copy diff --git a/lib/libalpm/alpm_list.h b/lib/libalpm/alpm_list.h index a2a06f4..8fce280 100644 --- a/lib/libalpm/alpm_list.h +++ b/lib/libalpm/alpm_list.h @@ -60,6 +60,7 @@ alpm_list_t *alpm_list_remove(alpm_list_t *haystack, const void *needle, alpm_li alpm_list_t *alpm_list_remove_node(alpm_list_t *node); alpm_list_t *alpm_list_remove_dupes(const alpm_list_t *list); alpm_list_t *alpm_list_strdup(const alpm_list_t *list); +alpm_list_t *alpm_list_copy(const alpm_list_t *list); alpm_list_t *alpm_list_reverse(alpm_list_t *list);
/* item accessors */ diff --git a/lib/libalpm/be_files.c b/lib/libalpm/be_files.c index 2bfdb95..59155cf 100644 --- a/lib/libalpm/be_files.c +++ b/lib/libalpm/be_files.c @@ -44,6 +44,7 @@ #include "error.h" #include "handle.h" #include "package.h" +#include "delta.h"
/* This function is used to convert the downloaded db file to the proper backend @@ -484,6 +485,24 @@ int _alpm_db_read(pmdb_t *db, pmpkg_t *info, pmdbinfrq_t inforeq) fp = NULL; }
+ /* DELTAS */ + if(inforeq & INFRQ_DELTAS) { + snprintf(path, PATH_MAX, "%s/%s-%s/deltas", db->path, info->name, info->version); + if((fp = fopen(path, "r"))) { + while(!feof(fp)) { + fgets(line, 255, fp); + _alpm_strtrim(line); + if(!strcmp(line, "%DELTAS%")) { + while(fgets(line, 512, fp) && strlen(_alpm_strtrim(line))) { + info->deltas = alpm_list_add(info->deltas, _alpm_delta_parse(line)); + } + } + } + fclose(fp); + fp = NULL; + } + } + /* INSTALL */ if(inforeq & INFRQ_SCRIPTLET) { snprintf(path, PATH_MAX, "%s/%s-%s/install", db->path, info->name, info->version); diff --git a/lib/libalpm/db.h b/lib/libalpm/db.h index 3ee4977..16e1298 100644 --- a/lib/libalpm/db.h +++ b/lib/libalpm/db.h @@ -33,8 +33,9 @@ typedef enum _pmdbinfrq_t { INFRQ_DEPENDS = 0x04, INFRQ_FILES = 0x08, INFRQ_SCRIPTLET = 0x10, + INFRQ_DELTAS = 0x20, /* ALL should be sum of all above */ - INFRQ_ALL = 0x1F + INFRQ_ALL = 0x3F } pmdbinfrq_t;
/* Database */ diff --git a/lib/libalpm/delta.c b/lib/libalpm/delta.c new file mode 100644 index 0000000..1760a3c --- /dev/null +++ b/lib/libalpm/delta.c @@ -0,0 +1,249 @@ +/* + * delta.c + * + * Copyright (c) 2002-2006 by Judd Vinet <jvinet@zeroflux.org> + * Copyright (c) 2005 by Aurelien Foret <orelien@chez.com> + * Copyright (c) 2005, 2006 by Christian Hamar <krics@linuxforum.hu> + * Copyright (c) 2005, 2006 by Miklos Vajna <vmiklos@frugalware.org> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, + * USA. + */ + +#include "config.h" + +#include <stdlib.h> +#include <string.h> + +/* libalpm */ +#include "delta.h" +#include "util.h" +#include "log.h" +#include "alpm_list.h" +#include "alpm.h" + +/** \addtogroup alpm_deltas Delta Functions + * @brief Functions to manipulate libalpm deltas + * @{ + */ + +char SYMEXPORT *alpm_delta_get_from(pmdelta_t *delta) +{ + ALPM_LOG_FUNC; + + return delta->from; +} + +char SYMEXPORT *alpm_delta_get_to(pmdelta_t *delta) +{ + ALPM_LOG_FUNC; + + return delta->to; +} + +unsigned long SYMEXPORT alpm_delta_get_size(pmdelta_t *delta) +{ + ALPM_LOG_FUNC; + + return delta->size; +} + +char SYMEXPORT *alpm_delta_get_filename(pmdelta_t *delta) +{ + ALPM_LOG_FUNC; + + return delta->filename; +} + +char SYMEXPORT *alpm_delta_get_md5sum(pmdelta_t *delta) +{ + ALPM_LOG_FUNC; + + return delta->md5sum; +} + +/** @} */ + +/** Calculates the combined size of a list of delta files. + * + * @param deltas the list of pmdelta_t * objects + * + * @return the combined size + */ +unsigned long _alpm_delta_path_size(alpm_list_t *deltas) +{ + unsigned long sum = 0; + alpm_list_t *dlts = deltas; + + while(dlts) { + pmdelta_t *d = (pmdelta_t *)alpm_list_getdata(dlts); + sum += d->size; + + dlts = alpm_list_next(dlts); + } + + return sum; +} + +/** Calculates the combined size of a list of delta files that are not + * in the cache. + * + * @param deltas the list of pmdelta_t * objects + * + * @return the combined size + */ +unsigned long _alpm_delta_path_size_uncached(alpm_list_t *deltas) +{ + unsigned long sum = 0; + alpm_list_t *dlts = deltas; + + while(dlts) { + pmdelta_t *d = (pmdelta_t *)alpm_list_getdata(dlts); + char *fname = _alpm_filecache_find(d->filename); + + if(!fname) { + sum += d->size; + } + + FREE(fname); + + dlts = alpm_list_next(dlts); + } + + return sum; +} + +/** Calculates the shortest path from one version to another. + * + * The shortest path is defined as the path with the smallest combined + * size, not the length of the path. + * + * The algorithm used is Dijkstra's shortest path algorithm. + * + * @param deltas the list of pmdelta_t * objects that a package has + * @param from the version to start from + * @param to the version to end at + * @param path the current path + * + * @return the list of pmdelta_t * objects that has the smallest size. + * NULL (the empty list) is returned if there is no path between the + * versions. + */ +static alpm_list_t *shortest_delta_path(alpm_list_t *deltas, + const char *from, const char *to, alpm_list_t *path) +{ + alpm_list_t *d; + alpm_list_t *shortest = NULL; + + /* Found the 'to' version, this is a good path so return it. */ + if(strcmp(from, to) == 0) + return path; + + for(d = deltas; d; d = alpm_list_next(d)) { + pmdelta_t *v = alpm_list_getdata(d); + + /* If this vertex has already been visited in the path, go to the + * next vertex. */ + if(alpm_list_find(path, v)) + continue; + + /* Once we find a vertex that starts at the 'from' version, + * recursively find the shortest path using the 'to' version of this + * current vertex as the 'from' version in the function call. */ + if(strcmp(v->from, from) == 0) { + alpm_list_t *newpath = alpm_list_copy(path); + newpath = alpm_list_add(newpath, v); + + if((newpath = shortest_delta_path(deltas, v->to, to, newpath))) { + /* The path returned works, now use it unless there is already a + * shorter path found. */ + if(shortest == NULL) { + shortest = newpath; + } else if(_alpm_delta_path_size(shortest) > _alpm_delta_path_size(newpath)) { + alpm_list_free(shortest); + shortest = newpath; + } else { + alpm_list_free(newpath); + } + } + } + } + + return shortest; +} + +/** Calculates the shortest path from one version to another. + * + * The shortest path is defined as the path with the smallest combined + * size, not the length of the path. + * + * @param deltas the list of pmdelta_t * objects that a package has + * @param from the version to start from + * @param to the version to end at + * + * @return the list of pmdelta_t * objects that has the smallest size. + * NULL (the empty list) is returned if there is no path between the + * versions. + */ +alpm_list_t SYMEXPORT *alpm_shortest_delta_path(alpm_list_t *deltas, const char *from, const char *to) +{ + alpm_list_t *path = NULL; + + path = shortest_delta_path(deltas, from, to, path); + + return path; +} + +/** Parses the string representation of a pmdelta_t object. + * + * This function assumes that the string is in the correct format. + * + * @param line the string to parse + * + * @return A pointer to the new pmdelta_t object + */ +pmdelta_t *_alpm_delta_parse(char *line) +{ + pmdelta_t *delta; + char *tmp = line, *tmp2; + + delta = malloc(sizeof(pmdelta_t)); + + tmp2 = tmp; + tmp = strchr(tmp, ' '); + *(tmp++) = '\0'; + delta->from = strdup(tmp2); + + tmp2 = tmp; + tmp = strchr(tmp, ' '); + *(tmp++) = '\0'; + delta->to = strdup(tmp2); + + tmp2 = tmp; + tmp = strchr(tmp, ' '); + *(tmp++) = '\0'; + delta->size = atol(tmp2); + + tmp2 = tmp; + tmp = strchr(tmp, ' '); + *(tmp++) = '\0'; + delta->filename = strdup(tmp2); + + delta->md5sum = strdup(tmp); + + return delta; +} + +/* vim: set ts=2 sw=2 noet: */ diff --git a/lib/libalpm/delta.h b/lib/libalpm/delta.h new file mode 100644 index 0000000..4655d7a --- /dev/null +++ b/lib/libalpm/delta.h @@ -0,0 +1,44 @@ +/* + * delta.h + * + * Copyright (c) 2002-2006 by Judd Vinet <jvinet@zeroflux.org> + * Copyright (c) 2005 by Aurelien Foret <orelien@chez.com> + * Copyright (c) 2006 by David Kimpe <dnaku@frugalware.org> + * Copyright (c) 2005, 2006 by Christian Hamar <krics@linuxforum.hu> + * Copyright (c) 2005, 2006 by Miklos Vajna <vmiklos@frugalware.org> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, + * USA. + */ +#ifndef _ALPM_DELTA_H +#define _ALPM_DELTA_H + +#include "alpm.h" + +struct __pmdelta_t { + char *from; + char *to; + unsigned long size; + char *filename; + char *md5sum; +}; + +unsigned long _alpm_delta_path_size(alpm_list_t *deltas); +unsigned long _alpm_delta_path_size_uncached(alpm_list_t *deltas); +pmdelta_t *_alpm_delta_parse(char *line); + +#endif /* _ALPM_DELTA_H */ + +/* vim: set ts=2 sw=2 noet: */ diff --git a/lib/libalpm/package.c b/lib/libalpm/package.c index 38e6e4c..6a8e0d8 100644 --- a/lib/libalpm/package.c +++ b/lib/libalpm/package.c @@ -446,6 +446,20 @@ alpm_list_t SYMEXPORT *alpm_pkg_get_provides(pmpkg_t *pkg) return pkg->provides; }
+alpm_list_t SYMEXPORT *alpm_pkg_get_deltas(pmpkg_t *pkg) +{ + ALPM_LOG_FUNC; + + /* Sanity checks */ + ASSERT(handle != NULL, return(NULL)); + ASSERT(pkg != NULL, return(NULL)); + + if(pkg->origin == PKG_FROM_CACHE && !(pkg->infolevel & INFRQ_DELTAS)) { + _alpm_db_read(pkg->origin_data.db, pkg, INFRQ_DELTAS); + } + return pkg->deltas; +} + alpm_list_t SYMEXPORT *alpm_pkg_get_replaces(pmpkg_t *pkg) { ALPM_LOG_FUNC; @@ -725,6 +739,7 @@ void _alpm_pkg_free(pmpkg_t *pkg) FREELIST(pkg->groups); FREELIST(pkg->provides); FREELIST(pkg->replaces); + FREELIST(pkg->deltas); if(pkg->origin == PKG_FROM_FILE) { FREE(pkg->origin_data.file); } diff --git a/lib/libalpm/package.h b/lib/libalpm/package.h index 42ebe0e..daaeb36 100644 --- a/lib/libalpm/package.h +++ b/lib/libalpm/package.h @@ -75,6 +75,7 @@ struct __pmpkg_t { alpm_list_t *requiredby; alpm_list_t *conflicts; alpm_list_t *provides; + alpm_list_t *deltas; /* internal */ pmpkgfrom_t origin; /* Replaced 'void *data' with this union as follows: -- 1.5.3.4
Wow, I'm impressed here. Great job. -Dan
On Mon, Oct 15, 2007 at 05:51:10PM -0400, Nathan Jones wrote:
+/** Calculates the shortest path from one version to another. + * + * The shortest path is defined as the path with the smallest combined + * size, not the length of the path. + * + * The algorithm used is Dijkstra's shortest path algorithm. + * + * @param deltas the list of pmdelta_t * objects that a package has + * @param from the version to start from + * @param to the version to end at + * @param path the current path + * + * @return the list of pmdelta_t * objects that has the smallest size. + * NULL (the empty list) is returned if there is no path between the + * versions. + */ +static alpm_list_t *shortest_delta_path(alpm_list_t *deltas, + const char *from, const char *to, alpm_list_t *path) +{ + alpm_list_t *d; + alpm_list_t *shortest = NULL; + + /* Found the 'to' version, this is a good path so return it. */ + if(strcmp(from, to) == 0) + return path; + + for(d = deltas; d; d = alpm_list_next(d)) { + pmdelta_t *v = alpm_list_getdata(d); + + /* If this vertex has already been visited in the path, go to the + * next vertex. */ + if(alpm_list_find(path, v)) + continue; + + /* Once we find a vertex that starts at the 'from' version, + * recursively find the shortest path using the 'to' version of this + * current vertex as the 'from' version in the function call. */ + if(strcmp(v->from, from) == 0) { + alpm_list_t *newpath = alpm_list_copy(path); + newpath = alpm_list_add(newpath, v); + + if((newpath = shortest_delta_path(deltas, v->to, to, newpath))) { + /* The path returned works, now use it unless there is already a + * shorter path found. */ + if(shortest == NULL) { + shortest = newpath; + } else if(_alpm_delta_path_size(shortest) > _alpm_delta_path_size(newpath)) { + alpm_list_free(shortest); + shortest = newpath; + } else { + alpm_list_free(newpath); + } + } + } + } + + return shortest; +} +
Is this really the Dijkstra algorithm? Looks like it does repetitive computation to me. Not that it really matters, given the very low values we work on. The only thing that bothers me slightly is the reference to it, in case I'm not mistaken. In any cases, I found the code clear and well written, so good job :)
On Thu, Oct 18, 2007 at 01:03:53AM +0200, Xavier wrote:
Is this really the Dijkstra algorithm? Looks like it does repetitive computation to me. Not that it really matters, given the very low values we work on. The only thing that bothers me slightly is the reference to it, in case I'm not mistaken.
I used the psuedocode from the Wikipedia article on Dijkstra's algorithm while writing this function. But you are right, I don't use the algorithm exactly as given. 'This algorithm is based on Dijkstra's shortest path algorithm' would be more accurate.
Signed-off-by: Nathan Jones <nathanj@insightbb.com> --- lib/libalpm/alpm.h | 1 + lib/libalpm/handle.c | 6 +++ lib/libalpm/handle.h | 1 + lib/libalpm/sync.c | 115 +++++++++++++++++++++++++++++++++++++++++++++++++- src/pacman/pacman.c | 3 + 5 files changed, 124 insertions(+), 2 deletions(-) diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 527822d..85c0905 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -138,6 +138,7 @@ void alpm_option_set_xfercommand(const char *cmd); unsigned short alpm_option_get_nopassiveftp(); void alpm_option_set_nopassiveftp(unsigned short nopasv); +void alpm_option_set_usedelta(unsigned short usedelta); pmdb_t *alpm_option_get_localdb(); alpm_list_t *alpm_option_get_syncdbs(); diff --git a/lib/libalpm/handle.c b/lib/libalpm/handle.c index e8f2147..242bbe5 100644 --- a/lib/libalpm/handle.c +++ b/lib/libalpm/handle.c @@ -64,6 +64,7 @@ pmhandle_t *_alpm_handle_new() handle->cachedirs = NULL; handle->lockfile = NULL; handle->logfile = NULL; + handle->usedelta = 0; return(handle); } @@ -496,4 +497,9 @@ void SYMEXPORT alpm_option_set_nopassiveftp(unsigned short nopasv) handle->nopassiveftp = nopasv; } +void SYMEXPORT alpm_option_set_usedelta(unsigned short usedelta) +{ + handle->usedelta = usedelta; +} + /* vim: set ts=2 sw=2 noet: */ diff --git a/lib/libalpm/handle.h b/lib/libalpm/handle.h index cf2e9d5..d8edf00 100644 --- a/lib/libalpm/handle.h +++ b/lib/libalpm/handle.h @@ -61,6 +61,7 @@ typedef struct _pmhandle_t { unsigned short nopassiveftp; /* Don't use PASV ftp connections */ time_t upgradedelay; /* Time to wait before upgrading a package */ char *xfercommand; /* External download command */ + unsigned short usedelta; /* Download deltas if possible */ } pmhandle_t; extern pmhandle_t *handle; diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index d24a1e5..8aa197d 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -48,6 +48,7 @@ #include "handle.h" #include "alpm.h" #include "server.h" +#include "delta.h" pmsyncpkg_t *_alpm_sync_new(int type, pmpkg_t *spkg, void *data) { @@ -700,9 +701,54 @@ cleanup: return(ret); } +/** Returns a list of deltas that should be downloaded instead of the + * package. + * + * Then it tests if a delta path exists between the currently installed + * version (if any) and the version to upgrade to. If so, the delta path + * is used if its size is below a set percentage of the package size, + * Otherwise, an empty list is returned. + * + * @param newpkg the new package to upgrade to + * @param db_local the local database + * + * @return the list of pmdelta_t * objects. NULL (the empty list) is + * returned if the package should be downloaded instead of deltas. + */ +static alpm_list_t *pkg_upgrade_delta_path(pmpkg_t *newpkg, pmdb_t *db_local) +{ + pmpkg_t *oldpkg = alpm_db_get_pkg(db_local, newpkg->name); + + if(oldpkg) { + const char *oldname = alpm_pkg_get_filename(oldpkg); + char *oldpath = _alpm_filecache_find(oldname); + + if(oldpath) { + alpm_list_t *deltas = alpm_shortest_delta_path( + alpm_pkg_get_deltas(newpkg), + alpm_pkg_get_version(oldpkg), + alpm_pkg_get_version(newpkg)); + + if(deltas) { + unsigned long dltsize = _alpm_delta_path_size(deltas); + unsigned long pkgsize = alpm_pkg_get_size(newpkg); + + if(dltsize < pkgsize * 0.7) { + return deltas; + } else { + alpm_list_free(deltas); + } + } + } + } + + return NULL; +} + int _alpm_sync_commit(pmtrans_t *trans, pmdb_t *db_local, alpm_list_t **data) { alpm_list_t *i, *j, *files = NULL; + alpm_list_t *patches = NULL; pmtrans_t *tr = NULL; int replaces = 0, retval = 0; const char *cachedir = NULL; @@ -733,8 +779,39 @@ int _alpm_sync_commit(pmtrans_t *trans, pmdb_t *db_local, alpm_list_t **data) } else { char *fpath = _alpm_filecache_find(fname); if(!fpath) { - /* file is not in the cache dir, so add it to the list */ - files = alpm_list_add(files, strdup(fname)); + if(handle->usedelta) { + alpm_list_t *delta_path = pkg_upgrade_delta_path(spkg, db_local); + + if(delta_path) { + alpm_list_t *dlts = NULL; + + for(dlts = delta_path; dlts; dlts = alpm_list_next(dlts)) { + pmdelta_t *d = (pmdelta_t *)alpm_list_getdata(dlts); + char *fpath2 = _alpm_filecache_find(d->filename); + + if(!fpath2) { + /* add the delta filename to the download list if + * it's not in the cache*/ + files = alpm_list_add(files, strdup(d->filename)); + } + + /* save the pkg and delta so that the xdelta patch + * command can be run after the downloads finish */ + patches = alpm_list_add(patches, spkg); + patches = alpm_list_add(patches, d); + } + + alpm_list_free(delta_path); + delta_path = NULL; + } else { + /* no deltas to download, so add the file to the + * download list */ + files = alpm_list_add(files, strdup(fname)); + } + } else { + /* not using deltas, so add the file to the download list */ + files = alpm_list_add(files, strdup(fname)); + } } FREE(fpath); } @@ -750,6 +827,40 @@ int _alpm_sync_commit(pmtrans_t *trans, pmdb_t *db_local, alpm_list_t **data) } FREELIST(files); } + + if(handle->usedelta) { + /* now apply any deltas that have been downloaded */ + alpm_list_t *p = patches; + while(p) { + pmpkg_t *pkg; + pmdelta_t *d; + char command[512]; + + pkg = alpm_list_getdata(p); + p = alpm_list_next(p); + + d = alpm_list_getdata(p); + p = alpm_list_next(p); + + snprintf(command, 512, + "xdelta patch %s/%s %s/%s-%s-%s.pkg.tar.gz %s/%s-%s-%s.pkg.tar.gz", + cachedir, d->filename, + cachedir, pkg->name, d->from, pkg->arch, + cachedir, pkg->name, d->to, pkg->arch); + _alpm_log(PM_LOG_DEBUG, _("command: %s\n"), command); + + printf("Generating %s-%s-%s.pkg.tar.gz with %s... ", + pkg->name, d->to, pkg->arch, d->filename); + if(system(command) == 0) { + printf("done.\n"); + } else { + printf("failed.\n"); + } + } + + alpm_list_free(patches); + patches = NULL; + } } if(trans->flags & PM_TRANS_FLAG_PRINTURIS) { return(0); diff --git a/src/pacman/pacman.c b/src/pacman/pacman.c index 8474020..2f6f928 100644 --- a/src/pacman/pacman.c +++ b/src/pacman/pacman.c @@ -548,6 +548,9 @@ static int _parseconfig(const char *file, const char *givensection, } else if(strcmp(key, "ShowSize") == 0 || strcmp(upperkey, "SHOWSIZE") == 0) { config->showsize = 1; pm_printf(PM_LOG_DEBUG, "config: showsize\n"); + } else if(strcmp(key, "UseDelta") == 0 || strcmp(upperkey, "USEDELTA") == 0) { + alpm_option_set_usedelta(1); + pm_printf(PM_LOG_DEBUG, "config: usedelta\n"); } else { pm_printf(PM_LOG_ERROR, _("config file %s, line %d: directive '%s' not recognized.\n"), file, linenum, key); -- 1.5.3.4
On 10/15/07, Nathan Jones <nathanj@insightbb.com> wrote:
Signed-off-by: Nathan Jones <nathanj@insightbb.com> --- lib/libalpm/alpm.h | 1 + lib/libalpm/handle.c | 6 +++ lib/libalpm/handle.h | 1 + lib/libalpm/sync.c | 115 +++++++++++++++++++++++++++++++++++++++++++++++++- src/pacman/pacman.c | 3 + 5 files changed, 124 insertions(+), 2 deletions(-)
diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 527822d..85c0905 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -138,6 +138,7 @@ void alpm_option_set_xfercommand(const char *cmd);
unsigned short alpm_option_get_nopassiveftp(); void alpm_option_set_nopassiveftp(unsigned short nopasv); +void alpm_option_set_usedelta(unsigned short usedelta);
pmdb_t *alpm_option_get_localdb(); alpm_list_t *alpm_option_get_syncdbs(); diff --git a/lib/libalpm/handle.c b/lib/libalpm/handle.c index e8f2147..242bbe5 100644 --- a/lib/libalpm/handle.c +++ b/lib/libalpm/handle.c @@ -64,6 +64,7 @@ pmhandle_t *_alpm_handle_new() handle->cachedirs = NULL; handle->lockfile = NULL; handle->logfile = NULL; + handle->usedelta = 0;
return(handle); } @@ -496,4 +497,9 @@ void SYMEXPORT alpm_option_set_nopassiveftp(unsigned short nopasv) handle->nopassiveftp = nopasv; }
+void SYMEXPORT alpm_option_set_usedelta(unsigned short usedelta) +{ + handle->usedelta = usedelta; +} + /* vim: set ts=2 sw=2 noet: */ diff --git a/lib/libalpm/handle.h b/lib/libalpm/handle.h index cf2e9d5..d8edf00 100644 --- a/lib/libalpm/handle.h +++ b/lib/libalpm/handle.h @@ -61,6 +61,7 @@ typedef struct _pmhandle_t { unsigned short nopassiveftp; /* Don't use PASV ftp connections */ time_t upgradedelay; /* Time to wait before upgrading a package */ char *xfercommand; /* External download command */ + unsigned short usedelta; /* Download deltas if possible */ } pmhandle_t;
extern pmhandle_t *handle; diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index d24a1e5..8aa197d 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -48,6 +48,7 @@ #include "handle.h" #include "alpm.h" #include "server.h" +#include "delta.h"
pmsyncpkg_t *_alpm_sync_new(int type, pmpkg_t *spkg, void *data) { @@ -700,9 +701,54 @@ cleanup: return(ret); }
+/** Returns a list of deltas that should be downloaded instead of the + * package. + * + * Then it tests if a delta path exists between the currently installed + * version (if any) and the version to upgrade to. If so, the delta path + * is used if its size is below a set percentage of the package size, + * Otherwise, an empty list is returned. + * + * @param newpkg the new package to upgrade to + * @param db_local the local database + * + * @return the list of pmdelta_t * objects. NULL (the empty list) is + * returned if the package should be downloaded instead of deltas. + */ +static alpm_list_t *pkg_upgrade_delta_path(pmpkg_t *newpkg, pmdb_t *db_local) +{ + pmpkg_t *oldpkg = alpm_db_get_pkg(db_local, newpkg->name); + + if(oldpkg) { + const char *oldname = alpm_pkg_get_filename(oldpkg); + char *oldpath = _alpm_filecache_find(oldname); + + if(oldpath) { + alpm_list_t *deltas = alpm_shortest_delta_path( + alpm_pkg_get_deltas(newpkg), + alpm_pkg_get_version(oldpkg), + alpm_pkg_get_version(newpkg)); + + if(deltas) { + unsigned long dltsize = _alpm_delta_path_size(deltas); + unsigned long pkgsize = alpm_pkg_get_size(newpkg); + + if(dltsize < pkgsize * 0.7) {
I'd feel a bit better if this was #defined in util.h. (Something like MAX_DELTA_RATIO?)
+ return deltas; + } else { + alpm_list_free(deltas); + } + } + } + } + + return NULL; +} + int _alpm_sync_commit(pmtrans_t *trans, pmdb_t *db_local, alpm_list_t **data) { alpm_list_t *i, *j, *files = NULL; + alpm_list_t *patches = NULL; pmtrans_t *tr = NULL; int replaces = 0, retval = 0; const char *cachedir = NULL; @@ -733,8 +779,39 @@ int _alpm_sync_commit(pmtrans_t *trans, pmdb_t *db_local, alpm_list_t **data) } else { char *fpath = _alpm_filecache_find(fname); if(!fpath) { - /* file is not in the cache dir, so add it to the list */ - files = alpm_list_add(files, strdup(fname)); + if(handle->usedelta) { + alpm_list_t *delta_path = pkg_upgrade_delta_path(spkg, db_local); + + if(delta_path) { + alpm_list_t *dlts = NULL; + + for(dlts = delta_path; dlts; dlts = alpm_list_next(dlts)) { + pmdelta_t *d = (pmdelta_t *)alpm_list_getdata(dlts); + char *fpath2 = _alpm_filecache_find(d->filename); + + if(!fpath2) { + /* add the delta filename to the download list if + * it's not in the cache*/ + files = alpm_list_add(files, strdup(d->filename)); + } + + /* save the pkg and delta so that the xdelta patch + * command can be run after the downloads finish */ + patches = alpm_list_add(patches, spkg); + patches = alpm_list_add(patches, d); + } + + alpm_list_free(delta_path); + delta_path = NULL; + } else { + /* no deltas to download, so add the file to the + * download list */ + files = alpm_list_add(files, strdup(fname)); + } + } else { + /* not using deltas, so add the file to the download list */ + files = alpm_list_add(files, strdup(fname)); + } } FREE(fpath); } @@ -750,6 +827,40 @@ int _alpm_sync_commit(pmtrans_t *trans, pmdb_t *db_local, alpm_list_t **data) } FREELIST(files); } + + if(handle->usedelta) { + /* now apply any deltas that have been downloaded */ + alpm_list_t *p = patches; + while(p) { + pmpkg_t *pkg; + pmdelta_t *d; + char command[512]; + + pkg = alpm_list_getdata(p); + p = alpm_list_next(p); + + d = alpm_list_getdata(p); + p = alpm_list_next(p); + + snprintf(command, 512,
Using PATH_MAX instead of a hardcoded value would probably be best. Its just a local 1-time alloc so memory usage is not much of a concern. I might also put a comment here showing an example string we are buildling.
+ "xdelta patch %s/%s %s/%s-%s-%s.pkg.tar.gz %s/%s-%s-%s.pkg.tar.gz", + cachedir, d->filename, + cachedir, pkg->name, d->from, pkg->arch, + cachedir, pkg->name, d->to, pkg->arch); + _alpm_log(PM_LOG_DEBUG, _("command: %s\n"), command); + + printf("Generating %s-%s-%s.pkg.tar.gz with %s... ", + pkg->name, d->to, pkg->arch, d->filename); + if(system(command) == 0) {
I don't have much experience with this, so I'd like to have someone else chime in. Is system() the best call to be making, or should we do it another way (fork/exec like the scriptlets comes to mind).
+ printf("done.\n"); + } else { + printf("failed.\n"); + } + } + + alpm_list_free(patches); + patches = NULL; + } } if(trans->flags & PM_TRANS_FLAG_PRINTURIS) { return(0); diff --git a/src/pacman/pacman.c b/src/pacman/pacman.c index 8474020..2f6f928 100644 --- a/src/pacman/pacman.c +++ b/src/pacman/pacman.c @@ -548,6 +548,9 @@ static int _parseconfig(const char *file, const char *givensection, } else if(strcmp(key, "ShowSize") == 0 || strcmp(upperkey, "SHOWSIZE") == 0) { config->showsize = 1; pm_printf(PM_LOG_DEBUG, "config: showsize\n"); + } else if(strcmp(key, "UseDelta") == 0 || strcmp(upperkey, "USEDELTA") == 0) { + alpm_option_set_usedelta(1); + pm_printf(PM_LOG_DEBUG, "config: usedelta\n"); } else { pm_printf(PM_LOG_ERROR, _("config file %s, line %d: directive '%s' not recognized.\n"), file, linenum, key); -- 1.5.3.4
_______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://archlinux.org/mailman/listinfo/pacman-dev
On Mon, Oct 15, 2007 at 09:50:57PM -0500, Dan McGee wrote:
+ "xdelta patch %s/%s %s/%s-%s-%s.pkg.tar.gz %s/%s-%s-%s.pkg.tar.gz", + cachedir, d->filename, + cachedir, pkg->name, d->from, pkg->arch, + cachedir, pkg->name, d->to, pkg->arch); + _alpm_log(PM_LOG_DEBUG, _("command: %s\n"), command); + + printf("Generating %s-%s-%s.pkg.tar.gz with %s... ", + pkg->name, d->to, pkg->arch, d->filename); + if(system(command) == 0) {
I don't have much experience with this, so I'd like to have someone else chime in. Is system() the best call to be making, or should we do it another way (fork/exec like the scriptlets comes to mind).
Isn't the cleanest way to use xdelta api? For example, we use libarchive instead of bsdtar, libdownload instead of wget (even though there is the XferCommand trick), a md5 driver instead of md5sum command, etc However, it looks quite common in *nix to build little command line tools, and then a graphical gui on top of them, but I don't know how it's usually done (system or fork/exec or what). So I also hope someone can help :)
On 10/17/07, Xavier <shiningxc@gmail.com> wrote:
On Mon, Oct 15, 2007 at 09:50:57PM -0500, Dan McGee wrote:
+ "xdelta patch %s/%s %s/%s-%s-%s.pkg.tar.gz %s/%s-%s-%s.pkg.tar.gz", + cachedir, d->filename, + cachedir, pkg->name, d->from, pkg->arch, + cachedir, pkg->name, d->to, pkg->arch); + _alpm_log(PM_LOG_DEBUG, _("command: %s\n"), command); + + printf("Generating %s-%s-%s.pkg.tar.gz with %s... ", + pkg->name, d->to, pkg->arch, d->filename); + if(system(command) == 0) {
I don't have much experience with this, so I'd like to have someone else chime in. Is system() the best call to be making, or should we do it another way (fork/exec like the scriptlets comes to mind).
Isn't the cleanest way to use xdelta api? For example, we use libarchive instead of bsdtar, libdownload instead of wget (even though there is the XferCommand trick), a md5 driver instead of md5sum command, etc
Technically yes, but I think in this case this is the only option available - either xdelta has no API, or using it is like a few hundred lines... I can't remember which
2007/10/18, Aaron Griffin <aaronmgriffin@gmail.com>:
On 10/17/07, Xavier <shiningxc@gmail.com> wrote:
Isn't the cleanest way to use xdelta api? For example, we use libarchive instead of bsdtar, libdownload instead of wget (even though there is the XferCommand trick), a md5 driver instead of md5sum command, etc
Technically yes, but I think in this case this is the only option available - either xdelta has no API, or using it is like a few hundred lines... I can't remember which
Xdelta3 does have an API: http://code.google.com/p/xdelta/wiki/ProgrammingGuide -- Roman Kyrylych (Роман Кирилич)
On 10/18/07, Roman Kyrylych <roman.kyrylych@gmail.com> wrote:
2007/10/18, Aaron Griffin <aaronmgriffin@gmail.com>:
On 10/17/07, Xavier <shiningxc@gmail.com> wrote:
Isn't the cleanest way to use xdelta api? For example, we use libarchive instead of bsdtar, libdownload instead of wget (even though there is the XferCommand trick), a md5 driver instead of md5sum command, etc
Technically yes, but I think in this case this is the only option available - either xdelta has no API, or using it is like a few hundred lines... I can't remember which
Xdelta3 does have an API: http://code.google.com/p/xdelta/wiki/ProgrammingGuide
Well, color me stupid! I didn't research this at all. So yeah, I'll side with Chantry here - it's probably better to use the actual API rather than external apps like that. Here's a good reason - in a "omg panic" situation where libs are broken, pacman.static can still work, but not if it has to call external apps like this that could be broken.
On 10/18/07, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On 10/18/07, Roman Kyrylych <roman.kyrylych@gmail.com> wrote:
2007/10/18, Aaron Griffin <aaronmgriffin@gmail.com>:
On 10/17/07, Xavier <shiningxc@gmail.com> wrote:
Isn't the cleanest way to use xdelta api? For example, we use libarchive instead of bsdtar, libdownload instead of wget (even though there is the XferCommand trick), a md5 driver instead of md5sum command, etc
Technically yes, but I think in this case this is the only option available - either xdelta has no API, or using it is like a few hundred lines... I can't remember which
Xdelta3 does have an API: http://code.google.com/p/xdelta/wiki/ProgrammingGuide
Well, color me stupid! I didn't research this at all.
So yeah, I'll side with Chantry here - it's probably better to use the actual API rather than external apps like that. Here's a good reason - in a "omg panic" situation where libs are broken, pacman.static can still work, but not if it has to call external apps like this that could be broken.
Thoughts but no for sure answers, I'll let those who are actually writing the code decide. 1. Using the API does introduce yet another dependency for pacman...our ldd output is starting to get a bit lengthy for what I feel should be a low level tool. If someone doesn't ever use the delta features of pacman, they would still have to have xdelta installed (because of the dynamic link at runtime, and yet some more mem usage). 2. xdelta3 tarball was chmodded all wrong. Please tell me the developer just made a mistake here, otherwise it is very windows focused. And the configure script is no longer there... 3. In an "omg panic" situation, I highly doubt someone would want to use delta packages anyway, I'd rather be absolutely positive I was getting the real stuff. -Dan
This displays the download size, taking into account delta files and cached files. This closes FS#4182. Signed-off-by: Nathan Jones <nathanj@insightbb.com> --- lib/libalpm/alpm.h | 2 ++ lib/libalpm/sync.c | 34 ++++++++++++++++++++++++++++++++++ src/pacman/query.c | 2 +- src/pacman/sync.c | 2 +- src/pacman/util.c | 22 +++++++--------------- src/pacman/util.h | 2 +- 6 files changed, 46 insertions(+), 18 deletions(-) diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 85c0905..fd18a01 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -215,6 +215,8 @@ alpm_list_t *alpm_pkg_get_files(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_backup(pmpkg_t *pkg); unsigned short alpm_pkg_has_scriptlet(pmpkg_t *pkg); +unsigned long alpm_pkg_download_size(pmpkg_t *newpkg, pmdb_t *db_local); + /* * Deltas */ diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index 8aa197d..c108f81 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -745,6 +745,40 @@ static alpm_list_t *pkg_upgrade_delta_path(pmpkg_t *newpkg, pmdb_t *db_local) return NULL; } +/** Returns the size of the files that will be downloaded to install a + * package. + * + * @param newpkg the new package to upgrade to + * @param db_local the local database + * + * @return the size of the download + */ +unsigned long SYMEXPORT alpm_pkg_download_size(pmpkg_t *newpkg, pmdb_t *db_local) +{ + char *fpath = _alpm_filecache_find(alpm_pkg_get_filename(newpkg)); + unsigned long size = 0; + + if(fpath) { + size = 0; + } else if(handle->usedelta) { + alpm_list_t *deltas = pkg_upgrade_delta_path(newpkg, db_local); + + if(deltas) { + size = _alpm_delta_path_size_uncached(deltas); + } else { + size = alpm_pkg_get_size(newpkg); + } + + alpm_list_free(deltas); + } else { + size = alpm_pkg_get_size(newpkg); + } + + FREE(fpath); + + return(size); +} + int _alpm_sync_commit(pmtrans_t *trans, pmdb_t *db_local, alpm_list_t **data) { alpm_list_t *i, *j, *files = NULL; diff --git a/src/pacman/query.c b/src/pacman/query.c index 4e4002c..8a01d82 100644 --- a/src/pacman/query.c +++ b/src/pacman/query.c @@ -219,7 +219,7 @@ static int query_upgrades(void) printf(_("Checking for package upgrades... \n")); if((syncpkgs = alpm_db_get_upgrades()) != NULL) { - display_targets(syncpkgs); + display_targets(syncpkgs, db_local); return(0); } diff --git a/src/pacman/sync.c b/src/pacman/sync.c index 6a7d50e..bf6eed1 100644 --- a/src/pacman/sync.c +++ b/src/pacman/sync.c @@ -601,7 +601,7 @@ int sync_trans(alpm_list_t *targets, int sync_only) if(!(alpm_trans_get_flags() & PM_TRANS_FLAG_PRINTURIS)) { int confirm; - display_targets(packages); + display_targets(packages, db_local); printf("\n"); if(config->op_s_downloadonly) { diff --git a/src/pacman/util.c b/src/pacman/util.c index 2c00753..8b6079d 100644 --- a/src/pacman/util.c +++ b/src/pacman/util.c @@ -329,14 +329,15 @@ void list_display(const char *title, const alpm_list_t *list) * retrieved from a transaction object */ /* TODO move to output.c? or just combine util and output */ -void display_targets(const alpm_list_t *syncpkgs) +void display_targets(const alpm_list_t *syncpkgs, pmdb_t *db_local) { char *str; const alpm_list_t *i, *j; alpm_list_t *targets = NULL, *to_remove = NULL; /* TODO these are some messy variable names */ - unsigned long size = 0, isize = 0, rsize = 0, dispsize = 0; - double mbsize = 0.0, mbisize = 0.0, mbrsize = 0.0, mbdispsize = 0.0; + unsigned long size = 0, isize = 0, rsize = 0, dispsize = 0, dlsize = 0; + double mbsize = 0.0, mbisize = 0.0, mbrsize = 0.0, mbdispsize = 0.0, + mbdlsize = 0.0; for(i = syncpkgs; i; i = alpm_list_next(i)) { pmsyncpkg_t *sync = alpm_list_getdata(i); @@ -361,6 +362,7 @@ void display_targets(const alpm_list_t *syncpkgs) dispsize = alpm_pkg_get_size(pkg); size += dispsize; + dlsize += alpm_pkg_download_size(pkg, db_local); isize += alpm_pkg_get_isize(pkg); /* print the package size with the output if ShowSize option set */ @@ -381,6 +383,7 @@ void display_targets(const alpm_list_t *syncpkgs) mbsize = size / (1024.0 * 1024.0); mbisize = isize / (1024.0 * 1024.0); mbrsize = rsize / (1024.0 * 1024.0); + mbdlsize = dlsize / (1024.0 * 1024.0); /* start displaying information */ printf("\n"); @@ -390,28 +393,17 @@ void display_targets(const alpm_list_t *syncpkgs) printf("\n"); FREELIST(to_remove); - /* round up if size is really small */ - if(mbrsize < 0.1) { - mbrsize = 0.1; - } printf(_("Total Removed Size: %.2f MB\n"), mbrsize); } list_display(_("Targets:"), targets); printf("\n"); - /* round up if size is really small */ - if(mbsize < 0.1) { - mbsize = 0.1; - } printf(_("Total Package Size: %.2f MB\n"), mbsize); + printf(_("Total Download Size %.2f MB\n"), mbdlsize); /* TODO because all pkgs don't include isize, this is a crude hack */ if(mbisize > mbsize) { - /*round up if size is really small */ - if(mbisize < 0.1) { - mbisize = 0.1; - } printf(_("Total Installed Size: %.2f MB\n"), mbisize); } diff --git a/src/pacman/util.h b/src/pacman/util.h index f3171ca..876f817 100644 --- a/src/pacman/util.h +++ b/src/pacman/util.h @@ -47,7 +47,7 @@ char *strtoupper(char *str); char *strtrim(char *str); char *strreplace(const char *str, const char *needle, const char *replace); void list_display(const char *title, const alpm_list_t *list); -void display_targets(const alpm_list_t *syncpkgs); +void display_targets(const alpm_list_t *syncpkgs, pmdb_t *db_local); int yesno(char *fmt, ...); int pm_printf(pmloglevel_t level, const char *format, ...) __attribute__((format(printf,2,3))); int pm_fprintf(FILE *stream, pmloglevel_t level, const char *format, ...) __attribute__((format(printf,3,4))); -- 1.5.3.4
On 10/15/07, Nathan Jones <nathanj@insightbb.com> wrote:
This displays the download size, taking into account delta files and cached files.
This closes FS#4182.
Signed-off-by: Nathan Jones <nathanj@insightbb.com> --- lib/libalpm/alpm.h | 2 ++ lib/libalpm/sync.c | 34 ++++++++++++++++++++++++++++++++++ src/pacman/query.c | 2 +- src/pacman/sync.c | 2 +- src/pacman/util.c | 22 +++++++--------------- src/pacman/util.h | 2 +- 6 files changed, 46 insertions(+), 18 deletions(-)
diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 85c0905..fd18a01 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -215,6 +215,8 @@ alpm_list_t *alpm_pkg_get_files(pmpkg_t *pkg); alpm_list_t *alpm_pkg_get_backup(pmpkg_t *pkg); unsigned short alpm_pkg_has_scriptlet(pmpkg_t *pkg);
+unsigned long alpm_pkg_download_size(pmpkg_t *newpkg, pmdb_t *db_local); + /* * Deltas */ diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index 8aa197d..c108f81 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -745,6 +745,40 @@ static alpm_list_t *pkg_upgrade_delta_path(pmpkg_t *newpkg, pmdb_t *db_local) return NULL; }
+/** Returns the size of the files that will be downloaded to install a + * package. + * + * @param newpkg the new package to upgrade to + * @param db_local the local database + * + * @return the size of the download + */ +unsigned long SYMEXPORT alpm_pkg_download_size(pmpkg_t *newpkg, pmdb_t *db_local) +{ + char *fpath = _alpm_filecache_find(alpm_pkg_get_filename(newpkg)); + unsigned long size = 0;
We probably should be using a size_t/ssize_t or whatever we are supposed to here, but that would be a whole different patch as we already use unsigned long elsewhere.
+ + if(fpath) { + size = 0; + } else if(handle->usedelta) { + alpm_list_t *deltas = pkg_upgrade_delta_path(newpkg, db_local); + + if(deltas) { + size = _alpm_delta_path_size_uncached(deltas);
Does this account for the delta is greater than 0.7 of the size case?
+ } else { + size = alpm_pkg_get_size(newpkg); + } + + alpm_list_free(deltas); + } else { + size = alpm_pkg_get_size(newpkg); + } + + FREE(fpath); + + return(size); +} + int _alpm_sync_commit(pmtrans_t *trans, pmdb_t *db_local, alpm_list_t **data) { alpm_list_t *i, *j, *files = NULL; diff --git a/src/pacman/query.c b/src/pacman/query.c index 4e4002c..8a01d82 100644 --- a/src/pacman/query.c +++ b/src/pacman/query.c @@ -219,7 +219,7 @@ static int query_upgrades(void) printf(_("Checking for package upgrades... \n"));
if((syncpkgs = alpm_db_get_upgrades()) != NULL) { - display_targets(syncpkgs); + display_targets(syncpkgs, db_local); return(0); }
diff --git a/src/pacman/sync.c b/src/pacman/sync.c index 6a7d50e..bf6eed1 100644 --- a/src/pacman/sync.c +++ b/src/pacman/sync.c @@ -601,7 +601,7 @@ int sync_trans(alpm_list_t *targets, int sync_only) if(!(alpm_trans_get_flags() & PM_TRANS_FLAG_PRINTURIS)) { int confirm;
- display_targets(packages); + display_targets(packages, db_local); printf("\n");
if(config->op_s_downloadonly) { diff --git a/src/pacman/util.c b/src/pacman/util.c index 2c00753..8b6079d 100644 --- a/src/pacman/util.c +++ b/src/pacman/util.c @@ -329,14 +329,15 @@ void list_display(const char *title, const alpm_list_t *list) * retrieved from a transaction object */ /* TODO move to output.c? or just combine util and output */ -void display_targets(const alpm_list_t *syncpkgs) +void display_targets(const alpm_list_t *syncpkgs, pmdb_t *db_local) { char *str; const alpm_list_t *i, *j; alpm_list_t *targets = NULL, *to_remove = NULL; /* TODO these are some messy variable names */ - unsigned long size = 0, isize = 0, rsize = 0, dispsize = 0; - double mbsize = 0.0, mbisize = 0.0, mbrsize = 0.0, mbdispsize = 0.0; + unsigned long size = 0, isize = 0, rsize = 0, dispsize = 0, dlsize = 0; + double mbsize = 0.0, mbisize = 0.0, mbrsize = 0.0, mbdispsize = 0.0, + mbdlsize = 0.0;
for(i = syncpkgs; i; i = alpm_list_next(i)) { pmsyncpkg_t *sync = alpm_list_getdata(i); @@ -361,6 +362,7 @@ void display_targets(const alpm_list_t *syncpkgs)
dispsize = alpm_pkg_get_size(pkg); size += dispsize; + dlsize += alpm_pkg_download_size(pkg, db_local); isize += alpm_pkg_get_isize(pkg);
/* print the package size with the output if ShowSize option set */ @@ -381,6 +383,7 @@ void display_targets(const alpm_list_t *syncpkgs) mbsize = size / (1024.0 * 1024.0); mbisize = isize / (1024.0 * 1024.0); mbrsize = rsize / (1024.0 * 1024.0); + mbdlsize = dlsize / (1024.0 * 1024.0);
/* start displaying information */ printf("\n"); @@ -390,28 +393,17 @@ void display_targets(const alpm_list_t *syncpkgs) printf("\n"); FREELIST(to_remove);
- /* round up if size is really small */ - if(mbrsize < 0.1) { - mbrsize = 0.1; - }
Any reason for removing this?
printf(_("Total Removed Size: %.2f MB\n"), mbrsize); }
list_display(_("Targets:"), targets); printf("\n");
- /* round up if size is really small */ - if(mbsize < 0.1) { - mbsize = 0.1; - }
Echo what I said above.
printf(_("Total Package Size: %.2f MB\n"), mbsize); + printf(_("Total Download Size %.2f MB\n"), mbdlsize);
/* TODO because all pkgs don't include isize, this is a crude hack */ if(mbisize > mbsize) { - /*round up if size is really small */ - if(mbisize < 0.1) { - mbisize = 0.1; - }
See above.
printf(_("Total Installed Size: %.2f MB\n"), mbisize); }
diff --git a/src/pacman/util.h b/src/pacman/util.h index f3171ca..876f817 100644 --- a/src/pacman/util.h +++ b/src/pacman/util.h @@ -47,7 +47,7 @@ char *strtoupper(char *str); char *strtrim(char *str); char *strreplace(const char *str, const char *needle, const char *replace); void list_display(const char *title, const alpm_list_t *list); -void display_targets(const alpm_list_t *syncpkgs); +void display_targets(const alpm_list_t *syncpkgs, pmdb_t *db_local); int yesno(char *fmt, ...); int pm_printf(pmloglevel_t level, const char *format, ...) __attribute__((format(printf,2,3))); int pm_fprintf(FILE *stream, pmloglevel_t level, const char *format, ...) __attribute__((format(printf,3,4))); -- 1.5.3.4
On Mon, Oct 15, 2007 at 09:59:07PM -0500, Dan McGee wrote:
+unsigned long SYMEXPORT alpm_pkg_download_size(pmpkg_t *newpkg, pmdb_t *db_local) +{ + char *fpath = _alpm_filecache_find(alpm_pkg_get_filename(newpkg)); + unsigned long size = 0; + + if(fpath) { + size = 0; + } else if(handle->usedelta) { + alpm_list_t *deltas = pkg_upgrade_delta_path(newpkg, db_local); + + if(deltas) { + size = _alpm_delta_path_size_uncached(deltas);
Does this account for the delta is greater than 0.7 of the size case?
When finding the shortest delta path, the code does not look to the cache to see if the file already exists. This function takes whatever delta files will actually be used and calculates the remaining downloaded size for the client to display, using 0 if a delta exists in the cache.
- /* round up if size is really small */ - if(mbrsize < 0.1) { - mbrsize = 0.1; - }
Any reason for removing this?
I removed it because it doesn't make as much sense with the download size. For example, downloading a 30K package would display 100K for the total package size but 30K for the download size. The download size could be rounded as well, but it is possible for it to be 0K if the file already exists in the cache, so displaying 100K would not be correct.
On 10/16/07, Nathan Jones <nathanj@insightbb.com> wrote:
On Mon, Oct 15, 2007 at 09:59:07PM -0500, Dan McGee wrote:
+unsigned long SYMEXPORT alpm_pkg_download_size(pmpkg_t *newpkg, pmdb_t *db_local) +{ + char *fpath = _alpm_filecache_find(alpm_pkg_get_filename(newpkg)); + unsigned long size = 0; + + if(fpath) { + size = 0; + } else if(handle->usedelta) { + alpm_list_t *deltas = pkg_upgrade_delta_path(newpkg, db_local); + + if(deltas) { + size = _alpm_delta_path_size_uncached(deltas);
Does this account for the delta is greater than 0.7 of the size case?
When finding the shortest delta path, the code does not look to the cache to see if the file already exists. This function takes whatever delta files will actually be used and calculates the remaining downloaded size for the client to display, using 0 if a delta exists in the cache.
- /* round up if size is really small */ - if(mbrsize < 0.1) { - mbrsize = 0.1; - }
Any reason for removing this?
I removed it because it doesn't make as much sense with the download size. For example, downloading a 30K package would display 100K for the total package size but 30K for the download size.
The download size could be rounded as well, but it is possible for it to be 0K if the file already exists in the cache, so displaying 100K would not be correct.
I was thinking the installed size/package size always displayed as MB. I know that when downloading we use K until we reach a certain point, but I thought the initial display of sizes is done all in MB. -Dan
On 10/16/07, Dan McGee <dpmcgee@gmail.com> wrote:
On 10/16/07, Nathan Jones <nathanj@insightbb.com> wrote:
On Mon, Oct 15, 2007 at 09:59:07PM -0500, Dan McGee wrote:
+unsigned long SYMEXPORT alpm_pkg_download_size(pmpkg_t *newpkg, pmdb_t *db_local) +{ + char *fpath = _alpm_filecache_find(alpm_pkg_get_filename(newpkg)); + unsigned long size = 0; + + if(fpath) { + size = 0; + } else if(handle->usedelta) { + alpm_list_t *deltas = pkg_upgrade_delta_path(newpkg, db_local); + + if(deltas) { + size = _alpm_delta_path_size_uncached(deltas);
Does this account for the delta is greater than 0.7 of the size case?
When finding the shortest delta path, the code does not look to the cache to see if the file already exists. This function takes whatever delta files will actually be used and calculates the remaining downloaded size for the client to display, using 0 if a delta exists in the cache.
- /* round up if size is really small */ - if(mbrsize < 0.1) { - mbrsize = 0.1; - }
Any reason for removing this?
I removed it because it doesn't make as much sense with the download size. For example, downloading a 30K package would display 100K for the total package size but 30K for the download size.
The download size could be rounded as well, but it is possible for it to be 0K if the file already exists in the cache, so displaying 100K would not be correct.
I was thinking the installed size/package size always displayed as MB. I know that when downloading we use K until we reach a certain point, but I thought the initial display of sizes is done all in MB.
-Dan
Some example output: $ pacS 855resolution resolving dependencies... done. looking for inter-conflicts... done. Targets: 855resolution-0.4-5 Total Package Size: 0.10 MB Proceed with installation? [Y/n] Since we print the values to 2 decimal places, we should probably round to 0.01 instead of 0.1. Or like you said, we could just get rid of the rounding. I'll do this on my local branch, I like your original thinking. :) -Dan
On 10/15/07, Nathan Jones <nathanj@insightbb.com> wrote:
This patchset adds delta information to repository databases and libalpm, allowing libalpm to download multiple delta files when upgrading a package.
I still have some work to do, but I thought I would post this now to get any suggestions and criticisms that anyone has.
I don't have any suggestions or criticisms but I did want to chime in and say keep up the good work! It's great to see that you've kept at it. I'm excited for the day when this capability will exist in pacman. Scott
On 10/15/07, Scott Horowitz <stonecrest@gmail.com> wrote:
On 10/15/07, Nathan Jones <nathanj@insightbb.com> wrote:
This patchset adds delta information to repository databases and libalpm, allowing libalpm to download multiple delta files when upgrading a package.
I still have some work to do, but I thought I would post this now to get any suggestions and criticisms that anyone has.
I don't have any suggestions or criticisms but I did want to chime in and say keep up the good work! It's great to see that you've kept at it. I'm excited for the day when this capability will exist in pacman.
Yeah, absolutely - amazing work so far. I'm going to meet up with Dan again this week and we can review these patches, but at a quick glance from work, it all looks good 8) -- Aaron
participants (7)
-
Aaron Griffin
-
Dan McGee
-
Kevin Piche
-
Nathan Jones
-
Roman Kyrylych
-
Scott Horowitz
-
Xavier