[pacman-dev] [PATCH] Add deltas to repository database.
This patch adds delta information to the repository database along with the other information about a package. This would be useful if two or more package versions have been built since a user upgrades. I don't think it would be too difficult to create an algorithm to decide which delta files, if any, would be downloaded and applied. Any comments? Sample output: $ ls -l *.pkg.tar.gz *.delta -rw-r--r-- 1 nathanj users 1045940 2007-05-12 21:03 pan-devel-0.129-1-i686.pkg.tar.gz -rw-r--r-- 1 nathanj users 583339 2007-10-05 18:38 pan-devel-0.129-1_to_0.130-1-i686.delta -rw-r--r-- 1 nathanj users 1044271 2007-05-19 13:17 pan-devel-0.130-1-i686.pkg.tar.gz -rw-r--r-- 1 nathanj users 425727 2007-10-05 18:38 pan-devel-0.130-1_to_0.131-1-i686.delta -rw-r--r-- 1 nathanj users 1055063 2007-05-26 18:40 pan-devel-0.131-1-i686.pkg.tar.gz -rw-r--r-- 1 nathanj users 680644 2007-10-05 18:38 pan-devel-0.131-1_to_0.132-1-i686.delta -rw-r--r-- 1 nathanj users 1025859 2007-08-15 22:03 pan-devel-0.132-1-i686.pkg.tar.gz -rw-r--r-- 1 nathanj users 1792 2007-10-05 18:38 pan-devel-0.132-1_to_0.132-2-i686.delta -rw-r--r-- 1 nathanj users 1025826 2007-08-01 17:04 pan-devel-0.132-2-i686.pkg.tar.gz $ ~/code/pacman/scripts/repo-add repo.db.tar.gz pan-devel-0.132-2-i686.pkg.tar.gz *.delta ==> Extracting database to a temporary location... ==> Adding package 'pan-devel-0.132-2-i686.pkg.tar.gz' -> Removing existing package 'pan-devel-0.132-2'... -> Creating 'desc' db entry... -> Computing md5 checksums... -> Creating 'depends' db entry... ==> Adding delta 'pan-devel-0.129-1_to_0.130-1-i686.delta' ==> Adding delta 'pan-devel-0.130-1_to_0.131-1-i686.delta' ==> Adding delta 'pan-devel-0.131-1_to_0.132-1-i686.delta' ==> Adding delta 'pan-devel-0.132-1_to_0.132-2-i686.delta' ==> Creating updated database file /home/nathanj/pkgs/pan-devel/repo.db.tar.gz $ tar zxOf repo.db.tar.gz pan-devel-0.132-2/deltas %DELTAS% 0.129-1 0.130-1 583339 0.130-1 0.131-1 425727 0.131-1 0.132-1 680644 0.132-1 0.132-2 1792 Signed-off-by: Nathan Jones <nathanj@insightbb.com> --- scripts/repo-add.sh.in | 64 +++++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 63 insertions(+), 1 deletions(-) diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index cb741d7..4a7949e 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -97,6 +97,61 @@ write_list_entry() { fi } +# write a delta entry to the pacman database +# arg1 - path to delta +db_write_delta() +{ + # blank out all variables and set deltafile + local deltafile=$(readlink -f "$1") + local pkgname fromver tover arch csize + + pkgname="$(echo "$1" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\1/')" + fromver="$(echo "$1" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\2/')" + tover="$(echo "$1" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\3/')" + arch="$(echo "$1" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\4/')" + + # get size of delta + csize=$(du -b -L "$deltafile" | cut -f 1) + + pushd "$gstmpdir" 2>&1 >/dev/null + + # ensure $pkgname and $pkgver variables were found + if [ -z "$pkgname" -o -z "$fromver" ]; then + error "$(gettext "Invalid delta file '%s'.")" "$deltafile" + popd 2>&1 >/dev/null + return 1 + fi + + # cd into the package's directory + local existing + local gotpackage=f + for existing in *; do + if [ "${existing%-*-*}" = "$pkgname" ]; then + cd "$existing" + gotpackage=t + break + fi + done + + # exit out if the package's directory does not exist + if [ "$gotpackage" == "f" ]; then + error "$(gettext "Cannot find directory for '%s'.")" "$pkgname" + popd 2>&1 >/dev/null + return 1 + fi + + # create deltas file if it doesn't exist + if [ ! -f "deltas" ]; then + echo -e "%DELTAS%" >>deltas + fi + + # add the entry for this delta file + echo -e "$fromver $tover $csize" >>deltas + + popd 2>&1 >/dev/null +} # end db_write_delta + + # write an entry to the pacman database # arg1 - path to package db_write_entry() @@ -242,7 +297,14 @@ for arg in "$@"; do fi else if [ -f "$arg" ]; then - if ! bsdtar -tf "$arg" .PKGINFO 2>&1 >/dev/null; then + # see if extension is .delta ... there's gotta be a better way + if [ "${arg%.delta}" != "$arg" ]; then + msg "$(gettext "Adding delta '%s'")" "$arg" + + if db_write_delta "$arg"; then + success=1 + fi + elif ! bsdtar -tf "$arg" .PKGINFO 2>&1 >/dev/null; then error "$(gettext "'%s' is not a package file, skipping")" "$arg" else msg "$(gettext "Adding package '%s'")" "$arg" -- 1.5.3.4
On 10/5/07, Nathan Jones <nathanj@insightbb.com> wrote:
This patch adds delta information to the repository database along with the other information about a package. This would be useful if two or more package versions have been built since a user upgrades.
I don't think it would be too difficult to create an algorithm to decide which delta files, if any, would be downloaded and applied.
Any comments?
Sample output:
$ ls -l *.pkg.tar.gz *.delta -rw-r--r-- 1 nathanj users 1045940 2007-05-12 21:03 pan-devel-0.129-1-i686.pkg.tar.gz -rw-r--r-- 1 nathanj users 583339 2007-10-05 18:38 pan-devel-0.129-1_to_0.130-1-i686.delta -rw-r--r-- 1 nathanj users 1044271 2007-05-19 13:17 pan-devel-0.130-1-i686.pkg.tar.gz -rw-r--r-- 1 nathanj users 425727 2007-10-05 18:38 pan-devel-0.130-1_to_0.131-1-i686.delta -rw-r--r-- 1 nathanj users 1055063 2007-05-26 18:40 pan-devel-0.131-1-i686.pkg.tar.gz -rw-r--r-- 1 nathanj users 680644 2007-10-05 18:38 pan-devel-0.131-1_to_0.132-1-i686.delta -rw-r--r-- 1 nathanj users 1025859 2007-08-15 22:03 pan-devel-0.132-1-i686.pkg.tar.gz -rw-r--r-- 1 nathanj users 1792 2007-10-05 18:38 pan-devel-0.132-1_to_0.132-2-i686.delta -rw-r--r-- 1 nathanj users 1025826 2007-08-01 17:04 pan-devel-0.132-2-i686.pkg.tar.gz $ ~/code/pacman/scripts/repo-add repo.db.tar.gz pan-devel-0.132-2-i686.pkg.tar.gz *.delta ==> Extracting database to a temporary location... ==> Adding package 'pan-devel-0.132-2-i686.pkg.tar.gz' -> Removing existing package 'pan-devel-0.132-2'... -> Creating 'desc' db entry... -> Computing md5 checksums... -> Creating 'depends' db entry... ==> Adding delta 'pan-devel-0.129-1_to_0.130-1-i686.delta' ==> Adding delta 'pan-devel-0.130-1_to_0.131-1-i686.delta' ==> Adding delta 'pan-devel-0.131-1_to_0.132-1-i686.delta' ==> Adding delta 'pan-devel-0.132-1_to_0.132-2-i686.delta' ==> Creating updated database file /home/nathanj/pkgs/pan-devel/repo.db.tar.gz $ tar zxOf repo.db.tar.gz pan-devel-0.132-2/deltas %DELTAS% 0.129-1 0.130-1 583339 0.130-1 0.131-1 425727 0.131-1 0.132-1 680644 0.132-1 0.132-2 1792
Signed-off-by: Nathan Jones <nathanj@insightbb.com> --- scripts/repo-add.sh.in | 64 +++++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 63 insertions(+), 1 deletions(-)
diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index cb741d7..4a7949e 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -97,6 +97,61 @@ write_list_entry() { fi }
+# write a delta entry to the pacman database +# arg1 - path to delta +db_write_delta() +{ + # blank out all variables and set deltafile + local deltafile=$(readlink -f "$1") + local pkgname fromver tover arch csize + + pkgname="$(echo "$1" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\1/')" + fromver="$(echo "$1" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\2/')" + tover="$(echo "$1" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\3/')" + arch="$(echo "$1" | sed -e 's/\(.*\)-\(.*-.*\)_to_\(.*-.*\)-\(.*\).delta/\4/')" + + # get size of delta + csize=$(du -b -L "$deltafile" | cut -f 1) + + pushd "$gstmpdir" 2>&1 >/dev/null + + # ensure $pkgname and $pkgver variables were found + if [ -z "$pkgname" -o -z "$fromver" ]; then + error "$(gettext "Invalid delta file '%s'.")" "$deltafile" + popd 2>&1 >/dev/null + return 1 + fi + + # cd into the package's directory + local existing + local gotpackage=f + for existing in *; do + if [ "${existing%-*-*}" = "$pkgname" ]; then + cd "$existing" + gotpackage=t + break + fi + done + + # exit out if the package's directory does not exist + if [ "$gotpackage" == "f" ]; then + error "$(gettext "Cannot find directory for '%s'.")" "$pkgname" + popd 2>&1 >/dev/null + return 1 + fi + + # create deltas file if it doesn't exist + if [ ! -f "deltas" ]; then + echo -e "%DELTAS%" >>deltas + fi + + # add the entry for this delta file + echo -e "$fromver $tover $csize" >>deltas + + popd 2>&1 >/dev/null +} # end db_write_delta + + # write an entry to the pacman database # arg1 - path to package db_write_entry() @@ -242,7 +297,14 @@ for arg in "$@"; do fi else if [ -f "$arg" ]; then - if ! bsdtar -tf "$arg" .PKGINFO 2>&1 >/dev/null; then + # see if extension is .delta ... there's gotta be a better way + if [ "${arg%.delta}" != "$arg" ]; then + msg "$(gettext "Adding delta '%s'")" "$arg" + + if db_write_delta "$arg"; then + success=1 + fi + elif ! bsdtar -tf "$arg" .PKGINFO 2>&1 >/dev/null; then error "$(gettext "'%s' is not a package file, skipping")" "$arg" else msg "$(gettext "Adding package '%s'")" "$arg" -- 1.5.3.4
This idea seems pretty smart to me. A few comments/questions: 1. We probably need a way to remove 'old' deltas. Of course, I don't know what to define as old (probably retain no more than 5 deltas or something). 2. Although sed magic is cool, there has to be a better way to get to that info (pkigname, oldver, etc.) 3. Could delta addition be automatic? Right now the entries would be deleted every time you upgrade a package in the database (since the old entry is removed). Perhaps on an add, we should save the old delta entries, copy them over to the new package, and then add any new deltas that we find? This is meant to be constructive criticism- please don't let that discourage you, I like this idea. -Dan
On 10/7/07, Dan McGee <dpmcgee@gmail.com> wrote:
1. We probably need a way to remove 'old' deltas. Of course, I don't know what to define as old (probably retain no more than 5 deltas or something).
I'd suggest doing it based on date (file creation date?) since some packages can get pkgrel bumps often in short periods of time (kernel26 anyone?). 6 months seems like a pretty good time span to purge old deltas.. Scott
On Sun, Oct 07, 2007 at 08:29:34PM -0500, Dan McGee wrote:
1. We probably need a way to remove 'old' deltas. Of course, I don't know what to define as old (probably retain no more than 5 deltas or something).
I was thinking that it would depend more on the size than the date or number. Once the combined size of the deltas becomes more than a certain percentage of the package's size, there is no reason to keep the older deltas around.
2. Although sed magic is cool, there has to be a better way to get to that info (pkigname, oldver, etc.)
Maybe deltas could be created as a tarball similarly to packages? file.delta.tar.gz/.DELTAINFO file.delta.tar.gz/delta
3. Could delta addition be automatic? Right now the entries would be deleted every time you upgrade a package in the database (since the old entry is removed). Perhaps on an add, we should save the old delta entries, copy them over to the new package, and then add any new deltas that we find?
I don't know how packages are upgraded (for my small repo, I just delete the repo file and recreate from scratch), so please correct me if this won't work. Instead of passing deltas via the command line, we would add any deltas that match the package. So 'repo-add repo pan-devel-0.132-2.pkg.tar.gz' would automatically add any files matching 'pan-devel-*-*_to_*-*.delta'. The deltas file would be recreated every upgrade.
participants (3)
-
Dan McGee
-
Nathan Jones
-
Scott Horowitz