[pacman-dev] delta support in libalpm

[Xavier]

On Sat, Nov 8, 2008 at 12:47 AM, Henning Garus
<henning.garus at googlemail.com> wrote:

>
> 2. create the package, but don't compress it with bsdtar, use gzip -n
> instead. This means we have to use gzip again, in libalpm, when we
> apply the delta.
>
> Seems better than 1, but makes makepkg and libalpm rely on gzip. Not
> sure if this is a good thing, especially for libalpm.
>

That sounds alright, I just noticed that xdelta3 has an option to
disable the external recompression : -R
So we don't even have extra decompression/recompression steps, there is no loss.

+                       snprintf(command, PATH_MAX, "xdelta3 -d -R -c
-s %s %s | gzip -n > %s", from, delta, to);


>
> In a previous mail Xavier toyed with the idea to put delta creation
> into repo-add, I have given this some thought, as it seems nice in
> principle, but there are drawbacks. For Arch this would mean creating
> deltas on Gerolde, which seems to be fairly strained already,
> according to the dev list. Furthermore this introduces some new
> variables to repo-add (at least repo location and an output location)
> this would be manageable, but doesn't look very nice.
>
> Delta creation in makepkg seems somehow ok (its already in there after
> all). But what I would really like is a separate tool for delta
> creation, which would allow the separation of building packages and
> creating deltas and setting up a separated delta server. This leaves
> us with options 2 and 3 and I am not really sure, which way to go.
>
>
> looking forward to your comments

I just went further than I ever did on this task, I seriously
considered a separate tool and spent some times thinking about all the
possibilities. I am still not sure what is best.
My thought was that it was very easy to generate the database info
during the delta creation. However, how do we keep this info?
Originally it was stored in the delta filename. But this was before
the database change. Now we need two filenames and two md5sums (old
pkg and new pkg), it does not seem realistic to store all this in the
delta filename.
Here are the options I considered :

1) delta support only in repo-add
No problem of a temporary storage of the info here, it goes directly
into the database. But maybe not flexible enough.

2) embed the .delta files into another format, eg delta.tar.gz archive
= delta file + DELTA metafile
Might be overkill? And we lost the ability of using xdelta directly

3) a separate tool creates the delta, generates the delta metainfo and
stores it in a file
This file can then be given to repo-add which basically just add its
contents to $pkgname-*/deltas

I gave a try to that third option. It's clearly not finished yet but I
am attaching the script in its current state to give an idea and to
know if I should move forward.
Example of usage:

$ create-xdelta libxml2-2.7.2-1-x86_64.pkg.tar.gz
libxml2-2.7.3-1-x86_64.pkg.tar.gz
$ create-xdelta libxml2-2.7.3-1-x86_64.pkg.tar.gz
libxml2-2.7.3-1.1-x86_64.pkg.tar.gz
(these two commands added one line each in a libxml2.pacdelta file)
$ repo-add db.tar.gz libxml2-2.7.3-1.1-x86_64.pkg.tar.gz
$ repo-add db.tar.gz libxml2.pacdelta
(pkg.tar.gz and pacdelta can be added together, but the order is
important, we need a package entry for libxml2 before adding deltas)

Now we can upgrade from 2.7.2-1 or 2.7.3-1 to 2.7.3-1.1 using deltas.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: create-xdelta
Type: application/octet-stream
Size: 2127 bytes
Desc: not available
URL: <http://www.archlinux.org/pipermail/pacman-dev/attachments/20090219/f07fc386/attachment.obj>