[pacman-dev] delta support in libalpm

[Aaron Griffin]

On Fri, Feb 27, 2009 at 6:18 AM, Xavier <shiningxc at gmail.com> wrote:
> On Fri, Feb 27, 2009 at 12:37 AM, Aaron Griffin <aaronmgriffin at gmail.com> wrote:
>> On Thu, Feb 26, 2009 at 5:21 PM, Xavier <shiningxc at gmail.com> wrote:
>>> On Thu, Feb 26, 2009 at 11:34 PM, Aaron Griffin <aaronmgriffin at gmail.com> wrote:
>>>>
>>>> So, ok, from a db-scripts point of view, we're going to have to do the
>>>> following:
>>>>
>>>> when a new package is added:
>>>>   copy old package file from ftp to build dir
>>>>   generate delta from old file -> new file (in staging)
>>>
>>> $ oldfile=tzdata-2009a-1-x86_64.pkg.tar.gz
>>> $ newfile=tzdata-2009b-1-x86_64.pkg.tar.gz
>>> $ pkgdelta $oldfile $newfile
>>> ==> Generating delta from version 2009a-1 to version 2009b-1
>>> ==> Generated delta : 'tzdata-2009a-1_to_2009b-1-x86_64.delta'
>> ...snip...
>>> Indeed, it should be possible to put that in repo-add, I seriously
>>> considered that option a while ago.
>>> But using pkgdelta and repo-add on delta files should be easy enough,
>>> as you can see above.
>>
>> Hmm, actually this isn't as straightforward as it seems as we always
>> have the possibility that multiple pkg files exist (haven't been
>> cleaned up yet), so we need to determine the previous file's full
>> name. That's why I mentioned doing it in repo-add, as repo-add knows
>> the entry in the DB already and can very easily generate a new delta
>> before it removes the old entry.
>>
>
> Hmm ok. So how do you control repo-add delta creation, do we need a
> new -d/--delta flag for it, or an environment variable REPO_DELTA=1 ?
> And where should repo-add assume that the oldfile is? Current working
> directory, same directory as the database, same directory as the
> package being added?
> And where should the newly created delta be put?
> Finally, db_remove_entry supports the odd corner case that there could
> be multiple entries for the same pkgname. How should delta creation
> deal with that? Only pick the first entry, or try to generate a delta
> for each entry?
>
> That is this kind of questions which bothers me the most, because I am
> never totally sure what the use case will be. You could be of a great
> help here :)

Hmm, good questions. I was thinking that we would simply run "repo-add
--deltas" or something, but it still requires us to figure out the old
package file and copy that as well.

That's the main issue here - determining the old/existing package
file. I guess it's not TOO hard, the more I think about it. I'll just
have to extract the DB and parse through it, unless someone else can
figure out a cleaner way to get the filename for an old package from a
DB knowing the new package's name (and version and all the other meta
info)

>> To do this in the db-scripts, we need to open the existing DB, find
>> the entry for this pkgname, and get the filename. Does pkgdelta
>> correctly construct deltas in different directories, or do we also
>> need to copy the file next to the new package before running pkgdelta
>> need to be next to each other?
>
> The old package and new package can be anywhere. But the new delta
> will be put next to the new package.
> What would be the best output location for the new delta?

That works fine. The way the dbscripts do operations is by:
a) locking the existing DB (with a lock file)
b) copying the db file to a temp dir
c) copying all staging files to that temp dir
d) adding everything to the db
e) copying the entire temp dir to the existing db location
f) unlocking the db

So putting deltas in-place works fine - I was just hoping we wouldn't
have to figure out additional files to copy to the temp dir for deltas