[pacman-dev] delta support in libalpm

[Allan McRae]

Henning Garus wrote:
> On Sat, Nov 8, 2008 at 1:38 PM, Nagy Gabor <ngaba at bibl.u-szeged.hu> wrote:
>   
>>> Hi,
>>>
>>> I have been looking through the current delta implementation in
>>> libalpm and have put some thought into changing makepkg/repo-add to
>>> support delta creation. However, I'm running into some problems,
>>> mostly due to md5sums and gzip.
>>>
>>> The current implementation works as follows. On a sync operation it is
>>> checked, whether a valid delta path exists and if the summed filesize
>>> of the deltas is smaller than the filesize of the whole download. When
>>> this is the case the deltas are downloaded and applied to the old
>>> file. After that the patched file is treated as if it was downloaded
>>> normally, this includes a check of the md5sum. Gzip files have a
>>> header, that has a timestamp, which will screw with this md5sum. When
>>> a patch is applied to a gzipped file by xdelta, xdelta will unzip the
>>> file, apply the patch and then rezip the file. The author of xdelta
>>> was obviously aware of the problems with the timestamp, because he
>>> decided to leave it empty. The same can be achieved by the -n option
>>> of gzip. But there comes the next problem, xdelta uses zlib for
>>> compression, gzip implements compression itself. And files created by
>>> gzip can differ from files created by zlib. Bsdtar uses zlib as well,
>>> but writes the timestamp and there is no option to prevent this (at
>>> least none that I can see).
>>>       
>> First of all, our current delta implementation doesn't work at all atm, see
>>        FS#12000. So any maintainer are welcome ;-)
>>
>>     
>
> I have yet to test this, but I think this comes down to repo-add not being in
> line with the current implementation, as you already pointed out in
> the discussion.
>
>
>   
>>> 2. create the package, but don't compress it with bsdtar, use gzip -n
>>> instead. This means we have to use gzip again, in libalpm, when we
>>> apply the delta.
>>>
>>> Seems better than 1, but makes makepkg and libalpm rely on gzip. Not
>>> sure if this is a good thing, especially for libalpm.
>>>       
>> maybe. But I don't see why we should use gzip in libalpm. Iirc we never
>> compress things in alpm.
>>
>>     
>
> Yes you do. libalpm uses system() to execute:
> xdelta patch [deltafile] [oldpkg] [newpkg]
>
> xdelta will unzip the old package, apply the patch and rezip the new package.
> Due to the zlib/gzip inconsistencies the md5sum for the patched
> package can differ from the md5sum of the new package, which was
> zipped with gzip. Unless that line is changed to something like
> xdelta patch -0 [deltafile] [oldpkg] - | gzip -cn > [newpkg]

Is any of this fixed by using the xdelta3 branch?  From memory that does 
not use gzip/bzip2.

Allan