[pacman-dev] [PATCH] pkgdelta: use highest compression ratio when creating deltas with xdelta3
matthias.krueger at famsik.de
Thu Mar 20 22:28:08 EDT 2014
On 03/16/2014 04:34 AM, Allan McRae wrote:
> On 15/03/14 13:12, Matthias Krüger wrote:
>> On 03/13/2014 07:11 AM, Allan McRae wrote:
>>> On 06/03/14 09:25, Matthias Krüger wrote:
>>>> Side note: it might be even more advantageous to use bsdiff instead of
>>>> comparing the /usr/bin/blender binaries of the above versions
>>>> (12:2.69.c7ac0e-1 and 13:2.69.13290d-1) :
>>>> xdelta3 10.4M
>>>> xdelta3 -9 9.9M
>>>> bsdiff 4.7M
>>> I took a look, and changing from xdelta3 to bsdiff would be very simple.
>>> It looks like it is a five minute patch...
>>> But what I need is for someone to generate deltas (with and without -9
>>> maybe) for a whole bunch of packages. Then generate diffs using bsdiff
>>> and compare the results. The comparison will need to include:
>>> 1) size of deltas/diffs
>>> 2) memory used when reconstructing package
>>> 3) time taken to reconstruct package.
>>> Once we have that information, we can make an informed decision.
>> I got some numbers for xdelta (-9), see attached file.
>> If someone provides some script or tool to run bsdiff on a package
>> properly (so that it does not diff the archives themselves), I can offer
>> to compute the numbers for bsdiff as well.
> The numbers of -9 look like there is no significant change. A quick look
> here showed also no change adding -S djw. I'll accept a patch adding
> both -S djw and -9 to the diff creation.
> I ran some tests on my system. bsdiff uses masses of memory when
> reconstructing the file (needs ~16x the size of the file). It can use
> less, but the performance penalty is massive. And it uses even more
> when creating the diff. Coupled with its lack of transparent
> decompression, I don't think we should consider that further.
Uhmm, according to the bsdiff website:
> bsdiff is quite memory-hungry. It requires max(17*n,9*n+m)+O(1) bytes
of memory, where n is the size of the old file and m is the size of the
> bspatch requires n+m+O(1) bytes.
I did a quick test and generating a 40 MB newfile (binary executable)
from a 40 MB oldfile and 4kb patch was done very quickly, ps_mem said
bspatch needed around 98.2 MB for the one time I managed to actually
Fedora seems to have some deltarpm tool https://gitorious.org/deltarpm
which seems to make use of bsdiff, maybe this can be tweaked to be used
for arch packages?
More information about the pacman-dev