[pacman-dev] [PATCH] pkgdelta: use highest compression ratio when creating deltas with xdelta3

[Matthias Krüger]

On 03/06/2014 12:33 AM, Allan McRae wrote:
> On 06/03/14 09:25, Matthias Krüger wrote:
>> Looking how pkgdelta works, I found this line
>>      xdelta3  -q -f -s "$oldfile" "$newfile" "$deltafile" || ret=$?
>> which seemed to be responsible for the actual delta generation, however
>>      man xdelta3
>> revealed that there were different compression levels (0-9) (not sure
>> which one is default).
>> To make it short, we could have had smaller deltas since pkgdelta was
>> introduced!
>>
>> Examples:
>>
>> -9          16660K
>> blender-12:2.69.c7ac0e-1_to_13:2.69.13290d-1-x86_64.delta
>>   default  17832K blender-12:2.69.c7ac0e-1_to_13:2.69.13290d-1-x86_64.delta
>>
>> -9         504K    xbmc-12.3-10_to_12.3-11-x86_64.delta
>> default  572K   xbmc-12.3-10_to_12.3-11-x86_64.delta
>>
> How is memory usage changed?   Mainly when regenerating the package from
> deltas?
Surprisingly, for blender both runs took ~96MB and 1:50m (+- a second).
>
>> The attached patch adds the "-9" option to the line above.
>>
>> Side note: it might be even more advantageous to use bsdiff instead of
>> xdelta3
>> comparing the /usr/bin/blender binaries of the above versions
>> (12:2.69.c7ac0e-1 and 13:2.69.13290d-1) :
>>
>> xdelta3     10.4M
>> xdelta3 -9   9.9M
>> bsdiff          4.7M
>>
>> ,however bsdiff is only for direct diff creation, bsdiff foo1.pkg.tar.xz
>> foo2.pkg.tar.xz will create a diff between the archives and not their
>> contents, so that'd require further coding which I am unable to do
>> (maybe someone else volunteers :) ).
> What do you mean "create a diff between the archives and not their
> contents".   That does seem a significant decrease.
To my understanding xdelta opens the archive and diffs the files 
contained in it while bsdiff will just generate a diff between the 
archives. At least it would fit the observations:

xdelta ${args} blender-12:2.69.c7ac0e-1-x86_64.pkg.tar.xz 
blender-13:2.69.13290d-1-x86_64.pkg.tar.xz out
=> 17 M
bsdiff blender-12:2.69.c7ac0e-1-x86_64.pkg.tar.xz 
blender-13:2.69.13290d-1-x86_64.pkg.tar.xz out
=> 37 M
(while creating this diff, bsdiff ate up to ~660M ram :/)

 From man bsdiff:

bsdiff uses memory equal to 17 times the size of ⟨oldfile⟩, and requires 
an absolute minimum working set size of 8 times the
size of oldfile.


Regards, Matthias