On 03/06/2014 12:33 AM, Allan McRae wrote:
On 06/03/14 09:25, Matthias Krüger wrote:
Looking how pkgdelta works, I found this line xdelta3 -q -f -s "$oldfile" "$newfile" "$deltafile" || ret=$? which seemed to be responsible for the actual delta generation, however man xdelta3 revealed that there were different compression levels (0-9) (not sure which one is default). To make it short, we could have had smaller deltas since pkgdelta was introduced!
Examples:
-9 16660K blender-12:2.69.c7ac0e-1_to_13:2.69.13290d-1-x86_64.delta default 17832K blender-12:2.69.c7ac0e-1_to_13:2.69.13290d-1-x86_64.delta
-9 504K xbmc-12.3-10_to_12.3-11-x86_64.delta default 572K xbmc-12.3-10_to_12.3-11-x86_64.delta
How is memory usage changed? Mainly when regenerating the package from deltas? Surprisingly, for blender both runs took ~96MB and 1:50m (+- a second).
The attached patch adds the "-9" option to the line above.
Side note: it might be even more advantageous to use bsdiff instead of xdelta3 comparing the /usr/bin/blender binaries of the above versions (12:2.69.c7ac0e-1 and 13:2.69.13290d-1) :
xdelta3 10.4M xdelta3 -9 9.9M bsdiff 4.7M
,however bsdiff is only for direct diff creation, bsdiff foo1.pkg.tar.xz foo2.pkg.tar.xz will create a diff between the archives and not their contents, so that'd require further coding which I am unable to do (maybe someone else volunteers :) ). What do you mean "create a diff between the archives and not their contents". That does seem a significant decrease. To my understanding xdelta opens the archive and diffs the files contained in it while bsdiff will just generate a diff between the archives. At least it would fit the observations:
xdelta ${args} blender-12:2.69.c7ac0e-1-x86_64.pkg.tar.xz blender-13:2.69.13290d-1-x86_64.pkg.tar.xz out => 17 M bsdiff blender-12:2.69.c7ac0e-1-x86_64.pkg.tar.xz blender-13:2.69.13290d-1-x86_64.pkg.tar.xz out => 37 M (while creating this diff, bsdiff ate up to ~660M ram :/) From man bsdiff: bsdiff uses memory equal to 17 times the size of ⟨oldfile⟩, and requires an absolute minimum working set size of 8 times the size of oldfile. Regards, Matthias