[pacman-dev] [PATCH] pkgdelta: use highest compression ratio when creating deltas with xdelta3
Matthias Krüger
matthias.krueger at famsik.de
Fri Mar 14 23:12:22 EDT 2014
On 03/13/2014 07:11 AM, Allan McRae wrote:
> On 06/03/14 09:25, Matthias Krüger wrote:
>> Side note: it might be even more advantageous to use bsdiff instead of
>> xdelta3
>> comparing the /usr/bin/blender binaries of the above versions
>> (12:2.69.c7ac0e-1 and 13:2.69.13290d-1) :
>>
>> xdelta3 10.4M
>> xdelta3 -9 9.9M
>> bsdiff 4.7M
>
> I took a look, and changing from xdelta3 to bsdiff would be very simple.
> It looks like it is a five minute patch...
>
> But what I need is for someone to generate deltas (with and without -9
> maybe) for a whole bunch of packages. Then generate diffs using bsdiff
> and compare the results. The comparison will need to include:
>
> 1) size of deltas/diffs
> 2) memory used when reconstructing package
> 3) time taken to reconstruct package.
>
> Once we have that information, we can make an informed decision.
>
> Allan
>
>
I got some numbers for xdelta (-9), see attached file.
If someone provides some script or tool to run bsdiff on a package
properly (so that it does not diff the archives themselves), I can offer
to compute the numbers for bsdiff as well.
-------------- next part --------------
Packages delta'd:
acpid-2.0.20-1-x86_64.pkg.tar.xz acpid-2.0.21-1-x86_64.pkg.tar.xz
boost-1.54.0-4-x86_64.pkg.tar.xz boost-1.55.0-2-x86_64.pkg.tar.xz
calligra-libs-2.8.0-1-x86_64.pkg.tar.xz calligra-libs-2.8.0-2-x86_64.pkg.tar.xz
ffmpeg-1:2.1-2-x86_64.pkg.tar.xz ffmpeg-1:2.1-3-x86_64.pkg.tar.xz
flashplugin-11.2.202.336-1-x86_64.pkg.tar.xz flashplugin-11.2.202.341-1-x86_64.pkg.tar.xz
gcc-multilib-4.8.2-7-x86_64.pkg.tar.xz gcc-multilib-4.8.2-8-x86_64.pkg.tar.xz
gedit-3.10.3-1-x86_64.pkg.tar.xz gedit-3.10.4-1-x86_64.pkg.tar.xz
imagemagick-6.8.8.6-1-x86_64.pkg.tar.xz imagemagick-6.8.8.7-1-x86_64.pkg.tar.xz
kdelibs-4.12.2-1-x86_64.pkg.tar.xz kdelibs-4.12.3-1-x86_64.pkg.tar.xz
libpng-1.6.10-1-x86_64.pkg.tar.xz libpng-1.6.9-1-x86_64.pkg.tar.xz
libreoffice-base-4.1.4-3-x86_64.pkg.tar.xz libreoffice-base-4.1.5-1-x86_64.pkg.tar.xz
libreoffice-calc-4.1.4-3-x86_64.pkg.tar.xz libreoffice-calc-4.1.5-1-x86_64.pkg.tar.xz
linux-3.13.5-1-x86_64.pkg.tar.xz linux-3.13.6-1-x86_64.pkg.tar.xz
linux-firmware-20140123.418320b-1-any.pkg.tar.xz linux-firmware-20140217.343e460-1-any.pkg.tar.xz
man-pages-3.60-1-any.pkg.tar.xz man-pages-3.61-1-any.pkg.tar.xz
mariadb-clients-5.5.35-1-x86_64.pkg.tar.xz mariadb-clients-5.5.36-1-x86_64.pkg.tar.xz
openjdk7-doc-7.u51_2.4.4-1-x86_64.pkg.tar.xz openjdk7-doc-7.u51_2.4.5-1-x86_64.pkg.tar.xz
pacman-4.1.2-4-x86_64.pkg.tar.xz pacman-4.1.2-5-x86_64.pkg.tar.xz
pidgin-2.10.8-2-x86_64.pkg.tar.xz pidgin-2.10.9-1-x86_64.pkg.tar.xz
python2-2.7.6-2-x86_64.pkg.tar.xz python2-2.7.6-3-x86_64.pkg.tar.xz
ruby-2.1.0-2-x86_64.pkg.tar.xz ruby-2.1.1-1-x86_64.pkg.tar.xz
subversion-1.8.5-3-x86_64.pkg.tar.xz subversion-1.8.8-1-x86_64.pkg.tar.xz
systemd-210-2-x86_64.pkg.tar.xz systemd-210-3-x86_64.pkg.tar.xz
texlive-formatsextra-2013.30214-1-any.pkg.tar.xz texlive-formatsextra-2013.31958-1-any.pkg.tar.xz
valgrind-3.9.0-1-x86_64.pkg.tar.xz valgrind-3.9.0-2-x86_64.pkg.tar.xz
vlc-2.1.3-1-x86_64.pkg.tar.xz vlc-2.1.4-1-x86_64.pkg.tar.xz
xbmc-12.3-11-x86_64.pkg.tar.xz xbmc-12.3-12-x86_64.pkg.tar.xz
xf86-video-ark-0.7.5-3-x86_64.pkg.tar.xz xf86-video-ark-0.7.5-4-x86_64.pkg.tar.xz
zsh-5.0.2-4-x86_64.pkg.tar.xz zsh-5.0.5-1-x86_64.pkg.tar.xz
xdelta3 -q -f -s (-9) in k:
20 (16) acpid-2.0.20-1_to_acpid-2.0.21-1.delta
13104 (11744) boost-1.54.0-4_to_boost-1.55.0-2.delta
364 (336) calligra-libs-2.8.0-1_to_calligra-libs-2.8.0-2.delta
792 (728) ffmpeg-1:2.1-2_to_ffmpeg-1:2.1-3.delta
2628 (2512) flashplugin-11.2.202.336-1_to_flashplugin-11.2.202.341-1.delta
11252 (10176) gcc-multilib-4.8.2-7_to_gcc-multilib-4.8.2-8.delta
436 (384) gedit-3.10.3-1_to_gedit-3.10.4-1.delta
612 (548) imagemagick-6.8.8.6-1_to_imagemagick-6.8.8.7-1.delta
6412 (5424) kdelibs-4.12.2-1_to_kdelibs-4.12.3-1.delta
104 (100) libpng-1.6.10-1_to_libpng-1.6.9-1.delta
1504 (1452) libreoffice-base-4.1.4-3_to_libreoffice-base-4.1.5-1.delta
5060 (4228) libreoffice-calc-4.1.4-3_to_libreoffice-calc-4.1.5-1.delta
47188 (46736) linux-3.13.5-1_tp_linux-3.13.6-1.delta
1128 (1104) man-pages-3.60-1_to_man-pages-3.61.delta
540 (496) mariadb-clients-5.5.35_to_mariadb-clients-5.5.36.delta
18172 (14360) openjdk7-doc-7.u51_2.4.4-1_to_openjdk7-doc-7.u51_2.4.5-1.delta
44 (44) pacman-4.1.2-4_to_pacman-4.1.2-5.delta
120 (112) pidgin-2.10.8-2_to_pidgin-2.10.9-1.delta
372 (352) python2-2.7.6-2_to_python2-2.7.6-3.delta
1228 (1108) ruby-2.1.0-2_to_ruby-2.1.1-1.delta
540 (476) subversion-1.8.5-3_to_subversion-1.8.8-1.delta
816 (740) systemd-210-2-_to_systemd-210-3.delta
12 (12) texlive-formatsextra-2013.30214-1_to_texlive-formatsextra-2013.31958-1.delta
8180 (7400) valgrind-3.9.0-1_to_valgrind-3.9.0-2.delta
3932 (3692) vlc-2.1.3-1_to_vlc-2.1.4-1.delta
168 (160) xbmc-12.3-11_to_xbmc-12.3-12.delta
8 (8) xf86-video-ark-0.7.5-3-_to_xf86-video-ark-0.7.5-4.delta
944 (892) zsh-5.0.2-4-x86_64_to_zsh-5.0.5-1.delta
time (time) and peak memory consuption (while true ; do ; ps_mem | grep xdelta ; done ) while composing packages from delta:
xdelta3 (-9) -q -f -s :
acpid: 0.121s, too fast (0m0.112s, too fast)
boost: 1m20.219s, 85.3 MiB (1m19.629s, 80.3 MiB)
calligra-libs: 0m10.975s, 16.4 MiB (0m10.289s, 16.4 MiB)
ffmpeg: 0m11.624s, 19.9 MiB (0m11.951s, 19.8 MiB)
flashplugin: 0m19.083s, 33.5 MiB (0m19.549s, 33.4 MiB)
gcc-multilib: 1m8.137s, 84.9 MiB (1m3.846s, 80.8 MiB)
gedit: 0m9.216s, 19.1 MiB (0m9.355s, 19.0 MiB)
imagemagic: 0m7.125s, 12.8 MiB (0m7.544s, 12.8 MiB)
kdelibs: 0m46.660s, 68.5 MiB (0m47.704s, 67.5 MiB)
libpng: 0m0.363s, too fast (0m0.405s, too fast)
libreoffice-base: 0m8.988s, 16.0 MiB (0m8.776s, 15.9 MiB)
libreoffice-calc: 0m20.316s, 30.9 MiB (0m20.166s, 30.1 MiB)
linux: 0m45.753s, 92.4 MiB (0m47.502s, 87.3 MiB)
man-pages: 0m4.197s, 14.3 MiB (0m4.296s, 14.2 MiB)
mariadb-clients: 0m6.221s, 21.6 MiB (0m6.461s, 21.5 MiB)
openjdk7-doc: 1m38.915s, 88.7 MiB (1m40.888s, 86.9 MiB)
pacman: 0m2.143s, 4.3 MiB (0m2.115s, 4.3 MiB)
pidgin: 0m2.553s, 6.0 MiB (0m2.741s, 6.0 MiB)
python: 0m48.090s, 71.0 MiB (0m48.613s, 71.0 MiB)
ruby: 0m14.514s, 23.4 MiB (0m14.678s, 23.3 MiB)
subversion: 0m25.814s, 36.5 MiB (0m25.852s, 36.4 MiB)
systemd: 0m13.930s, 21.5 MiB (0m13.330s, 21.4 MiB)
texlive-formatsextra: 0m0.755s, too fast (0m0.730s, 1.4 MiB)
valgrind: 0m40.352s, 69.4 MiB (0m40.980s, 68.6 MiB)
vlc: 0m34.229s, 56.6 MiB (0m34.159s, 56.4 MiB)
xbmc: 0m44.762s, 66.2 MiB (0m45.335s, 66.2 MiB)
xf86-video-ark: 0m0.044s, too fast (0m0.041s, too fast)
zsh: 0m4.045s, 8.1 MiB (0m3.929s, 8.0 MiB)
Note that not all of the listed deltas actually are "worth it", as they are bigger than the new package and would be deleted.
Also my system is not the latest, CPU: Intel Core 2 Duo CPU P8600 (2.4Ghz)
More information about the pacman-dev
mailing list