[pacman-dev] parallel compression in makepkg
Andreas Radke
a.radke at arcor.de
Fri Dec 31 07:59:28 EST 2010
ArchLinux prefers xz compression now. It has a very good compression
ratio but poor speed due not being threaded for smp usage.
It seems the upstream xz developer lost the intention to implement it
after the final xz 5.0 release:
http://sourceforge.net/projects/lzmautils/forums/forum/708858/topic/4015685
I've made some local quick tests for compressing large binary packages
like LibreOffice on my Intel Quad core Q6600+ system using /tmp (tmpfs):
time tar -xvf ~/arch64/packages/libreoffice-3.3.0.2-2-x86_64.pkg.tar.xz
real 0m11.679s - that's well acceptable
user 0m10.686s
sys 0m1.117s
[andyrtr at workstation64 libo]$ du -sch
298M .
We currently use this in makepkg:
time bsdtar -cf - * | xz -zf > LibO.tar.xz
real 3m53.461s - very poor!
user 3m52.382s
sys 0m1.287s
[andyrtr at workstation64 libo]$ ls -lha LibO.tar.xz
-rw-r--r-- 1 andyrtr users 80M 31. Dez 13:14 LibO.tar.xz
1st alternative: PXZ
http://jnovy.fedorapeople.org/pxz/
Using http://aur.archlinux.org/packages.php?ID=36362 I get this:
[andyrtr at workstation64 libo]$ time bsdtar -cf - * | pxz -zf >
LibO.tar.xz
real 1m9.924s - not bad :)
user 3m41.709s
sys 0m1.530s
[andyrtr at workstation64 libo]$ ls -lha LibO.tar.xz
-rw-r--r-- 1 andyrtr users 81M 31. Dez 13:16 LibO.tar.xz
but I'm a bit afraid of the comments in AUR saying decompressing fails
sometimes. I couldn't find something helpful in google for broken
decompression.
2nd alternative: from the forum xz thread let's try parallel 7zip
[andyrtr at workstation64 libo]$ time bsdtar -cf - * | 7z a -txz -si
LibO.tar.xz
7-Zip [64] 9.13 beta Copyright (c) 1999-2010 Igor Pavlov 2010-04-15
p7zip Version 9.13 (locale=de_DE.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)
Creating archive LibO.tar.xz
Compressing [Content]
Everything is Ok
real 1m12.984s - also nice :)
user 3m19.284s
sys 0m1.667s
[andyrtr at workstation64 libo]$ ls -lha LibO.tar.xz
-rw-r--r-- 1 andyrtr users 79M 31. Dez 13:40 LibO.tar.xz
3rd alternative: lxz from the forum thread I couldn't find so far and
should get further testing before hitting production state in a
distribution.
What's your opinion? Is using p7zip a solution we can add to makepkg
for now?
-Andy
More information about the pacman-dev
mailing list