[pacman-dev] parallel compression in makepkg

[Andreas Radke]

ArchLinux prefers xz compression now. It has a very good compression
ratio but poor speed due not being threaded for smp usage.

It seems the upstream xz developer lost the intention to implement it
after the final xz 5.0 release:
http://sourceforge.net/projects/lzmautils/forums/forum/708858/topic/4015685

I've made some local quick tests for compressing large binary packages
like LibreOffice on my Intel Quad core Q6600+ system using /tmp (tmpfs):

time tar -xvf ~/arch64/packages/libreoffice-3.3.0.2-2-x86_64.pkg.tar.xz

real	0m11.679s - that's well acceptable
user	0m10.686s
sys	0m1.117s

[andyrtr at workstation64 libo]$ du -sch
298M	.

We currently use this in makepkg:
time bsdtar -cf - * | xz -zf > LibO.tar.xz
real	3m53.461s - very poor!
user	3m52.382s
sys	0m1.287s
[andyrtr at workstation64 libo]$ ls -lha LibO.tar.xz 
-rw-r--r-- 1 andyrtr users 80M 31. Dez 13:14 LibO.tar.xz

1st alternative: PXZ
http://jnovy.fedorapeople.org/pxz/
Using http://aur.archlinux.org/packages.php?ID=36362 I get this:
[andyrtr at workstation64 libo]$ time bsdtar -cf - * | pxz -zf >
LibO.tar.xz 
real	1m9.924s - not bad :)
user	3m41.709s
sys	0m1.530s
[andyrtr at workstation64 libo]$ ls -lha LibO.tar.xz 
-rw-r--r-- 1 andyrtr users 81M 31. Dez 13:16 LibO.tar.xz

but I'm a bit afraid of the comments in AUR saying decompressing fails
sometimes. I couldn't find something helpful in google for broken
decompression.

2nd alternative: from the forum xz thread let's try parallel 7zip
[andyrtr at workstation64 libo]$ time bsdtar -cf - * | 7z a -txz -si
LibO.tar.xz

7-Zip [64] 9.13 beta  Copyright (c) 1999-2010 Igor Pavlov  2010-04-15
p7zip Version 9.13 (locale=de_DE.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)
Creating archive LibO.tar.xz

Compressing  [Content]      

Everything is Ok

real	1m12.984s - also nice :)
user	3m19.284s
sys	0m1.667s
[andyrtr at workstation64 libo]$ ls -lha LibO.tar.xz 
-rw-r--r-- 1 andyrtr users 79M 31. Dez 13:40 LibO.tar.xz

3rd alternative: lxz from the forum thread I couldn't find so far and
should get further testing before hitting production state in a
distribution.

What's your opinion? Is using p7zip a solution we can add to makepkg
for now?

-Andy