ArchLinux prefers xz compression now. It has a very good compression ratio but poor speed due not being threaded for smp usage.
It seems the upstream xz developer lost the intention to implement it after the final xz 5.0 release: http://sourceforge.net/projects/lzmautils/forums/forum/708858/topic/4015685
I've made some local quick tests for compressing large binary packages like LibreOffice on my Intel Quad core Q6600+ system using /tmp (tmpfs):
time tar -xvf ~/arch64/packages/libreoffice-3.3.0.2-2-x86_64.pkg.tar.xz
real 0m11.679s - that's well acceptable user 0m10.686s sys 0m1.117s
[andyrtr@workstation64 libo]$ du -sch 298M .
We currently use this in makepkg: time bsdtar -cf - * | xz -zf > LibO.tar.xz real 3m53.461s - very poor! user 3m52.382s sys 0m1.287s [andyrtr@workstation64 libo]$ ls -lha LibO.tar.xz -rw-r--r-- 1 andyrtr users 80M 31. Dez 13:14 LibO.tar.xz
1st alternative: PXZ http://jnovy.fedorapeople.org/pxz/ Using http://aur.archlinux.org/packages.php?ID=36362 I get this: [andyrtr@workstation64 libo]$ time bsdtar -cf - * | pxz -zf > LibO.tar.xz real 1m9.924s - not bad :) user 3m41.709s sys 0m1.530s [andyrtr@workstation64 libo]$ ls -lha LibO.tar.xz -rw-r--r-- 1 andyrtr users 81M 31. Dez 13:16 LibO.tar.xz
but I'm a bit afraid of the comments in AUR saying decompressing fails sometimes. I couldn't find something helpful in google for broken decompression.
2nd alternative: from the forum xz thread let's try parallel 7zip [andyrtr@workstation64 libo]$ time bsdtar -cf - * | 7z a -txz -si LibO.tar.xz
7-Zip [64] 9.13 beta Copyright (c) 1999-2010 Igor Pavlov 2010-04-15 p7zip Version 9.13 (locale=de_DE.UTF-8,Utf16=on,HugeFiles=on,4 CPUs) Creating archive LibO.tar.xz
Compressing [Content]
Everything is Ok
real 1m12.984s - also nice :) user 3m19.284s sys 0m1.667s [andyrtr@workstation64 libo]$ ls -lha LibO.tar.xz -rw-r--r-- 1 andyrtr users 79M 31. Dez 13:40 LibO.tar.xz
3rd alternative: lxz from the forum thread I couldn't find so far and should get further testing before hitting production state in a distribution.
What's your opinion? Is using p7zip a solution we can add to makepkg for now?
-Andy
Am 31.12.2010 13:59, schrieb Andreas Radke:
ArchLinux prefers xz compression now. It has a very good compression ratio but poor speed due not being threaded for smp usage.
It seems the upstream xz developer lost the intention to implement it after the final xz 5.0 release: http://sourceforge.net/projects/lzmautils/forums/forum/708858/topic/4015685
He didn't loose interest, not as far as I can see. This just has to be done right.
3rd alternative: lxz from the forum thread I couldn't find so far and should get further testing before hitting production state in a distribution.
http://lacos.hu/ Builds fine and works fine as well.
What's your opinion? Is using p7zip a solution we can add to makepkg for now?
I wouldn't add new code to makepkg for every new compression tool we find. This should be configurable in makepkg.conf, so everyone can use different compression commands.
Am Fri, 31 Dec 2010 15:29:16 +0100 schrieb Thomas Bächler thomas@archlinux.org:
http://lacos.hu/ Builds fine and works fine as well.
yes, it's another option.
[andyrtr@workstation64 libo]$ time bsdtar -cf - * | lxz > LibO.tar.xz
real 1m0.053s user 3m28.886s sys 0m2.763s [andyrtr@workstation64 libo]$ ls -lha LibO.tar.xz -rw-r--r-- 1 andyrtr users 82M 1. Jan 18:27 LibO.tar.xz
I wouldn't add new code to makepkg for every new compression tool we find. This should be configurable in makepkg.conf, so everyone can use different compression commands.
yes, something like that would be nice.
-Andy
On Sat, Jan 1, 2011 at 11:32 AM, Andreas Radke a.radke@arcor.de wrote:
Am Fri, 31 Dec 2010 15:29:16 +0100 schrieb Thomas Bächler thomas@archlinux.org:
http://lacos.hu/ Builds fine and works fine as well.
yes, it's another option.
[andyrtr@workstation64 libo]$ time bsdtar -cf - * | lxz > LibO.tar.xz
real 1m0.053s user 3m28.886s sys 0m2.763s [andyrtr@workstation64 libo]$ ls -lha LibO.tar.xz -rw-r--r-- 1 andyrtr users 82M 1. Jan 18:27 LibO.tar.xz
I wouldn't add new code to makepkg for every new compression tool we find. This should be configurable in makepkg.conf, so everyone can use different compression commands.
yes, something like that would be nice.
Be aware this is not near as simple as you might hope. Delta generation requires we pipeline things together, and also throws some curveballs to ensure we can properly reproduce packages that have valid checksums. Your specifiers would also need to allow changing PKGEXT on the fly for certain packages. See some of the following recent commits and changes:
http://projects.archlinux.org/pacman.git/commit/?id=38a60f49bdd7335f8e51e3d8... http://projects.archlinux.org/pacman.git/commit/?id=815557118318779f784e82a3... http://projects.archlinux.org/pacman.git/commit/?id=93862bd7cb91129cece07391... http://projects.archlinux.org/pacman.git/commit/?id=7ddb645bd7cf7e1c56a50832... http://projects.archlinux.org/pacman.git/commit/?id=c8beffa7904abe7e0ad01fed...
-Dan
pacman-dev@lists.archlinux.org