[arch-general] About Arch pkg compress format
Hi all I just have a sugguestion about the pkg format. Currently archlinux is using gzip compress method to pack the package, I think it is better to use bzip2. since bzip2 format can save about 10%-20% disk space than gzip. As I know, FreeBSD has already switch its package format from gzip to bzip2 a few year ago. The back is bzip2 takes more time to compress/decompress. since we are using morden computer. it is not a big problem. (arch only support i686/x64) Regards. Qian -------------------------------------- GANBARE! NIPPON! Win your ticket to Olympic Games 2008. http://pr.mail.yahoo.co.jp/ganbare-nippon/
Bendany Qian wrote:
I just have a sugguestion about the pkg format.
That is a very common suggestion. See http://bugs.archlinux.org/task/2264 for more information. Allan
On Tue, 2008-05-20 at 15:55 +0900, Bendany Qian wrote:
Hi all
I just have a sugguestion about the pkg format.
Currently archlinux is using gzip compress method to pack the package, I think it is better to use bzip2. since bzip2 format can save about 10%-20% disk space than gzip.
As I know, FreeBSD has already switch its package format from gzip to bzip2 a few year ago.
The back is bzip2 takes more time to compress/decompress. since we are using morden computer. it is not a big problem. (arch only support i686/x64)
Regards. Qian
-------------------------------------- GANBARE! NIPPON! Win your ticket to Olympic Games 2008. http://pr.mail.yahoo.co.jp/ganbare-nippon/
Pacman itself is ready for .tar.bz2 package files. The whole issue with .bz2 files is that compression and decompression times increase a lot without giving the same amount of size reduction back. We've done some recent tests with LZMA, which compresses just as good as bzip2 at the lowest compression rate, but does it at the same speed as gzip. The downside is that LZMA is not supported by libarchive, and won't be supported officially either, because libarchive is BSD licensed and LZMA is GPL licensed.
On Tue, May 20, 2008 at 10:14 AM, Jan de Groot
Pacman itself is ready for .tar.bz2 package files. The whole issue with .bz2 files is that compression and decompression times increase a lot without giving the same amount of size reduction back. We've done some recent tests with LZMA, which compresses just as good as bzip2 at the lowest compression rate, but does it at the same speed as gzip. The downside is that LZMA is not supported by libarchive, and won't be supported officially either, because libarchive is BSD licensed and LZMA is GPL licensed.
Is it possible and/or realistic to imagine a BSD licensed implementation of LZMA?
Idézés Jan de Groot
On Tue, 2008-05-20 at 15:55 +0900, Bendany Qian wrote:
Hi all
I just have a sugguestion about the pkg format.
Currently archlinux is using gzip compress method to pack the package, I think it is better to use bzip2. since bzip2 format can save about 10%-20% disk space than gzip.
As I know, FreeBSD has already switch its package format from gzip to bzip2 a few year ago.
The back is bzip2 takes more time to compress/decompress. since we are using morden computer. it is not a big problem. (arch only support i686/x64)
Regards. Qian
-------------------------------------- GANBARE! NIPPON! Win your ticket to Olympic Games 2008. http://pr.mail.yahoo.co.jp/ganbare-nippon/
Pacman itself is ready for .tar.bz2 package files. The whole issue with .bz2 files is that compression and decompression times increase a lot without giving the same amount of size reduction back. We've done some recent tests with LZMA, which compresses just as good as bzip2 at the lowest compression rate, but does it at the same speed as gzip. The downside is that LZMA is not supported by libarchive, and won't be supported officially either, because libarchive is BSD licensed and LZMA is GPL licensed.
I also like LZMA. It's time to create a GPL licensed fork of libarchive;-) (btw, I hate these license wars.) Bye ---------------------------------------------------- SZTE Egyetemi Könyvtár - http://www.bibl.u-szeged.hu This mail sent through IMP: http://horde.org/imp/
Hi, On Tue, 2008-05-20 at 10:14 +0200, Jan de Groot wrote:
Pacman itself is ready for .tar.bz2 package files. The whole issue with .bz2 files is that compression and decompression times increase a lot without giving the same amount of size reduction back. We've done some recent tests with LZMA, which compresses just as good as bzip2 at the lowest compression rate, but does it at the same speed as gzip.
About LZMA I should add that when using higher (actually the default) compression rate, compression is much better than bzip2 but takes more time/memory. Decompression however, which is what really matters in a packaging format, is kept fast and lightweight.
The downside is that LZMA is not supported by libarchive, and won't be supported officially either, because libarchive is BSD licensed and LZMA is GPL licensed.
Can't this problem be circumvented by spawning the lzma command line utility, and piping all data to it? I understand that this perhaps negates the purpose of libarchive, but the overhead should be small. Thanks, Dimitris
On Wed, May 21, 2008 at 5:47 PM, Dimitrios Apostolou
Can't this problem be circumvented by spawning the lzma command line utility, and piping all data to it? I understand that this perhaps negates the purpose of libarchive, but the overhead should be small.
That indeed totally defeats the purpose. We currently have a neat and transparent way to access all kind of different archives supported by libarchive, without any hacks.
On Wed, 2008-05-21 at 18:47 +0300, Dimitrios Apostolou wrote:
Hi,
On Tue, 2008-05-20 at 10:14 +0200, Jan de Groot wrote:
Pacman itself is ready for .tar.bz2 package files. The whole issue with .bz2 files is that compression and decompression times increase a lot without giving the same amount of size reduction back. We've done some recent tests with LZMA, which compresses just as good as bzip2 at the lowest compression rate, but does it at the same speed as gzip.
About LZMA I should add that when using higher (actually the default) compression rate, compression is much better than bzip2 but takes more time/memory. Decompression however, which is what really matters in a packaging format, is kept fast and lightweight.
What's the memory usage when unzipping an LZMA file? Is it much higher than the needs of gzip? We already have problems supporting low-memory systems with our installer, adding a compression algorithm that eats more memory will cause even more problems for these systems.
The downside is that LZMA is not supported by libarchive, and won't be supported officially either, because libarchive is BSD licensed and LZMA is GPL licensed.
Can't this problem be circumvented by spawning the lzma command line utility, and piping all data to it? I understand that this perhaps negates the purpose of libarchive, but the overhead should be small.
libarchive can't spawn external commands to unzip archive files. bsdtar can, but as pacman is a binary with libarchive integrated, I don't see this happening.
On Wed, May 21, 2008 at 12:39 PM, Jan de Groot
On Wed, 2008-05-21 at 18:47 +0300, Dimitrios Apostolou wrote:
Hi,
On Tue, 2008-05-20 at 10:14 +0200, Jan de Groot wrote:
Pacman itself is ready for .tar.bz2 package files. The whole issue with .bz2 files is that compression and decompression times increase a lot without giving the same amount of size reduction back. We've done some recent tests with LZMA, which compresses just as good as bzip2 at the lowest compression rate, but does it at the same speed as gzip.
About LZMA I should add that when using higher (actually the default) compression rate, compression is much better than bzip2 but takes more time/memory. Decompression however, which is what really matters in a packaging format, is kept fast and lightweight.
What's the memory usage when unzipping an LZMA file? Is it much higher than the needs of gzip? We already have problems supporting low-memory systems with our installer, adding a compression algorithm that eats more memory will cause even more problems for these systems.
That's actually not entirely true. Dan and I investigated this. The previous low memory issues were caused by the entire install system never leaving the initramfs, and remaining entirely in RAM - which soaked far more than pacman ever will. Additionally, with the dynamic package patch (dunno if this is in the master branch yet), memory usage sank by leaps and bounds.
Aaron Griffin wrote:
That's actually not entirely true. Dan and I investigated this. The previous low memory issues were caused by the entire install system never leaving the initramfs, and remaining entirely in RAM - which soaked far more than pacman ever will. Additionally, with the dynamic package patch (dunno if this is in the master branch yet), memory usage sank by leaps and bounds.
Yes it is, go go Dan :) http://toofishes.net/blog/valgrind-330-and-new-massif/
On Wed, 2008-05-21 at 19:39 +0200, Jan de Groot wrote:
What's the memory usage when unzipping an LZMA file? Is it much higher than the needs of gzip? We already have problems supporting low-memory systems with our installer, adding a compression algorithm that eats more memory will cause even more problems for these systems.
Yes it eats more memory in comparison to gzip, at least when high compression is chosen. But still the memory it uses is not much. From the official site of the LZMA utils: http://tukaani.org/lzma/benchmarks RAM usage on decompression gzip bzip2 lzmash lzmash -e 1 <1 MB 1 MB 1 MB - 2 <1 MB 2 MB 2 MB - 3 <1 MB 2 MB 1 MB 1 MB 4 <1 MB 2 MB 2 MB 2 MB 5 <1 MB 3 MB 3 MB 3 MB 6 <1 MB 3 MB 5 MB 5 MB 7 <1 MB 3 MB 9 MB 9 MB 8 <1 MB 4 MB 17 MB 17 MB 9 <1 MB 4 MB 33 MB 33 MB The memory usage of lzma stays competitive with bzip2 when files have been compressed with "lzmash -6" or with a smaller option. The files compressed with the default "lzmash -7" can still be decompressed, even on machines with only 16 MB of RAM, but sometimes you don't have even that much memory available. If you compress with "lzmash -8" or "lzmash -9", you should think if the users need to be able to decompress your files also on "ancient" computers. I fully understand the license problem and personally I am pretty happy with the gzip algorithm, I am just pointing facts so that a technically correct conclusion is reached. Dimitris
On Tue, May 20, 2008 at 5:14 AM, Jan de Groot
The downside is that LZMA is not supported by libarchive, and won't be supported officially either, because libarchive is BSD licensed and LZMA is GPL licensed.
Hi, It seems that LZMA lib is licensed with LGPL and has an special exception that permits to link (statically or dinamicaly) without being bound by the LGPL terms. See http://www.7-zip.org/sdk.html Maybe libarchive could be modified to support lzma also. -- ------------------------------------------- Denis A. Altoe Falqueto -------------------------------------------
On Wed, 21 May 2008, Denis Alessandro Altoe Falqueto wrote:
It seems that LZMA lib is licensed with LGPL and has an special exception that permits to link (statically or dinamicaly) without being bound by the LGPL terms.
I thought LGPL permist statically linking anyways, which is why WXwidgets eg is using it. -T
On Thursday 22 May 2008 00:13:27 Tobias Kieslich wrote:
On Wed, 21 May 2008, Denis Alessandro Altoe Falqueto wrote:
It seems that LZMA lib is licensed with LGPL and has an special exception that permits to link (statically or dinamicaly) without being bound by the LGPL terms.
I thought LGPL permist statically linking anyways, which is why WXwidgets eg is using it.
The LGPL is intentionaly hard to read and confuses a lot of people aparantly. Grats for that FSI. Yes you are allowed to link dynamicly, now read if you are allowed to run the linked program. Not speaking of actually distributing it. Oh and btw, the copyright law of the country of the author applies, which makes it just about a gazillion times more complex. Somone adding that exception indicates they actually read it and found they just want people to link it without consulting a lawyer. But uuuh hopefully people won't go bezerk now and check all the packages in the repo that violate copyright laws. Its about one third. Please keep at least this part of archlinux intact . Arch folks never gave a shit about those things. well until cdrecord.... -- best regards/Mit freundlichen Grüßen Arvid Ephraim Picciani
On Wed, May 21, 2008 at 7:45 PM, Arvid Ephraim Picciani
Yes you are allowed to link dynamicly, now read if you are allowed to run the linked program. Not speaking of actually distributing it.
Strange, why would it be allowed to link and not to run the linked program? It doesn't make sense. -- ------------------------------------------- Denis A. Altoe Falqueto -------------------------------------------
participants (10)
-
Aaron Griffin
-
Allan McRae
-
Arvid Ephraim Picciani
-
Bendany Qian
-
Denis Alessandro Altoe Falqueto
-
Dimitrios Apostolou
-
Jan de Groot
-
Nagy Gabor
-
Tobias Kieslich
-
Xavier