[arch-general] Adopt tar.xz for official repo packaging ?
Corrently Archlinux uses a tar+gzip combination and it works nicely. But right now there's a stable lzma compression available using "tar Jcvf", which could create much smaller packages and still fast decompressions. (of course you need to have xz-utils) think about all the bandwidth that could be saved ! my little experiment shows kernel26-2.6.30-5-x86_64.pkg.tar.gz coreutils-7.4-1-x86_64.pkg.tar.gz tar.gz 35338866 (100%) 4853937 (100%) tar.xz 24892732 (70.4%) 1843592 (38.2%) i guess one of the major concerns is that tar & gzip is more common than xz. but since GNU is also packing their coreutils in xz packs, i guess that a good time to switch?
Excerpts from b4283's message of Fri Jul 03 06:32:14 -0400 2009:
i guess one of the major concerns is that tar & gzip is more common than xz. but since GNU is also packing their coreutils in xz packs, i guess that a good time to switch?
Well, one nice thing about being a distro is that you have some ability to choose what stuff the users will have on their computers :) And you really raise a solid argument in my view. That's like half the size! Does compression/decompression take longer? Are there any other bad parts to using this compression algorithm that are worth mentioning aside from lack of popularity? (Note, isn't even bzip better at filesize than gzip?) -- Andrei Thorp, Developer: Xandros Corp. (http://www.xandros.com)
On Fri, 03 Jul 2009 09:24:32 -0400 Andrei Thorp <garoth@gmail.com> wrote:
Excerpts from b4283's message of Fri Jul 03 06:32:14 -0400 2009:
i guess one of the major concerns is that tar & gzip is more common than xz. but since GNU is also packing their coreutils in xz packs, i guess that a good time to switch?
Well, one nice thing about being a distro is that you have some ability to choose what stuff the users will have on their computers :)
And you really raise a solid argument in my view. That's like half the size!
Does compression/decompression take longer? Are there any other bad parts to using this compression algorithm that are worth mentioning aside from lack of popularity?
(Note, isn't even bzip better at filesize than gzip?)
read through the archived posts, i'm sure this was discussed already .. iirc the issue was with implementing this in libarchive and some licensing issues.
Excerpts from Nathan K. Bathory's message of Fri Jul 03 09:28:22 -0400 2009:
On Fri, 03 Jul 2009 09:24:32 -0400 Andrei Thorp <garoth@gmail.com> wrote:
Excerpts from b4283's message of Fri Jul 03 06:32:14 -0400 2009:
i guess one of the major concerns is that tar & gzip is more common than xz. but since GNU is also packing their coreutils in xz packs, i guess that a good time to switch?
Well, one nice thing about being a distro is that you have some ability to choose what stuff the users will have on their computers :)
And you really raise a solid argument in my view. That's like half the size!
Does compression/decompression take longer? Are there any other bad parts to using this compression algorithm that are worth mentioning aside from lack of popularity?
(Note, isn't even bzip better at filesize than gzip?)
read through the archived posts, i'm sure this was discussed already .. iirc the issue was with implementing this in libarchive and some licensing issues.
Hmm, that's unfortunate. Though cutting bandwidth is similar to cutting costs and maybe even decreasing throttling on the servers, no? Would be interesting to do. Another possible issue is the question of whether this sort of compression works as well for the deltas system in pacman. -- Andrei Thorp, Developer: Xandros Corp. (http://www.xandros.com)
On Fri, Jul 3, 2009 at 3:33 PM, Andrei Thorp<garoth@gmail.com> wrote:
Another possible issue is the question of whether this sort of compression works as well for the deltas system in pacman.
This only depends on the following trivial patch to xdelta3 (add xz support) : http://code.google.com/p/xdelta/issues/detail?id=87 deltas are useless if the compression used is not recognized (and thus without uncompress / recompress done automatically).
Excerpts from Xavier's message of Fri Jul 03 10:30:32 -0400 2009:
On Fri, Jul 3, 2009 at 3:33 PM, Andrei Thorp<garoth@gmail.com> wrote:
Another possible issue is the question of whether this sort of compression works as well for the deltas system in pacman.
This only depends on the following trivial patch to xdelta3 (add xz support) : http://code.google.com/p/xdelta/issues/detail?id=87
deltas are useless if the compression used is not recognized (and thus without uncompress / recompress done automatically).
Alright, awesome :) -- Andrei Thorp, Developer: Xandros Corp. (http://www.xandros.com)
On Friday 03 July 2009 15:28:22 Nathan K. Bathory wrote:
read through the archived posts, i'm sure this was discussed already .. iirc the issue was with implementing this in libarchive and some licensing issues.
pacman can already handle xz compressed packages (using libarchive). Support for makepkg will be available with pacman 3.3. (which does not mean we will use it though) -- Pierre Schmitz, http://users.archlinux.de/~pierre
Nathan K. Bathory schrieb:
read through the archived posts, i'm sure this was discussed already .. iirc the issue was with implementing this in libarchive and some licensing issues.
Actually, pacman does support xz and lzma decompression (except one lzma/xz-related bug in libarchive which will be fixed in the 2.7.1 release) and I think makepkg either supports it or it is trivial to implement. The same goes for bzip2. Our db scripts and devtools always assume .tar.gz though, this is also easy to change. Anyway, pacman doesn't care, you can have a wild mix of bzip2, gzip, xz and lzma-compressed packages (and compress the db.tar.gz the way you want) and it'll still work.
On Friday 03 July 2009 16:02:38 Thomas Bächler wrote:
Our db scripts and devtools always assume .tar.gz though, this is also easy to change.
Patches are already in git. But atm it does only support one compression method at once. So we either need a patch to allow both during transition or we recompress the whole repo... -- Pierre Schmitz, http://users.archlinux.de/~pierre
Pierre Schmitz wrote:
On Friday 03 July 2009 16:02:38 Thomas Bächler wrote:
Our db scripts and devtools always assume .tar.gz though, this is also easy to change.
Patches are already in git. But atm it does only support one compression method at once. So we either need a patch to allow both during transition or we recompress the whole repo...
Needs to be both as we would want to keep pacman + libarchive + libfetch as .tar.gz for a while Allan
Pierre Schmitz schrieb:
On Friday 03 July 2009 16:02:38 Thomas Bächler wrote:
Our db scripts and devtools always assume .tar.gz though, this is also easy to change.
Patches are already in git. But atm it does only support one compression method at once. So we either need a patch to allow both during transition or we recompress the whole repo...
devtools and db scripts should accept whatever resides in staging IMO, regardless of the compression. So if Andy (for example) wants smaller OOo packages, he simply sets makepkg to use .xz, uploads it and devtools adds the .xz package (not caring that the rest of the repo is gz, I think repo-add should have no problem with that). This way any developer could choose if (de)compression time or size is more important and we would have a very smooth transition (if we were to completely switch to gz).
On Friday 03 July 2009 16:28:13 Thomas Bächler wrote:
devtools and db scripts should accept whatever resides in staging IMO, regardless of the compression. So if Andy (for example) wants smaller OOo packages, he simply sets makepkg to use .xz, uploads it and devtools adds the .xz package (not caring that the rest of the repo is gz, I think repo-add should have no problem with that). This way any developer could choose if (de)compression time or size is more important and we would have a very smooth transition (if we were to completely switch to gz).
Afaik the pkgext is often cut to get the pkgname. For split packages we need to get this info from .PKGINFO. From this point on the filename or compression method does not matter anymore. -- Pierre Schmitz, http://users.archlinux.de/~pierre
participants (7)
-
Allan McRae
-
Andrei Thorp
-
b4283
-
Nathan K. Bathory
-
Pierre Schmitz
-
Thomas Bächler
-
Xavier