[pacman-dev] [PATCH] Add support for the xz archive format
This simple patch adds support for the xz archive format to makepkg and repo- add. Xz can be used as source, package and package db file type. Signed-off-by: Pierre Schmitz <pierre@archlinux.de> --- scripts/makepkg.sh.in | 5 +++++ scripts/repo-add.sh.in | 1 + 2 files changed, 6 insertions(+), 0 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index d978107..cc59f1b 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -648,6 +648,8 @@ extract_sources() { cmd="gunzip -d -f $file" ;; *application/x-bzip*) cmd="bunzip2 -f $file" ;; + *application/x-xz*) + cmd="xz -d -f $file" ;; *) # Don't know what to use to extract this file, # skip to the next file @@ -979,6 +981,7 @@ create_package() { case "$PKGEXT" in *tar.gz) EXT=${PKGEXT%.gz} ;; *tar.bz2) EXT=${PKGEXT%.bz2} ;; + *tar.xz) EXT=${PKGEXT%.xz} ;; *) warning "$(gettext "'%s' is not a valid archive extension.")" \ "$PKGEXT" ; EXT=$PKGEXT ;; esac @@ -996,6 +999,7 @@ create_package() { case "$PKGEXT" in *tar.gz) gzip -f -n "$pkg_file" ;; *tar.bz2) bzip2 -f "$pkg_file" ;; + *tar.xz) xz -z -f "$pkg_file" ;; esac ret=$? fi @@ -1054,6 +1058,7 @@ create_srcpackage() { case "$SRCEXT" in *tar.gz) TAR_OPT="z" ;; *tar.bz2) TAR_OPT="j" ;; + *tar.xz) TAR_OPT="J" ;; *) warning "$(gettext "'%s' is not a valid archive extension.")" \ "$SRCEXT" ;; esac diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index 2390a92..bbd41a2 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -462,6 +462,7 @@ if [ $success -eq 1 ]; then case "$REPO_DB_FILE" in *tar.gz) TAR_OPT="z" ;; *tar.bz2) TAR_OPT="j" ;; + *tar.xz) TAR_OPT="J" ;; *) warning "$(gettext "'%s' does not have a valid archive extension.")" \ "$REPO_DB_FILE" ;; esac -- 1.6.3 -- Pierre Schmitz Clemens-August-Straße 76 53115 Bonn Telefon 0228 9716608 Mobil 0160 95269831 Jabber pierre@jabber.archlinux.de WWW http://www.archlinux.de
Am Mittwoch, 13. Mai 2009 08:21:11 schrieb Pierre Schmitz:
This simple patch adds support for the xz archive format to makepkg and repo- add. Xz can be used as source, package and package db file type.
I did not test this patch at all, but it should work. You'll need xz-utils and recompile libarchive. I did not check acman itself, but it should support it out-of-the-box, right? Next step will be adding support for xz in our devtools. -- Pierre Schmitz Clemens-August-Straße 76 53115 Bonn Telefon 0228 9716608 Mobil 0160 95269831 Jabber pierre@jabber.archlinux.de WWW http://www.archlinux.de
On Wed, May 13, 2009 at 8:23 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
Am Mittwoch, 13. Mai 2009 08:21:11 schrieb Pierre Schmitz:
This simple patch adds support for the xz archive format to makepkg and repo- add. Xz can be used as source, package and package db file type.
I did not test this patch at all, but it should work. You'll need xz-utils and recompile libarchive.
I did not check acman itself, but it should support it out-of-the-box, right?
Next step will be adding support for xz in our devtools.
We may have a problem with the xdelta code, but maybe not. For being useful, xdelta needs to decompress package first before computing a diff, so it needs to support and recognize the compression used for packages. http://code.google.com/p/xdelta/wiki/ExternalCompression It supports gzip and bzip2 but apparently not xz. When applying a xdelta diff, we had a second problem that xdelta didn't apply the same gzip timestamp, so we handle the recompression of a package generated by xdelta manually. http://projects.archlinux.org/?p=pacman.git;a=blob;f=lib/libalpm/sync.c;h=1d... However, since xdelta probably does not recognize xz, probably the -R argument for disabling external recompression has no effect, and in this case, the current pacman code should be fine. To sum up the problems I expect : * xdelta will be inefficient with xz packages since it does not recognize it * when it recognizes it, the sync.c code will need to be updated.
Am Mittwoch, 13. Mai 2009 10:00:55 schrieb Xavier:
To sum up the problems I expect : * xdelta will be inefficient with xz packages since it does not recognize it * when it recognizes it, the sync.c code will need to be updated.
I did not have a closer look at pacman's code but can't we just uncompress the packages ourself and let xdelta handle plain tar files? -- Pierre Schmitz Clemens-August-Straße 76 53115 Bonn Telefon 0228 9716608 Mobil 0160 95269831 Jabber pierre@jabber.archlinux.de WWW http://www.archlinux.de
On Wed, May 13, 2009 at 12:43 PM, Pierre Schmitz <pierre@archlinux.de> wrote:
Am Mittwoch, 13. Mai 2009 10:00:55 schrieb Xavier:
To sum up the problems I expect : * xdelta will be inefficient with xz packages since it does not recognize it * when it recognizes it, the sync.c code will need to be updated.
I did not have a closer look at pacman's code but can't we just uncompress the packages ourself and let xdelta handle plain tar files?
This was one of the available option. Maybe we didn't go for it because it requires slightly more work. But it may be the best option after all. I first thought we would just have to change xdelta generation in create-xdelta and xdelta patching in libalpm/sync.c , but I just remembered I used the xdelta meta-info in repo-add : http://projects.archlinux.org/?p=pacman.git;a=blob;f=scripts/repo-add.sh.in;... So I use xdelta to get the original package file and destination package file from a .delta file, to add this information to pacman database. However, if we do delta between .tar files, the origin and destination will also be plain tar. But the packages in the file cache are not plain tar but compressed tar. So this requires more thinking and attention.
On Wed, May 13, 2009 at 10:00 AM, Xavier <shiningxc@gmail.com> wrote:
To sum up the problems I expect : * xdelta will be inefficient with xz packages since it does not recognize it * when it recognizes it, the sync.c code will need to be updated.
The good part is that it is very easy to add xz support to xdelta, just a one line patch : http://bbs.archlinux.org/viewtopic.php?pid=552545#p552545 Also we do not have the same problem than with gzip timestamps. the md5sum of the patched file is identical without any tricks. I wonder about bzip2. In any cases, the sync.c code will needs to be patched, and maybe it is possible to do it now by reworking the code a bit, to support both the current and future situation.
On Wed, May 13, 2009 at 1:23 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
Am Mittwoch, 13. Mai 2009 08:21:11 schrieb Pierre Schmitz:
This simple patch adds support for the xz archive format to makepkg and repo- add. Xz can be used as source, package and package db file type.
I did not test this patch at all, but it should work. You'll need xz-utils and recompile libarchive.
I did not check acman itself, but it should support it out-of-the-box, right?
Next step will be adding support for xz in our devtools.
Applied, thanks. We still have some concerns with the delta stuff but no need to hold this work up. -Dan
participants (3)
-
Dan McGee
-
Pierre Schmitz
-
Xavier