[pacman-dev] [PATCH] makepkg: Fix xz extraction

newer
[pacman-dev] yesno -> noyes for...

older
Re: [pacman-dev] [aur-general] The...

Cedric Staniewski

4 Aug 2009 4 Aug '09

1:24 p.m.

Yesterday, I wanted to use a xz compressed source in a PKGBUILD and had to notice that xz extraction is currently broken in makepkg (not .tar.xz though) even if the file is actually extracted. This is caused by xz's attempt to remove the symlink to the source in the src directory. It fails (because it is a symlink) and therefore xz returns an error code. Luckily, it is fairly simple to fix this by adding an additional flag to the xz extraction command. $ ln -s ../test.xz $ ls test.xz $ xz -d test.xz xz: test.xz: Is a symbolic link, skipping $ echo $? 2 $ ls test.xz $ xz -d -f test.xz xz: test.xz: File seems to be moved, not removing $ echo $? 1 $ ls test test.xz $ rm test $ ls test.xz $ xz -d -f -k test.xz $ echo $? 0 $ ls test test.xz

...

From b2acc55fdf7da81a7581ed10c6351aed234285c5 Mon Sep 17 00:00:00 2001 From: Cedric Staniewski Date: Tue, 4 Aug 2009 15:10:17 +0200 Subject: [PATCH] makepkg: Fix xz extraction

xz cannot remove symlinks and therefore it returns an error code after the (successful) extraction. $ ln -s ../test.xz $ xz -d -f test.xz xz: test.xz: File seems to be moved, not removing $ echo $? 1 $ ls test test.xz $ rm test $ xz -d -f -k test.xz $ echo $? 0 $ ls test test.xz Signed-off-by: Cedric Staniewski --- scripts/makepkg.sh.in | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 56ad2c0..d4b03a0 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -650,7 +650,7 @@ extract_sources() { *application/x-bzip*) cmd="bunzip2 -f" ;; *application/x-xz*) - cmd="xz -d -f" ;; + cmd="xz -d -f -k" ;; *) # Don't know what to use to extract this file, # skip to the next file -- 1.6.4

Show replies by date

Xavier

4 Aug 4 Aug

3:32 p.m.

On Tue, Aug 4, 2009 at 3:24 PM, Cedric Staniewski wrote:

...

Yesterday, I wanted to use a xz compressed source in a PKGBUILD and had to notice that xz extraction is currently broken in makepkg (not .tar.xz though) even if the file is actually extracted. This is caused by xz's attempt to remove the symlink to the source in the src directory. It fails (because it is a symlink) and therefore xz returns an error code. Luckily, it is fairly simple to fix this by adding an additional flag to the xz extraction command.

$ ln -s ../test.xz $ ls test.xz $ xz -d test.xz xz: test.xz: Is a symbolic link, skipping $ echo $? 2 $ ls test.xz $ xz -d -f test.xz xz: test.xz: File seems to be moved, not removing $ echo $? 1 $ ls test test.xz $ rm test $ ls test.xz $ xz -d -f -k test.xz $ echo $? 0 $ ls test test.xz

...
From b2acc55fdf7da81a7581ed10c6351aed234285c5 Mon Sep 17 00:00:00 2001 From: Cedric Staniewski Date: Tue, 4 Aug 2009 15:10:17 +0200 Subject: [PATCH] makepkg: Fix xz extraction

xz cannot remove symlinks and therefore it returns an error code after the (successful) extraction.

$ ln -s ../test.xz $ xz -d -f test.xz xz: test.xz: File seems to be moved, not removing $ echo $? 1 $ ls test test.xz $ rm test $ xz -d -f -k test.xz $ echo $? 0 $ ls test test.xz

Signed-off-by: Cedric Staniewski --- scripts/makepkg.sh.in | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 56ad2c0..d4b03a0 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -650,7 +650,7 @@ extract_sources() { *application/x-bzip*) cmd="bunzip2 -f" ;; *application/x-xz*) - cmd="xz -d -f" ;; + cmd="xz -d -f -k" ;; *) # Don't know what to use to extract this file, # skip to the next file -- 1.6.4 _______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://www.archlinux.org/mailman/listinfo/pacman-dev

But should we remove the symlink manually then?

Cedric Staniewski

5:06 p.m.

Xavier wrote:

...

But should we remove the symlink manually then? _______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://www.archlinux.org/mailman/listinfo/pacman-dev

I'm not sure. Always thought they are there on purpose (mostly used bsdtar), even though they are usually not needful. The current behaviors are: bsdtar keeps symlinks gunzip removes symlinks bunzip2 removes symlinks xz keeps symlinks

Xavier

5:23 p.m.

On Tue, Aug 4, 2009 at 7:06 PM, Cedric Staniewski wrote:

...

Xavier wrote:

...
But should we remove the symlink manually then? _______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://www.archlinux.org/mailman/listinfo/pacman-dev

I'm not sure. Always thought they are there on purpose (mostly used bsdtar), even though they are usually not needful.

The current behaviors are: bsdtar keeps symlinks gunzip removes symlinks bunzip2 removes symlinks xz keeps symlinks

Thank you, thats very informative :) So we were already inconsistent before. What about always keeping symlinks, ie adding -k to gunzip and bunzip2 as well?

Dan McGee

5:35 p.m.

On Tue, Aug 4, 2009 at 12:23 PM, Xavier wrote:

...

On Tue, Aug 4, 2009 at 7:06 PM, Cedric Staniewski wrote:

...
Xavier wrote:

...
But should we remove the symlink manually then? _______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://www.archlinux.org/mailman/listinfo/pacman-dev

I'm not sure. Always thought they are there on purpose (mostly used bsdtar), even though they are usually not needful.

The current behaviors are: bsdtar keeps symlinks gunzip removes symlinks bunzip2 removes symlinks xz keeps symlinks

Thank you, thats very informative :) So we were already inconsistent before. What about always keeping symlinks, ie adding -k to gunzip and bunzip2 as well?

We should always keep, this was probably an oversight when adding direct decompression of non-tar formats. That way someone can very easily see what source was copied in that directory both before and after decompression. -Dan

Cedric Staniewski

9:51 p.m.

Dan McGee wrote:

...

On Tue, Aug 4, 2009 at 12:23 PM, Xavier wrote:

...
On Tue, Aug 4, 2009 at 7:06 PM, Cedric Staniewski wrote:

...
Xavier wrote:

...
But should we remove the symlink manually then? _______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://www.archlinux.org/mailman/listinfo/pacman-dev I'm not sure. Always thought they are there on purpose (mostly used bsdtar), even though they are usually not needful.

The current behaviors are: bsdtar keeps symlinks gunzip removes symlinks bunzip2 removes symlinks xz keeps symlinks Thank you, thats very informative :) So we were already inconsistent before. What about always keeping symlinks, ie adding -k to gunzip and bunzip2 as well?

We should always keep, this was probably an oversight when adding direct decompression of non-tar formats. That way someone can very easily see what source was copied in that directory both before and after decompression.

-Dan _______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://www.archlinux.org/mailman/listinfo/pacman-dev

Unfortunately, gzip/gunzip do not have something like a -k option and it seems that there are no plans to add one [1,2]. The only possibilities are recreating the link or decompressing to stdout (gzip -dc file.gz > file). By the way, another issue might be the use of file, because decompressing via gzip/bzip2/xz without the -c option relies on the suffix. [1] http://www.mail-archive.com/bug-gzip@gnu.org/msg00039.html [2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=240539

Xavier

10:32 p.m.

On Tue, Aug 4, 2009 at 11:51 PM, Cedric Staniewski wrote:

...

Unfortunately, gzip/gunzip do not have something like a -k option and it seems that there are no plans to add one [1,2]. The only possibilities are recreating the link or decompressing to stdout (gzip -dc file.gz > file).

By the way, another issue might be the use of file, because decompressing via gzip/bzip2/xz without the -c option relies on the suffix.

[1] http://www.mail-archive.com/bug-gzip@gnu.org/msg00039.html [2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=240539

yeah, I realized after writing my mail that gzip didn't have -k. crazy.. I wanted to go the lazy way and only fix bzip2 and xz, and not worry about gzip for now. About file type vs file extension : I don't see what we could do if the file extension is wrong. It is not possible to compute a destination file, so we can only fail there. And this is what should happen currently. However, if we want to fix the issue with gzip -dc file.gz > file , we probably need to check the extension, before trying to strip it. And if the extension does not match, we would probably fail here too.

Cedric Staniewski

5 Aug 5 Aug

10:51 a.m.

Xavier wrote:

...

On Tue, Aug 4, 2009 at 11:51 PM, Cedric Staniewski wrote:

...
Unfortunately, gzip/gunzip do not have something like a -k option and it seems that there are no plans to add one [1,2]. The only possibilities are recreating the link or decompressing to stdout (gzip -dc file.gz > file).

By the way, another issue might be the use of file, because decompressing via gzip/bzip2/xz without the -c option relies on the suffix.

[1] http://www.mail-archive.com/bug-gzip@gnu.org/msg00039.html [2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=240539

yeah, I realized after writing my mail that gzip didn't have -k. crazy..

I wanted to go the lazy way and only fix bzip2 and xz, and not worry about gzip for now.

About file type vs file extension : I don't see what we could do if the file extension is wrong. It is not possible to compute a destination file, so we can only fail there. And this is what should happen currently.

However, if we want to fix the issue with gzip -dc file.gz > file , we probably need to check the extension, before trying to strip it. And if the extension does not match, we would probably fail here too. _______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://www.archlinux.org/mailman/listinfo/pacman-dev

I would prefer skipping files with unsuitable extensions rather than failing and forcing to add these files to noextract. A fairly robust patch (gzip suffixes -gz, -z, _z are not supported though) can be found in the following mails. It is based on master, not on the first patch of this thread.

Cedric Staniewski

10:56 a.m.

New subject: [pacman-dev] [PATCH 1/2] makepkg: simplify extraction command selection

Signed-off-by: Cedric Staniewski --- scripts/makepkg.sh.in | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 56ad2c0..9141e1c 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -640,16 +640,16 @@ extract_sources() { fi # fix flyspray #6246 - local file_type=$(file -bizL "$file") + local file_type=$(file -bz --mime-type "$file") local cmd='' case "$file_type" in - *application/x-tar*|*application/zip*|*application/x-zip*|*application/x-cpio*) + application/x-tar|application/zip|application/x-zip|application/x-cpio) cmd="bsdtar -x -f" ;; - *application/x-gzip*) + application/x-gzip) cmd="gunzip -d -f" ;; - *application/x-bzip*) + application/x-bzip) cmd="bunzip2 -f" ;; - *application/x-xz*) + application/x-xz) cmd="xz -d -f" ;; *) # Don't know what to use to extract this file, -- 1.6.4

Cedric Staniewski

10:56 a.m.

New subject: [pacman-dev] [PATCH 2/2] makepkg: always keep sources symlinks

Add -k (keep) option to the bunzip2/xz commands and make gunzip decompressing to stdout, because it does not offer something like a -k option. Additionally the selection of the decompression command for gzip/bzip2/xz compressed files now also depends on the file suffix, since the decompression programs rely on them when not using -c option. Signed-off-by: Cedric Staniewski --- scripts/makepkg.sh.in | 33 +++++++++++++++++++++------------ 1 files changed, 21 insertions(+), 12 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 9141e1c..10a99f0 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -644,22 +644,31 @@ extract_sources() { local cmd='' case "$file_type" in application/x-tar|application/zip|application/x-zip|application/x-cpio) - cmd="bsdtar -x -f" ;; - application/x-gzip) - cmd="gunzip -d -f" ;; - application/x-bzip) - cmd="bunzip2 -f" ;; - application/x-xz) - cmd="xz -d -f" ;; + cmd="bsdtar xf '$file'" ;; *) - # Don't know what to use to extract this file, - # skip to the next file - continue;; + case "$file" in + *.gz|*.z|*.Z) + [ "$file_type" = "application/x-gzip" ] \ + && cmd="gunzip -cf '$file' > '${file%.*}'" + ;; + *.bz2|*.bz) + [ "$file_type" = "application/x-bzip" ] \ + && cmd="bunzip2 -fk '$file'" + ;; + *.xz) + [ "$file_type" = "application/x-xz" ] \ + && cmd="xz -dfk '$file'" + ;; + *) + # Don't know what to use to extract this file, + # skip to the next file + continue ;; + esac esac local ret=0 - msg2 '%s' "$cmd \"$file\"" - $cmd "$file" || ret=$? + msg2 '%s' "$cmd" + $cmd || ret=$? if [ $ret -ne 0 ]; then error "$(gettext "Failed to extract %s")" "$file" plain "$(gettext "Aborting...")" -- 1.6.4

Cedric Staniewski

noon

New subject: [pacman-dev] [PATCH 2/2] makepkg: always keep sources symlinks

This time a more tested patch... Probably someone knows a possibility to quote the file name in the cmd string so that the characters are recognized as quoting chars and not as part of the filename without using eval.

...

From 0aad0c6f6030806cea1804152afc0650d491b307 Mon Sep 17 00:00:00 2001 From: Cedric Staniewski Date: Wed, 5 Aug 2009 12:43:03 +0200 Subject: [PATCH] makepkg: always keep sources symlinks

Add -k (keep) option to the bunzip2/xz commands and make gunzip decompressing to stdout, because it does not offer something like a -k option. Additionally the selection of the decompression command for gzip/bzip2/xz compressed files now also depends on the file suffix, since the decompression programs rely on them when not using -c option. Signed-off-by: Cedric Staniewski --- scripts/makepkg.sh.in | 36 ++++++++++++++++++++++++------------ 1 files changed, 24 insertions(+), 12 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 56ad2c0..c7a6434 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -644,22 +644,34 @@ extract_sources() { local cmd='' case "$file_type" in *application/x-tar*|*application/zip*|*application/x-zip*|*application/x-cpio*) - cmd="bsdtar -x -f" ;; - *application/x-gzip*) - cmd="gunzip -d -f" ;; - *application/x-bzip*) - cmd="bunzip2 -f" ;; - *application/x-xz*) - cmd="xz -d -f" ;; + cmd="bsdtar xf '$file'" ;; *) - # Don't know what to use to extract this file, - # skip to the next file - continue;; + case "$file" in + *.gz|*.z|*.Z) + [[ "$file_type" = *application/x-gzip* ]] \ + && cmd="gunzip -cf '$file' > '${file%.*}'" \ + || continue + ;; + *.bz2|*.bz) + [[ "$file_type" = *application/x-bzip* ]] \ + && cmd="bunzip2 -fk '$file'" \ + || continue + ;; + *.xz) + [[ "$file_type" = *application/x-xz* ]] \ + && cmd="xz -dfk '$file'" \ + || continue + ;; + *) + # Don't know what to use to extract this file, + # skip to the next file + continue ;; + esac esac local ret=0 - msg2 '%s' "$cmd \"$file\"" - $cmd "$file" || ret=$? + msg2 '%s' "$cmd" + eval $cmd || ret=$? if [ $ret -ne 0 ]; then error "$(gettext "Failed to extract %s")" "$file" plain "$(gettext "Aborting...")" -- 1.6.4

Dan McGee

12:10 p.m.

New subject: [pacman-dev] [PATCH 2/2] makepkg: always keep sources symlinks

On Wed, Aug 5, 2009 at 7:00 AM, Cedric Staniewski wrote:

...

This time a more tested patch... Probably someone knows a possibility to quote the file name in the cmd string so that the characters are recognized as quoting chars and not as part of the filename without using eval.

...
From 0aad0c6f6030806cea1804152afc0650d491b307 Mon Sep 17 00:00:00 2001 From: Cedric Staniewski Date: Wed, 5 Aug 2009 12:43:03 +0200 Subject: [PATCH] makepkg: always keep sources symlinks

Add -k (keep) option to the bunzip2/xz commands and make gunzip decompressing to stdout, because it does not offer something like a -k option.

Additionally the selection of the decompression command for gzip/bzip2/xz compressed files now also depends on the file suffix, since the decompression programs rely on them when not using -c option.

Signed-off-by: Cedric Staniewski --- scripts/makepkg.sh.in | 36 ++++++++++++++++++++++++------------ 1 files changed, 24 insertions(+), 12 deletions(-)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 56ad2c0..c7a6434 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -644,22 +644,34 @@ extract_sources() { local cmd='' case "$file_type" in *application/x-tar*|*application/zip*|*application/x-zip*|*application/x-cpio*) - cmd="bsdtar -x -f" ;; - *application/x-gzip*) - cmd="gunzip -d -f" ;; - *application/x-bzip*) - cmd="bunzip2 -f" ;; - *application/x-xz*) - cmd="xz -d -f" ;; + cmd="bsdtar xf '$file'" ;; *) - # Don't know what to use to extract this file, - # skip to the next file - continue;; + case "$file" in + *.gz|*.z|*.Z) + [[ "$file_type" = *application/x-gzip* ]] \ + && cmd="gunzip -cf '$file' > '${file%.*}'" \ + || continue + ;; + *.bz2|*.bz) + [[ "$file_type" = *application/x-bzip* ]] \ + && cmd="bunzip2 -fk '$file'" \ + || continue + ;; + *.xz) + [[ "$file_type" = *application/x-xz* ]] \ + && cmd="xz -dfk '$file'" \ + || continue + ;;

This patch just makes me cry a little bit...its like taking a sledgehammer to kill a fly, unfortunately. All we wanted to do was keep a symlink around, and we've now regressed to using extensions, which we haven't done since pre 3.1.0. On a side note, if anyone can figure out how to annotate/blame an old version of a file that no longer exists (as it has been renamed), please let me know. git might be failing here: $ git blame scripts/makepkg.in e19d7da4~1 fatal: cannot stat path 'scripts/makepkg.in': No such file or directory $ git annotate --follow "scripts/makepkg.sh.in" e19d7da4~1 fatal: no such path scripts/makepkg.sh.in in e19d7da4~1 -Dan

Dan McGee

1:59 p.m.

New subject: [pacman-dev] [PATCH 2/2] makepkg: always keep sources symlinks

On Wed, Aug 5, 2009 at 7:10 AM, Dan McGee wrote:

...

On Wed, Aug 5, 2009 at 7:00 AM, Cedric Staniewski wrote:

...
This time a more tested patch... Probably someone knows a possibility to quote the file name in the cmd string so that the characters are recognized as quoting chars and not as part of the filename without using eval.

...
From 0aad0c6f6030806cea1804152afc0650d491b307 Mon Sep 17 00:00:00 2001 From: Cedric Staniewski Date: Wed, 5 Aug 2009 12:43:03 +0200 Subject: [PATCH] makepkg: always keep sources symlinks

Add -k (keep) option to the bunzip2/xz commands and make gunzip decompressing to stdout, because it does not offer something like a -k option.

Additionally the selection of the decompression command for gzip/bzip2/xz compressed files now also depends on the file suffix, since the decompression programs rely on them when not using -c option.

Signed-off-by: Cedric Staniewski --- scripts/makepkg.sh.in | 36 ++++++++++++++++++++++++------------ 1 files changed, 24 insertions(+), 12 deletions(-)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 56ad2c0..c7a6434 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -644,22 +644,34 @@ extract_sources() { local cmd='' case "$file_type" in *application/x-tar*|*application/zip*|*application/x-zip*|*application/x-cpio*) - cmd="bsdtar -x -f" ;; - *application/x-gzip*) - cmd="gunzip -d -f" ;; - *application/x-bzip*) - cmd="bunzip2 -f" ;; - *application/x-xz*) - cmd="xz -d -f" ;; + cmd="bsdtar xf '$file'" ;; *) - # Don't know what to use to extract this file, - # skip to the next file - continue;; + case "$file" in + *.gz|*.z|*.Z) + [[ "$file_type" = *application/x-gzip* ]] \ + && cmd="gunzip -cf '$file' > '${file%.*}'" \ + || continue + ;; + *.bz2|*.bz) + [[ "$file_type" = *application/x-bzip* ]] \ + && cmd="bunzip2 -fk '$file'" \ + || continue + ;; + *.xz) + [[ "$file_type" = *application/x-xz* ]] \ + && cmd="xz -dfk '$file'" \ + || continue + ;;

This patch just makes me cry a little bit...its like taking a sledgehammer to kill a fly, unfortunately. All we wanted to do was keep a symlink around, and we've now regressed to using extensions, which we haven't done since pre 3.1.0.

(See below so it flows better for a better response on this)

...

On a side note, if anyone can figure out how to annotate/blame an old version of a file that no longer exists (as it has been renamed), please let me know. git might be failing here:

$ git blame scripts/makepkg.in e19d7da4~1 fatal: cannot stat path 'scripts/makepkg.in': No such file or directory

$ git annotate --follow "scripts/makepkg.sh.in" e19d7da4~1 fatal: no such path scripts/makepkg.sh.in in e19d7da4~1

I found what I wanted with the following, thanks to the GIT ML: $ git blame e19d7~1 -- scripts/makepkg.in which leads to this commit: $ git show 7ed7977e which changed from extension-based to mime-type based to fix FS#6246, http://bugs.archlinux.org/task/6246, so I'd rather not regress something we fixed looooong ago. -Dan

Cedric Staniewski

2:12 p.m.

New subject: [pacman-dev] [PATCH 2/2] makepkg: always keep sources symlinks

Dan McGee wrote:

...

This patch just makes me cry a little bit...its like taking a sledgehammer to kill a fly, unfortunately. All we wanted to do was keep a symlink around, and we've now regressed to using extensions, which we haven't done since pre 3.1.0.

The extension part is only needed here because gzip/bzip2/xz rely on the correct extension. Probably I should have split the patch into two.

...

$ touch test && xz test && mv test.xz test.abc && file -i test.abc && xz -d test.abc; echo $? `test.xz' -> `test.abc' test.abc: application/x-xz; charset=binary xz: test.abc: Filename has an unknown suffix, skipping 2

I see three options to address this issue: - do not handle it at all; just throw an error and require a noextract entry - test for the mime header AND the extension - always decompress to stdout and write it to 'file - extension', e.g. bzip2 -dcf test.abc > test -> this will not work if there is no extension -> throws an error; noextract entry required Possibly, option 1 is fine as it might not be the common case that the compressed files use unsuitable extensions.

Allan McRae

12:48 p.m.

New subject: [pacman-dev] [PATCH 2/2] makepkg: always keep sources symlinks

Cedric Staniewski wrote:

...

This time a more tested patch... Probably someone knows a possibility to quote the file name in the cmd string so that the characters are recognized as quoting chars and not as part of the filename without using eval.

...
From 0aad0c6f6030806cea1804152afc0650d491b307 Mon Sep 17 00:00:00 2001 From: Cedric Staniewski Date: Wed, 5 Aug 2009 12:43:03 +0200 Subject: [PATCH] makepkg: always keep sources symlinks

Add -k (keep) option to the bunzip2/xz commands and make gunzip decompressing to stdout, because it does not offer something like a -k option.

Additionally the selection of the decompression command for gzip/bzip2/xz compressed files now also depends on the file suffix, since the decompression programs rely on them when not using -c option.

Signed-off-by: Cedric Staniewski --- scripts/makepkg.sh.in | 36 ++++++++++++++++++++++++------------ 1 files changed, 24 insertions(+), 12 deletions(-)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 56ad2c0..c7a6434 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -644,22 +644,34 @@ extract_sources() { local cmd='' case "$file_type" in *application/x-tar*|*application/zip*|*application/x-zip*|*application/x-cpio*) - cmd="bsdtar -x -f" ;; - *application/x-gzip*) - cmd="gunzip -d -f" ;; - *application/x-bzip*) - cmd="bunzip2 -f" ;; - *application/x-xz*) - cmd="xz -d -f" ;; + cmd="bsdtar xf '$file'" ;; *) - # Don't know what to use to extract this file, - # skip to the next file - continue;; + case "$file" in + *.gz|*.z|*.Z) + [[ "$file_type" = *application/x-gzip* ]] \ + && cmd="gunzip -cf '$file' > '${file%.*}'" \ + || continue + ;; + *.bz2|*.bz) + [[ "$file_type" = *application/x-bzip* ]] \ + && cmd="bunzip2 -fk '$file'" \ + || continue + ;; + *.xz) + [[ "$file_type" = *application/x-xz* ]] \ + && cmd="xz -dfk '$file'" \ + || continue + ;; + *) + # Don't know what to use to extract this file, + # skip to the next file + continue ;; + esac esac

local ret=0 - msg2 '%s' "$cmd \"$file\"" - $cmd "$file" || ret=$? + msg2 '%s' "$cmd" + eval $cmd || ret=$? if [ $ret -ne 0 ]; then error "$(gettext "Failed to extract %s")" "$file" plain "$(gettext "Aborting...")"

I'd really prefer this whole thing to look like: ext=${file/*./} case "$file_type" in *application/x-tar*|*application/zip*|*application/x-zip*|*application/x-cpio*) cmd="bsdtar xf '$file'" ;; *application/x-gzip*) [ "$ext" == ".gz" -o "$ext" == ".z" -o "$ext" == ".Z" ] && cmd="gunzip -d -f '$file' or something like that... it is much cleaner. Allan

Cedric Staniewski

2:12 p.m.

New subject: [pacman-dev] [PATCH 1/2] makepkg: always keep sources symlinks

Add -k (keep) option to the bunzip2/xz commands and make gunzip decompressing to stdout, because it does not offer something like a -k option. Signed-off-by: Cedric Staniewski --- scripts/makepkg.sh.in | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 56ad2c0..2f1db38 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -644,13 +644,13 @@ extract_sources() { local cmd='' case "$file_type" in *application/x-tar*|*application/zip*|*application/x-zip*|*application/x-cpio*) - cmd="bsdtar -x -f" ;; + cmd="bsdtar -x -f '$file'" ;; *application/x-gzip*) - cmd="gunzip -d -f" ;; + cmd="gunzip -c -f '$file' > '${file%.*}'" ;; *application/x-bzip*) - cmd="bunzip2 -f" ;; + cmd="bunzip2 -f -k '$file'" ;; *application/x-xz*) - cmd="xz -d -f" ;; + cmd="xz -d -f -k '$file'" ;; *) # Don't know what to use to extract this file, # skip to the next file @@ -658,8 +658,8 @@ extract_sources() { esac local ret=0 - msg2 '%s' "$cmd \"$file\"" - $cmd "$file" || ret=$? + msg2 '%s' "$cmd" + eval $cmd || ret=$? if [ $ret -ne 0 ]; then error "$(gettext "Failed to extract %s")" "$file" plain "$(gettext "Aborting...")" -- 1.6.4

Cedric Staniewski

2:12 p.m.

New subject: [pacman-dev] [PATCH 2/2] makepkg: test for appropriate file suffixes when extracting sources

The selection of the decompression command for gzip/bzip2/xz compressed files now also depends on the file suffix, since the decompression programs rely on them when not using -c option. Signed-off-by: Cedric Staniewski --- scripts/makepkg.sh.in | 13 ++++++++++--- 1 files changed, 10 insertions(+), 3 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 2f1db38..702e4c1 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -641,16 +641,23 @@ extract_sources() { # fix flyspray #6246 local file_type=$(file -bizL "$file") + local ext=${file##*.} local cmd='' case "$file_type" in *application/x-tar*|*application/zip*|*application/x-zip*|*application/x-cpio*) cmd="bsdtar -x -f '$file'" ;; *application/x-gzip*) - cmd="gunzip -c -f '$file' > '${file%.*}'" ;; + [ "$ext" = "gz" -o "$ext" = "z" -o "$ext" = "Z" ] \ + && cmd="gunzip -cf '$file' > '${file%.*}'" \ + || continue ;; *application/x-bzip*) - cmd="bunzip2 -f -k '$file'" ;; + [ "$ext" = "bz2" -o "$ext" = "bz" ] \ + && cmd="bunzip2 -fk '$file'" \ + || continue ;; *application/x-xz*) - cmd="xz -d -f -k '$file'" ;; + [ "$ext" = "xz" ] \ + && cmd="xz -dfk '$file'" \ + || continue ;; *) # Don't know what to use to extract this file, # skip to the next file -- 1.6.4

Xavier

2:30 p.m.

New subject: [pacman-dev] [PATCH 2/2] makepkg: test for appropriate file suffixes when extracting sources

On Wed, Aug 5, 2009 at 4:12 PM, Cedric Staniewski wrote:

...

The selection of the decompression command for gzip/bzip2/xz compressed files now also depends on the file suffix, since the decompression programs rely on them when not using -c option.

Signed-off-by: Cedric Staniewski --- scripts/makepkg.sh.in | 13 ++++++++++--- 1 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 2f1db38..702e4c1 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -641,16 +641,23 @@ extract_sources() {

# fix flyspray #6246 local file_type=$(file -bizL "$file") + local ext=${file##*.} local cmd='' case "$file_type" in *application/x-tar*|*application/zip*|*application/x-zip*|*application/x-cpio*) cmd="bsdtar -x -f '$file'" ;; *application/x-gzip*) - cmd="gunzip -c -f '$file' > '${file%.*}'" ;; + [ "$ext" = "gz" -o "$ext" = "z" -o "$ext" = "Z" ] \ + && cmd="gunzip -cf '$file' > '${file%.*}'" \ + || continue ;; *application/x-bzip*) - cmd="bunzip2 -f -k '$file'" ;; + [ "$ext" = "bz2" -o "$ext" = "bz" ] \ + && cmd="bunzip2 -fk '$file'" \ + || continue ;; *application/x-xz*) - cmd="xz -d -f -k '$file'" ;; + [ "$ext" = "xz" ] \ + && cmd="xz -dfk '$file'" \ + || continue ;; *) # Don't know what to use to extract this file, # skip to the next file -- 1.6.4

an example of a compressed file with a different extension which came to my mind : http://en.wikipedia.org/wiki/PK3_%28file_extension%29 So without this patch, makepkg would just error out, unless we put the file in noextract With this patch, it is just skipped. This situation is probably quite rare, but I am totally fine with this patch.

Cedric Staniewski

11:44 a.m.

New subject: [pacman-dev] [PATCH 1/2] makepkg: simplify extraction command selection

Forget about this patch. It breaks too much because the string which is matched in the case construct is actually not always the MIME header.

Dan McGee

11:55 a.m.

New subject: [pacman-dev] [PATCH 1/2] makepkg: simplify extraction command selection

On Wed, Aug 5, 2009 at 6:44 AM, Cedric Staniewski wrote:

...

Forget about this patch. It breaks too much because the string which is matched in the case construct is actually not always the MIME header.

Yeah, I feel like we did the globbing on purpose originally because file doesn't always spit you back what you want (especially when looking at zipped files, you sometimes get two types). -Dan

5377

Age (days ago)

5378

Last active (days ago)

List overview

Download

19 comments

4 participants

participants (4)

Allan McRae
Cedric Staniewski
Dan McGee
Xavier