[pacman-dev] [PATCH 1/3] makepkg: refactor archive compression for reusability
This allows for more easily extending the list of allowed compression methods, as it has to be modified in only one place. Also allow the user to specify their own preferred command + options for source packages in addition to compiled packages. Currently, makepkg.conf(5) erroneously claims this is already possible. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- I'm not 100% sure how to handle this, perhaps it should encompass bsdtar/$pkg_file as well? scripts/makepkg.sh.in | 57 ++++++++++++++++++++++++++------------------------- 1 file changed, 29 insertions(+), 28 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 20e9dd7e..a5c216dd 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -698,6 +698,26 @@ list_package_files() { sed -e 's|^\./||' | tr '\n' '\0' } +# Wrapper around many stream compression formats, for use in the middle of a +# pipeline. A tar archive is passed on stdin and compressed to stdout. +compress_as() { + # $1: final archive filename extension for compression type detection + + local ext="$1" + + case "$ext" in + *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; + *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; + *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; + *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; + *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; + *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; + *tar) cat ;; + *) warning "$(gettext "'%s' is not a valid archive extension.")" \ + "$ext"; cat ;; + esac +} + create_package() { (( NOARCHIVE )) && return @@ -748,21 +768,8 @@ create_package() { msg2 "$(gettext "Compressing package...")" # TODO: Maybe this can be set globally for robustness shopt -s -o pipefail - # bsdtar's gzip compression always saves the time stamp, making one - # archive created using the same command line distinct from another. - # Disable bsdtar compression and use gzip -n for now. list_package_files | LANG=C bsdtar -cnf - --null --files-from - | - case "$PKGEXT" in - *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; - *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; - *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; - *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; - *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; - *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; - *tar) cat ;; - *) warning "$(gettext "'%s' is not a valid archive extension.")" \ - "$PKGEXT"; cat ;; - esac > "${pkg_file}" || ret=$? + compress_as "$PKGEXT" > "${pkg_file}" || ret=$? shopt -u -o pipefail @@ -843,26 +850,20 @@ create_srcpackage() { done done - local TAR_OPT - case "$SRCEXT" in - *tar.gz) TAR_OPT="-z" ;; - *tar.bz2) TAR_OPT="-j" ;; - *tar.xz) TAR_OPT="-J" ;; - *tar.lrz) TAR_OPT="--lrzip" ;; - *tar.lzo) TAR_OPT="--lzop" ;; - *tar.Z) TAR_OPT="-Z" ;; - *tar) TAR_OPT="" ;; - *) warning "$(gettext "'%s' is not a valid archive extension.")" \ - "$SRCEXT" ;; - esac - local fullver=$(get_full_version) local pkg_file="$SRCPKGDEST/${pkgbase}-${fullver}${SRCEXT}" # tar it up msg2 "$(gettext "Compressing source package...")" cd_safe "${srclinks}" - if ! LANG=C bsdtar -cL ${TAR_OPT} -f "$pkg_file" ${pkgbase}; then + + # TODO: Maybe this can be set globally for robustness + shopt -s -o pipefail + LANG=C bsdtar -cLf - ${pkgbase} | compress_as "$SRCEXT" > "${pkg_file}" || ret=$? + + shopt -u -o pipefail + + if (( ret )); then error "$(gettext "Failed to create source package file.")" exit 1 # TODO: error code fi -- 2.14.1
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- This shouldn't be added until we are sure pacman actually knows how to read .zst files, which requires libarchive support that is currently only in git master. I have no idea when either libarchive or pacman plan to make a new release. Though at least this isn't the default, and would require explicit opt-in... doc/makepkg.conf.5.txt | 3 ++- etc/makepkg.conf.in | 1 + scripts/makepkg.sh.in | 1 + 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/makepkg.conf.5.txt b/doc/makepkg.conf.5.txt index 5bd3472f..d01c13b6 100644 --- a/doc/makepkg.conf.5.txt +++ b/doc/makepkg.conf.5.txt @@ -246,6 +246,7 @@ Options **COMPRESSGZ=**"(gzip -c -f -n)":: **COMPRESSBZ2=**"(bzip2 -c -f)":: **COMPRESSXZ=**"(xz -c -z -)":: +**COMPRESSZST=**"(zstd -c -z -)":: **COMPRESSLZO**"(lzop -q)":: **COMPRESSLRZ=**"(lrzip -q)":: **COMPRESSZ=**"(compress -c -f)":: @@ -254,7 +255,7 @@ Options **PKGEXT=**".pkg.tar.gz", **SRCEXT=**".src.tar.gz":: Sets the compression used when making compiled or source packages. - Valid suffixes are `.tar`, `.tar.gz`, `.tar.bz2`, `.tar.xz`, + Valid suffixes are `.tar`, `.tar.gz`, `.tar.bz2`, `.tar.xz`, `.tar.zst`, `.tar.lzo`, `.tar.lrz`, and `.tar.Z`. Do not touch these unless you know what you are doing. diff --git a/etc/makepkg.conf.in b/etc/makepkg.conf.in index 71293970..4787855c 100644 --- a/etc/makepkg.conf.in +++ b/etc/makepkg.conf.in @@ -126,6 +126,7 @@ PURGE_TARGETS=(usr/{,share}/info/dir .packlist *.pod) COMPRESSGZ=(gzip -c -f -n) COMPRESSBZ2=(bzip2 -c -f) COMPRESSXZ=(xz -c -z -) +COMPRESSXZ=(zstd -c -z -q -) COMPRESSLRZ=(lrzip -q) COMPRESSLZO=(lzop -q) COMPRESSZ=(compress -c -f) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index a5c216dd..d9b5ea2d 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -709,6 +709,7 @@ compress_as() { *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; + *tar.zst) ${COMPRESSZST[@]:-zstd -c -z -q -} ;; *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; -- 2.14.1
On 29/08/17 15:01, Eli Schwartz wrote:
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> ---
This shouldn't be added until we are sure pacman actually knows how to read .zst files, which requires libarchive support that is currently only in git master.
I have no idea when either libarchive or pacman plan to make a new release. Though at least this isn't the default, and would require explicit opt-in...
Hrm... if this was C code, we could easily configure test this and #ifdef it. Not sure how we should handle this in bash code. Suggestions welcome. This may also apply to some other formats we support, although I think we have a fairly stringent libarchive version requirement anyway. A
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- Unlike makepkg, this is implemented as a bsdtar option which means we *need* libarchive support just to create a database, never mind whether pacman can read it. scripts/repo-add.sh.in | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index 68edbf4b..ccc0c89b 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -300,6 +300,7 @@ verify_repo_extension() { *.db.tar.gz) TAR_OPT="z" ;; *.db.tar.bz2) TAR_OPT="j" ;; *.db.tar.xz) TAR_OPT="J" ;; + *.db.tar.zst) TAR_OPT="--zstd" ;; *.db.tar.Z) TAR_OPT="Z" ;; *.db.tar) TAR_OPT="" ;; *) error "$(gettext "'%s' does not have a valid database archive extension.")" \ -- 2.14.1
On 29/08/17 15:01, Eli Schwartz wrote:
This allows for more easily extending the list of allowed compression methods, as it has to be modified in only one place.
Also allow the user to specify their own preferred command + options for source packages in addition to compiled packages. Currently, makepkg.conf(5) erroneously claims this is already possible.
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> ---
I'm not 100% sure how to handle this, perhaps it should encompass bsdtar/$pkg_file as well?
scripts/makepkg.sh.in | 57 ++++++++++++++++++++++++++------------------------- 1 file changed, 29 insertions(+), 28 deletions(-)
diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 20e9dd7e..a5c216dd 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -698,6 +698,26 @@ list_package_files() { sed -e 's|^\./||' | tr '\n' '\0' }
+# Wrapper around many stream compression formats, for use in the middle of a +# pipeline. A tar archive is passed on stdin and compressed to stdout. +compress_as() {
This should really make its way to libmakepkg in util/compress.sh. I have also internal debated the name for a while. compress_as() vs compress_to() vs compress(). I was slightly leaning to the last one...
+ # $1: final archive filename extension for compression type detection + + local ext="$1" + + case "$ext" in + *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; + *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; + *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; + *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; + *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; + *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;;
Aside: we need to do something about all these COMPRESSFOO variables... Adding a new one to our example makepkg.conf with each new addition is not ideal. Perhaps we can just document them in the man page but no have default entries in makepkg.conf? Or is there a better way?
+ *tar) cat ;; + *) warning "$(gettext "'%s' is not a valid archive extension.")" \ + "$ext"; cat ;; + esac +} +
On 09/14/2017 02:27 AM, Allan McRae wrote:
+# Wrapper around many stream compression formats, for use in the middle of a +# pipeline. A tar archive is passed on stdin and compressed to stdout. +compress_as() {
This should really make its way to libmakepkg in util/compress.sh.
I suppose I can do that, anything else that should go there or is it a one-function library?
I have also internal debated the name for a while. compress_as() vs compress_to() vs compress(). I was slightly leaning to the last one...
I'm not really sure what to call it either. Worth noting though that compress() will not work -- POSIX has a compress/decompress command for handling *.Z which is in fact utilized in the function itself, so that would be rather recursive if anyone tried to use PKGEXT=.pkg.tar.Z for some outdated reason. OTOH I cannot find a repository package to provide that currently (but gzip does provide uncompress as a second copy of the `gunzip -d` wrapper which is also installed as gunzip). Although the AUR package "ncompress" appears to be the modern canonical source of a non-patented `compress` implementation. So it is essentially dead (or at least AUR) code. (Gripe: why does this AUR package claim the license for a public domain/unlicense'd codebase is "GPL".)
+ # $1: final archive filename extension for compression type detection + + local ext="$1" + + case "$ext" in + *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; + *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; + *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; + *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; + *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; + *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;;
Aside: we need to do something about all these COMPRESSFOO variables... Adding a new one to our example makepkg.conf with each new addition is not ideal. Perhaps we can just document them in the man page but no have default entries in makepkg.conf? Or is there a better way?
Well, the makepkg.conf is already rather duplicative of the effort which goes into its manpage. But I don't think we add new compression methods all that often. I don't really have a strong opinion on this.
+ *tar) cat ;; + *) warning "$(gettext "'%s' is not a valid archive extension.")" \ + "$ext"; cat ;; + esac +} +
-- Eli Schwartz
This allows for more easily extending the list of allowed compression methods, as it has to be modified in only one place. Also allow the user to specify their own preferred command + options for source packages in addition to compiled packages. Currently, makepkg.conf(5) erroneously claims this is already possible. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- v2: refactor into a new libmakepkg library Fix gitignore I still prefer compress_as() over compress_to(), unless you'd like this function to write the file as well, rather than just compress from stdin -> stdout scripts/Makefile.am | 1 + scripts/libmakepkg/.gitignore | 6 +---- scripts/libmakepkg/util/compress.sh.in | 47 ++++++++++++++++++++++++++++++++++ scripts/makepkg.sh.in | 37 +++++++------------------- 4 files changed, 58 insertions(+), 33 deletions(-) create mode 100644 scripts/libmakepkg/util/compress.sh.in diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 0a36fc58..3ec652ce 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -96,6 +96,7 @@ LIBMAKEPKG_IN = \ libmakepkg/tidy/strip.sh \ libmakepkg/tidy/zipman.sh \ libmakepkg/util.sh \ + libmakepkg/util/compress.sh \ libmakepkg/util/message.sh \ libmakepkg/util/option.sh \ libmakepkg/util/parseopts.sh \ diff --git a/scripts/libmakepkg/.gitignore b/scripts/libmakepkg/.gitignore index ceb7bfc4..cf1426a2 100644 --- a/scripts/libmakepkg/.gitignore +++ b/scripts/libmakepkg/.gitignore @@ -10,8 +10,4 @@ srcinfo.sh tidy.sh tidy/*.sh util.sh -util/message.sh -util/option.sh -util/parseopts.sh -util/pkgbuild.sh -util/source.sh +util/*.sh diff --git a/scripts/libmakepkg/util/compress.sh.in b/scripts/libmakepkg/util/compress.sh.in new file mode 100644 index 00000000..4e020e7f --- /dev/null +++ b/scripts/libmakepkg/util/compress.sh.in @@ -0,0 +1,47 @@ +#!/bin/bash +# +# compress.sh - functions to compress archives in a uniform manner +# +# Copyright (c) 2017 Pacman Development Team <pacman-dev@archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_UTIL_COMPRESS_SH" ]] && return +LIBMAKEPKG_UTIL_COMPRESS_SH=1 + +LIBRARY=${LIBRARY:-'@libmakepkgdir@'} + +source "$LIBRARY/util/message.sh" + + +# Wrapper around many stream compression formats, for use in the middle of a +# pipeline. A tar archive is passed on stdin and compressed to stdout. +compress_as() { + # $1: final archive filename extension for compression type detection + + local filename="$1" + + case "$filename" in + *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; + *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; + *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; + *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; + *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; + *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; + *tar) cat ;; + *) warning "$(gettext "'%s' is not a valid archive extension.")" \ + "$ext"; cat ;; + esac +} diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index fdc8bb6a..6449accd 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -754,21 +754,8 @@ create_package() { msg2 "$(gettext "Compressing package...")" # TODO: Maybe this can be set globally for robustness shopt -s -o pipefail - # bsdtar's gzip compression always saves the time stamp, making one - # archive created using the same command line distinct from another. - # Disable bsdtar compression and use gzip -n for now. list_package_files | LANG=C bsdtar -cnf - --null --files-from - | - case "$PKGEXT" in - *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; - *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; - *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; - *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; - *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; - *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; - *tar) cat ;; - *) warning "$(gettext "'%s' is not a valid archive extension.")" \ - "$PKGEXT"; cat ;; - esac > "${pkg_file}" || ret=$? + compress_as "$PKGEXT" > "${pkg_file}" || ret=$? shopt -u -o pipefail @@ -849,26 +836,20 @@ create_srcpackage() { done done - local TAR_OPT - case "$SRCEXT" in - *tar.gz) TAR_OPT="-z" ;; - *tar.bz2) TAR_OPT="-j" ;; - *tar.xz) TAR_OPT="-J" ;; - *tar.lrz) TAR_OPT="--lrzip" ;; - *tar.lzo) TAR_OPT="--lzop" ;; - *tar.Z) TAR_OPT="-Z" ;; - *tar) TAR_OPT="" ;; - *) warning "$(gettext "'%s' is not a valid archive extension.")" \ - "$SRCEXT" ;; - esac - local fullver=$(get_full_version) local pkg_file="$SRCPKGDEST/${pkgbase}-${fullver}${SRCEXT}" # tar it up msg2 "$(gettext "Compressing source package...")" cd_safe "${srclinks}" - if ! LANG=C bsdtar -cL ${TAR_OPT} -f "$pkg_file" ${pkgbase}; then + + # TODO: Maybe this can be set globally for robustness + shopt -s -o pipefail + LANG=C bsdtar -cLf - ${pkgbase} | compress_as "$SRCEXT" > "${pkg_file}" || ret=$? + + shopt -u -o pipefail + + if (( ret )); then error "$(gettext "Failed to create source package file.")" exit 1 # TODO: error code fi -- 2.14.2
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- doc/makepkg.conf.5.txt | 3 ++- etc/makepkg.conf.in | 1 + scripts/libmakepkg/util/compress.sh.in | 1 + 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/makepkg.conf.5.txt b/doc/makepkg.conf.5.txt index 7ff29401..b50423a0 100644 --- a/doc/makepkg.conf.5.txt +++ b/doc/makepkg.conf.5.txt @@ -243,6 +243,7 @@ Options **COMPRESSGZ=**"(gzip -c -f -n)":: **COMPRESSBZ2=**"(bzip2 -c -f)":: **COMPRESSXZ=**"(xz -c -z -)":: +**COMPRESSZST=**"(zstd -c -z -)":: **COMPRESSLZO**"(lzop -q)":: **COMPRESSLRZ=**"(lrzip -q)":: **COMPRESSZ=**"(compress -c -f)":: @@ -251,7 +252,7 @@ Options **PKGEXT=**".pkg.tar.gz", **SRCEXT=**".src.tar.gz":: Sets the compression used when making compiled or source packages. - Valid suffixes are `.tar`, `.tar.gz`, `.tar.bz2`, `.tar.xz`, + Valid suffixes are `.tar`, `.tar.gz`, `.tar.bz2`, `.tar.xz`, `.tar.zst`, `.tar.lzo`, `.tar.lrz`, and `.tar.Z`. Do not touch these unless you know what you are doing. diff --git a/etc/makepkg.conf.in b/etc/makepkg.conf.in index dc27e33a..5f1394d3 100644 --- a/etc/makepkg.conf.in +++ b/etc/makepkg.conf.in @@ -126,6 +126,7 @@ PURGE_TARGETS=(usr/{,share}/info/dir .packlist *.pod) COMPRESSGZ=(gzip -c -f -n) COMPRESSBZ2=(bzip2 -c -f) COMPRESSXZ=(xz -c -z -) +COMPRESSXZ=(zstd -c -z -q -) COMPRESSLRZ=(lrzip -q) COMPRESSLZO=(lzop -q) COMPRESSZ=(compress -c -f) diff --git a/scripts/libmakepkg/util/compress.sh.in b/scripts/libmakepkg/util/compress.sh.in index 4e020e7f..b246197e 100644 --- a/scripts/libmakepkg/util/compress.sh.in +++ b/scripts/libmakepkg/util/compress.sh.in @@ -37,6 +37,7 @@ compress_as() { *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; + *tar.zst) ${COMPRESSZST[@]:-zstd -c -z -q -} ;; *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; -- 2.14.2
On 08/10/17 16:50, Eli Schwartz wrote:
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- doc/makepkg.conf.5.txt | 3 ++- etc/makepkg.conf.in | 1 + scripts/libmakepkg/util/compress.sh.in | 1 + 3 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/doc/makepkg.conf.5.txt b/doc/makepkg.conf.5.txt index 7ff29401..b50423a0 100644 --- a/doc/makepkg.conf.5.txt +++ b/doc/makepkg.conf.5.txt @@ -243,6 +243,7 @@ Options **COMPRESSGZ=**"(gzip -c -f -n)":: **COMPRESSBZ2=**"(bzip2 -c -f)":: **COMPRESSXZ=**"(xz -c -z -)":: +**COMPRESSZST=**"(zstd -c -z -)":: **COMPRESSLZO**"(lzop -q)":: **COMPRESSLRZ=**"(lrzip -q)":: **COMPRESSZ=**"(compress -c -f)":: @@ -251,7 +252,7 @@ Options
**PKGEXT=**".pkg.tar.gz", **SRCEXT=**".src.tar.gz":: Sets the compression used when making compiled or source packages. - Valid suffixes are `.tar`, `.tar.gz`, `.tar.bz2`, `.tar.xz`, + Valid suffixes are `.tar`, `.tar.gz`, `.tar.bz2`, `.tar.xz`, `.tar.zst`, `.tar.lzo`, `.tar.lrz`, and `.tar.Z`. Do not touch these unless you know what you are doing.
diff --git a/etc/makepkg.conf.in b/etc/makepkg.conf.in index dc27e33a..5f1394d3 100644 --- a/etc/makepkg.conf.in +++ b/etc/makepkg.conf.in @@ -126,6 +126,7 @@ PURGE_TARGETS=(usr/{,share}/info/dir .packlist *.pod) COMPRESSGZ=(gzip -c -f -n) COMPRESSBZ2=(bzip2 -c -f) COMPRESSXZ=(xz -c -z -) +COMPRESSXZ=(zstd -c -z -q -)
Typo.
COMPRESSLRZ=(lrzip -q) COMPRESSLZO=(lzop -q) COMPRESSZ=(compress -c -f) diff --git a/scripts/libmakepkg/util/compress.sh.in b/scripts/libmakepkg/util/compress.sh.in index 4e020e7f..b246197e 100644 --- a/scripts/libmakepkg/util/compress.sh.in +++ b/scripts/libmakepkg/util/compress.sh.in @@ -37,6 +37,7 @@ compress_as() { *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; + *tar.zst) ${COMPRESSZST[@]:-zstd -c -z -q -} ;; *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;;
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- I'm now wondering if it makes sense to import libmakepkg/util/compress.sh here for reuse. Currently repo-add checks the extensions twice and aborts on unknown extensions, but I'm not sure why we need to be stricter here than in makepkg itself. scripts/repo-add.sh.in | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index 41a2ea93..76c3ee46 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -300,6 +300,7 @@ verify_repo_extension() { *.db.tar.gz) TAR_OPT="z" ;; *.db.tar.bz2) TAR_OPT="j" ;; *.db.tar.xz) TAR_OPT="J" ;; + *.db.tar.zst) TAR_OPT="--zstd" ;; *.db.tar.Z) TAR_OPT="Z" ;; *.db.tar) TAR_OPT="" ;; *) error "$(gettext "'%s' does not have a valid database archive extension.")" \ -- 2.14.2
On 08/10/17 16:50, Eli Schwartz wrote:
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> ---
I'm now wondering if it makes sense to import libmakepkg/util/compress.sh here for reuse. Currently repo-add checks the extensions twice and aborts on unknown extensions, but I'm not sure why we need to be stricter here than in makepkg itself.
OK - but won't be pulled until a new libarchive release is made. Also, we are not setup to use libmakepkg in repo-add. And I have vague plans to make repo-add a libalpm based program, so it is probably not worth it. A
On 08/10/17 16:50, Eli Schwartz wrote:
This allows for more easily extending the list of allowed compression methods, as it has to be modified in only one place.
Also allow the user to specify their own preferred command + options for source packages in addition to compiled packages. Currently, makepkg.conf(5) erroneously claims this is already possible.
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> ---
v2: refactor into a new libmakepkg library Fix gitignore
I still prefer compress_as() over compress_to(), unless you'd like this function to write the file as well, rather than just compress from stdin -> stdout
scripts/Makefile.am | 1 + scripts/libmakepkg/.gitignore | 6 +---- scripts/libmakepkg/util/compress.sh.in | 47 ++++++++++++++++++++++++++++++++++ scripts/makepkg.sh.in | 37 +++++++------------------- 4 files changed, 58 insertions(+), 33 deletions(-) create mode 100644 scripts/libmakepkg/util/compress.sh.in
diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 0a36fc58..3ec652ce 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -96,6 +96,7 @@ LIBMAKEPKG_IN = \ libmakepkg/tidy/strip.sh \ libmakepkg/tidy/zipman.sh \ libmakepkg/util.sh \ + libmakepkg/util/compress.sh \ libmakepkg/util/message.sh \ libmakepkg/util/option.sh \ libmakepkg/util/parseopts.sh \ diff --git a/scripts/libmakepkg/.gitignore b/scripts/libmakepkg/.gitignore index ceb7bfc4..cf1426a2 100644 --- a/scripts/libmakepkg/.gitignore +++ b/scripts/libmakepkg/.gitignore @@ -10,8 +10,4 @@ srcinfo.sh tidy.sh tidy/*.sh util.sh -util/message.sh -util/option.sh -util/parseopts.sh -util/pkgbuild.sh -util/source.sh +util/*.sh
Unrelated change. Separate patch please. In fact, since aca153bf, that whole file can just be *.sh
diff --git a/scripts/libmakepkg/util/compress.sh.in b/scripts/libmakepkg/util/compress.sh.in new file mode 100644 index 00000000..4e020e7f --- /dev/null +++ b/scripts/libmakepkg/util/compress.sh.in @@ -0,0 +1,47 @@ +#!/bin/bash +# +# compress.sh - functions to compress archives in a uniform manner +# +# Copyright (c) 2017 Pacman Development Team <pacman-dev@archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_UTIL_COMPRESS_SH" ]] && return +LIBMAKEPKG_UTIL_COMPRESS_SH=1 + +LIBRARY=${LIBRARY:-'@libmakepkgdir@'} + +source "$LIBRARY/util/message.sh" + + +# Wrapper around many stream compression formats, for use in the middle of a +# pipeline. A tar archive is passed on stdin and compressed to stdout. +compress_as() { + # $1: final archive filename extension for compression type detection + + local filename="$1" + + case "$filename" in + *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; + *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; + *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; + *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; + *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; + *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; + *tar) cat ;; + *) warning "$(gettext "'%s' is not a valid archive extension.")" \ + "$ext"; cat ;; + esac +} diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index fdc8bb6a..6449accd 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -754,21 +754,8 @@ create_package() { msg2 "$(gettext "Compressing package...")" # TODO: Maybe this can be set globally for robustness shopt -s -o pipefail - # bsdtar's gzip compression always saves the time stamp, making one - # archive created using the same command line distinct from another. - # Disable bsdtar compression and use gzip -n for now. list_package_files | LANG=C bsdtar -cnf - --null --files-from - | - case "$PKGEXT" in - *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; - *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; - *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; - *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; - *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; - *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; - *tar) cat ;; - *) warning "$(gettext "'%s' is not a valid archive extension.")" \ - "$PKGEXT"; cat ;; - esac > "${pkg_file}" || ret=$? + compress_as "$PKGEXT" > "${pkg_file}" || ret=$?
shopt -u -o pipefail
@@ -849,26 +836,20 @@ create_srcpackage() { done done
- local TAR_OPT - case "$SRCEXT" in - *tar.gz) TAR_OPT="-z" ;; - *tar.bz2) TAR_OPT="-j" ;; - *tar.xz) TAR_OPT="-J" ;; - *tar.lrz) TAR_OPT="--lrzip" ;; - *tar.lzo) TAR_OPT="--lzop" ;; - *tar.Z) TAR_OPT="-Z" ;; - *tar) TAR_OPT="" ;; - *) warning "$(gettext "'%s' is not a valid archive extension.")" \ - "$SRCEXT" ;; - esac - local fullver=$(get_full_version) local pkg_file="$SRCPKGDEST/${pkgbase}-${fullver}${SRCEXT}"
# tar it up msg2 "$(gettext "Compressing source package...")" cd_safe "${srclinks}" - if ! LANG=C bsdtar -cL ${TAR_OPT} -f "$pkg_file" ${pkgbase}; then + + # TODO: Maybe this can be set globally for robustness + shopt -s -o pipefail + LANG=C bsdtar -cLf - ${pkgbase} | compress_as "$SRCEXT" > "${pkg_file}" || ret=$? + + shopt -u -o pipefail + + if (( ret )); then error "$(gettext "Failed to create source package file.")" exit 1 # TODO: error code fi
Prior to commit aca153bfa6b1bcd828f0b35db453bb9fea6a08bf some .sh files were not generated and simply included directly, and it was necessary to explicitly iterate all ignored files to prevent git from ignoring the directly-included files. However, now all .sh files are in fact generated so it makes no sense to list each one separately in the .gitignore file. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- scripts/libmakepkg/.gitignore | 18 +----------------- 1 file changed, 1 insertion(+), 17 deletions(-) diff --git a/scripts/libmakepkg/.gitignore b/scripts/libmakepkg/.gitignore index ceb7bfc4..c97f963b 100644 --- a/scripts/libmakepkg/.gitignore +++ b/scripts/libmakepkg/.gitignore @@ -1,17 +1 @@ -integrity.sh -integrity/*.sh -lint_package.sh -lint_package/*.sh -lint_pkgbuild.sh -lint_pkgbuild/*.sh -source.sh -source/*.sh -srcinfo.sh -tidy.sh -tidy/*.sh -util.sh -util/message.sh -util/option.sh -util/parseopts.sh -util/pkgbuild.sh -util/source.sh +*.sh -- 2.15.0
This allows for more easily extending the list of allowed compression methods, as it has to be modified in only one place. Also allow the user to specify their own preferred command + options for source packages in addition to compiled packages. Currently, makepkg.conf(5) erroneously claims this is already possible. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- v3: .gitignore separated into own commit scripts/Makefile.am | 1 + scripts/libmakepkg/util/compress.sh.in | 47 ++++++++++++++++++++++++++++++++++ scripts/makepkg.sh.in | 37 +++++++------------------- 3 files changed, 57 insertions(+), 28 deletions(-) create mode 100644 scripts/libmakepkg/util/compress.sh.in diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 86e44cb6..75e439e8 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -96,6 +96,7 @@ LIBMAKEPKG_IN = \ libmakepkg/tidy/strip.sh \ libmakepkg/tidy/zipman.sh \ libmakepkg/util.sh \ + libmakepkg/util/compress.sh \ libmakepkg/util/message.sh \ libmakepkg/util/option.sh \ libmakepkg/util/parseopts.sh \ diff --git a/scripts/libmakepkg/util/compress.sh.in b/scripts/libmakepkg/util/compress.sh.in new file mode 100644 index 00000000..4e020e7f --- /dev/null +++ b/scripts/libmakepkg/util/compress.sh.in @@ -0,0 +1,47 @@ +#!/bin/bash +# +# compress.sh - functions to compress archives in a uniform manner +# +# Copyright (c) 2017 Pacman Development Team <pacman-dev@archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_UTIL_COMPRESS_SH" ]] && return +LIBMAKEPKG_UTIL_COMPRESS_SH=1 + +LIBRARY=${LIBRARY:-'@libmakepkgdir@'} + +source "$LIBRARY/util/message.sh" + + +# Wrapper around many stream compression formats, for use in the middle of a +# pipeline. A tar archive is passed on stdin and compressed to stdout. +compress_as() { + # $1: final archive filename extension for compression type detection + + local filename="$1" + + case "$filename" in + *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; + *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; + *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; + *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; + *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; + *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; + *tar) cat ;; + *) warning "$(gettext "'%s' is not a valid archive extension.")" \ + "$ext"; cat ;; + esac +} diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 54998fdb..462ee4bb 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -754,21 +754,8 @@ create_package() { msg2 "$(gettext "Compressing package...")" # TODO: Maybe this can be set globally for robustness shopt -s -o pipefail - # bsdtar's gzip compression always saves the time stamp, making one - # archive created using the same command line distinct from another. - # Disable bsdtar compression and use gzip -n for now. list_package_files | LANG=C bsdtar -cnf - --null --files-from - | - case "$PKGEXT" in - *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; - *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; - *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; - *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; - *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; - *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; - *tar) cat ;; - *) warning "$(gettext "'%s' is not a valid archive extension.")" \ - "$PKGEXT"; cat ;; - esac > "${pkg_file}" || ret=$? + compress_as "$PKGEXT" > "${pkg_file}" || ret=$? shopt -u -o pipefail @@ -849,26 +836,20 @@ create_srcpackage() { done done - local TAR_OPT - case "$SRCEXT" in - *tar.gz) TAR_OPT="-z" ;; - *tar.bz2) TAR_OPT="-j" ;; - *tar.xz) TAR_OPT="-J" ;; - *tar.lrz) TAR_OPT="--lrzip" ;; - *tar.lzo) TAR_OPT="--lzop" ;; - *tar.Z) TAR_OPT="-Z" ;; - *tar) TAR_OPT="" ;; - *) warning "$(gettext "'%s' is not a valid archive extension.")" \ - "$SRCEXT" ;; - esac - local fullver=$(get_full_version) local pkg_file="$SRCPKGDEST/${pkgbase}-${fullver}${SRCEXT}" # tar it up msg2 "$(gettext "Compressing source package...")" cd_safe "${srclinks}" - if ! LANG=C bsdtar -cL ${TAR_OPT} -f "$pkg_file" ${pkgbase}; then + + # TODO: Maybe this can be set globally for robustness + shopt -s -o pipefail + LANG=C bsdtar -cLf - ${pkgbase} | compress_as "$SRCEXT" > "${pkg_file}" || ret=$? + + shopt -u -o pipefail + + if (( ret )); then error "$(gettext "Failed to create source package file.")" exit 1 # TODO: error code fi -- 2.15.0
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- v3: fix typo doc/makepkg.conf.5.txt | 3 ++- etc/makepkg.conf.in | 1 + scripts/libmakepkg/util/compress.sh.in | 1 + 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/makepkg.conf.5.txt b/doc/makepkg.conf.5.txt index 5bd3472f..d01c13b6 100644 --- a/doc/makepkg.conf.5.txt +++ b/doc/makepkg.conf.5.txt @@ -246,6 +246,7 @@ Options **COMPRESSGZ=**"(gzip -c -f -n)":: **COMPRESSBZ2=**"(bzip2 -c -f)":: **COMPRESSXZ=**"(xz -c -z -)":: +**COMPRESSZST=**"(zstd -c -z -)":: **COMPRESSLZO**"(lzop -q)":: **COMPRESSLRZ=**"(lrzip -q)":: **COMPRESSZ=**"(compress -c -f)":: @@ -254,7 +255,7 @@ Options **PKGEXT=**".pkg.tar.gz", **SRCEXT=**".src.tar.gz":: Sets the compression used when making compiled or source packages. - Valid suffixes are `.tar`, `.tar.gz`, `.tar.bz2`, `.tar.xz`, + Valid suffixes are `.tar`, `.tar.gz`, `.tar.bz2`, `.tar.xz`, `.tar.zst`, `.tar.lzo`, `.tar.lrz`, and `.tar.Z`. Do not touch these unless you know what you are doing. diff --git a/etc/makepkg.conf.in b/etc/makepkg.conf.in index 71293970..7782d0de 100644 --- a/etc/makepkg.conf.in +++ b/etc/makepkg.conf.in @@ -126,6 +126,7 @@ PURGE_TARGETS=(usr/{,share}/info/dir .packlist *.pod) COMPRESSGZ=(gzip -c -f -n) COMPRESSBZ2=(bzip2 -c -f) COMPRESSXZ=(xz -c -z -) +COMPRESSZST=(zstd -c -z -q -) COMPRESSLRZ=(lrzip -q) COMPRESSLZO=(lzop -q) COMPRESSZ=(compress -c -f) diff --git a/scripts/libmakepkg/util/compress.sh.in b/scripts/libmakepkg/util/compress.sh.in index 4e020e7f..b246197e 100644 --- a/scripts/libmakepkg/util/compress.sh.in +++ b/scripts/libmakepkg/util/compress.sh.in @@ -37,6 +37,7 @@ compress_as() { *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; + *tar.zst) ${COMPRESSZST[@]:-zstd -c -z -q -} ;; *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; -- 2.15.0
On 31/10/17 04:15, Eli Schwartz wrote:
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> ---
This and the repo-add patch look fine, but will not be pulled until libarchive supporting these formats is released.
v3: fix typo
doc/makepkg.conf.5.txt | 3 ++- etc/makepkg.conf.in | 1 + scripts/libmakepkg/util/compress.sh.in | 1 + 3 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/doc/makepkg.conf.5.txt b/doc/makepkg.conf.5.txt index 5bd3472f..d01c13b6 100644 --- a/doc/makepkg.conf.5.txt +++ b/doc/makepkg.conf.5.txt @@ -246,6 +246,7 @@ Options **COMPRESSGZ=**"(gzip -c -f -n)":: **COMPRESSBZ2=**"(bzip2 -c -f)":: **COMPRESSXZ=**"(xz -c -z -)":: +**COMPRESSZST=**"(zstd -c -z -)":: **COMPRESSLZO**"(lzop -q)":: **COMPRESSLRZ=**"(lrzip -q)":: **COMPRESSZ=**"(compress -c -f)"::
There has got to be a better way than continuously extending this variable list... A
On 12/7/17 12:51 AM, Allan McRae wrote:
On 31/10/17 04:15, Eli Schwartz wrote:
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> ---
This and the repo-add patch look fine, but will not be pulled until libarchive supporting these formats is released.
libarchive 3.3.3 is now released and in [testing] as of 45 minutes ago so this is good to go! -- Eli Schwartz Bug Wrangler and Trusted User
On 4/9/18 6:14 pm, Eli Schwartz wrote:
On 12/7/17 12:51 AM, Allan McRae wrote:
On 31/10/17 04:15, Eli Schwartz wrote:
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> ---
This and the repo-add patch look fine, but will not be pulled until libarchive supporting these formats is released.
libarchive 3.3.3 is now released and in [testing] as of 45 minutes ago so this is good to go!
rebased and applied. A
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- no changes I decided not to refactor repo-add, at least not right now. The current code makes too many assumptions about its differing implementation. I'm not sure why we fail on unknown extensions, really, considering pacman doesn't care if we fall back to plain old tar, and we take advantage of this in makepkg. scripts/repo-add.sh.in | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index 68edbf4b..ccc0c89b 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -300,6 +300,7 @@ verify_repo_extension() { *.db.tar.gz) TAR_OPT="z" ;; *.db.tar.bz2) TAR_OPT="j" ;; *.db.tar.xz) TAR_OPT="J" ;; + *.db.tar.zst) TAR_OPT="--zstd" ;; *.db.tar.Z) TAR_OPT="Z" ;; *.db.tar) TAR_OPT="" ;; *) error "$(gettext "'%s' does not have a valid database archive extension.")" \ -- 2.15.0
participants (2)
-
Allan McRae
-
Eli Schwartz