Signed-off-by: Eli Schwartz eschwartz@archlinux.org --- doc/makepkg.conf.5.txt | 3 ++- etc/makepkg.conf.in | 1 + scripts/libmakepkg/util/compress.sh.in | 1 + 3 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/doc/makepkg.conf.5.txt b/doc/makepkg.conf.5.txt index 7ff29401..b50423a0 100644 --- a/doc/makepkg.conf.5.txt +++ b/doc/makepkg.conf.5.txt @@ -243,6 +243,7 @@ Options **COMPRESSGZ=**"(gzip -c -f -n)":: **COMPRESSBZ2=**"(bzip2 -c -f)":: **COMPRESSXZ=**"(xz -c -z -)":: +**COMPRESSZST=**"(zstd -c -z -)":: **COMPRESSLZO**"(lzop -q)":: **COMPRESSLRZ=**"(lrzip -q)":: **COMPRESSZ=**"(compress -c -f)":: @@ -251,7 +252,7 @@ Options

**PKGEXT=**".pkg.tar.gz", **SRCEXT=**".src.tar.gz":: Sets the compression used when making compiled or source packages. - Valid suffixes are `.tar`, `.tar.gz`, `.tar.bz2`, `.tar.xz`, + Valid suffixes are `.tar`, `.tar.gz`, `.tar.bz2`, `.tar.xz`, `.tar.zst`, `.tar.lzo`, `.tar.lrz`, and `.tar.Z`. Do not touch these unless you know what you are doing.

diff --git a/etc/makepkg.conf.in b/etc/makepkg.conf.in index dc27e33a..5f1394d3 100644 --- a/etc/makepkg.conf.in +++ b/etc/makepkg.conf.in @@ -126,6 +126,7 @@ PURGE_TARGETS=(usr/{,share}/info/dir .packlist *.pod) COMPRESSGZ=(gzip -c -f -n) COMPRESSBZ2=(bzip2 -c -f) COMPRESSXZ=(xz -c -z -) +COMPRESSXZ=(zstd -c -z -q -) COMPRESSLRZ=(lrzip -q) COMPRESSLZO=(lzop -q) COMPRESSZ=(compress -c -f) diff --git a/scripts/libmakepkg/util/compress.sh.in b/scripts/libmakepkg/util/compress.sh.in index 4e020e7f..b246197e 100644 --- a/scripts/libmakepkg/util/compress.sh.in +++ b/scripts/libmakepkg/util/compress.sh.in @@ -37,6 +37,7 @@ compress_as() { *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; + *tar.zst) ${COMPRESSZST[@]:-zstd -c -z -q -} ;; *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;;

Signed-off-by: Eli Schwartz eschwartz@archlinux.org ---

I'm now wondering if it makes sense to import libmakepkg/util/compress.sh here for reuse. Currently repo-add checks the extensions twice and aborts on unknown extensions, but I'm not sure why we need to be stricter here than in makepkg itself.

scripts/repo-add.sh.in | 1 + 1 file changed, 1 insertion(+)

diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index 41a2ea93..76c3ee46 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -300,6 +300,7 @@ verify_repo_extension() { *.db.tar.gz) TAR_OPT="z" ;; *.db.tar.bz2) TAR_OPT="j" ;; *.db.tar.xz) TAR_OPT="J" ;; + *.db.tar.zst) TAR_OPT="--zstd" ;; *.db.tar.Z) TAR_OPT="Z" ;; *.db.tar) TAR_OPT="" ;; *) error "$(gettext "'%s' does not have a valid database archive extension.")" \

On 08/10/17 16:50, Eli Schwartz wrote:

This allows for more easily extending the list of allowed compression methods, as it has to be modified in only one place.

Also allow the user to specify their own preferred command + options for source packages in addition to compiled packages. Currently, makepkg.conf(5) erroneously claims this is already possible.

Signed-off-by: Eli Schwartz eschwartz@archlinux.org

v2: refactor into a new libmakepkg library Fix gitignore

I still prefer compress_as() over compress_to(), unless you'd like this function to write the file as well, rather than just compress from stdin -> stdout

scripts/Makefile.am | 1 + scripts/libmakepkg/.gitignore | 6 +---- scripts/libmakepkg/util/compress.sh.in | 47 ++++++++++++++++++++++++++++++++++ scripts/makepkg.sh.in | 37 +++++++------------------- 4 files changed, 58 insertions(+), 33 deletions(-) create mode 100644 scripts/libmakepkg/util/compress.sh.in

diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 0a36fc58..3ec652ce 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -96,6 +96,7 @@ LIBMAKEPKG_IN = \ libmakepkg/tidy/strip.sh \ libmakepkg/tidy/zipman.sh \ libmakepkg/util.sh \

• libmakepkg/util/compress.sh \ libmakepkg/util/message.sh \ libmakepkg/util/option.sh \ libmakepkg/util/parseopts.sh \

diff --git a/scripts/libmakepkg/.gitignore b/scripts/libmakepkg/.gitignore index ceb7bfc4..cf1426a2 100644 --- a/scripts/libmakepkg/.gitignore +++ b/scripts/libmakepkg/.gitignore @@ -10,8 +10,4 @@ srcinfo.sh tidy.sh tidy/*.sh util.sh -util/message.sh -util/option.sh -util/parseopts.sh -util/pkgbuild.sh -util/source.sh +util/*.sh

Unrelated change. Separate patch please.

In fact, since aca153bf, that whole file can just be *.sh

diff --git a/scripts/libmakepkg/util/compress.sh.in b/scripts/libmakepkg/util/compress.sh.in new file mode 100644 index 00000000..4e020e7f --- /dev/null +++ b/scripts/libmakepkg/util/compress.sh.in @@ -0,0 +1,47 @@ +#!/bin/bash +# +# compress.sh - functions to compress archives in a uniform manner +# +# Copyright (c) 2017 Pacman Development Team pacman-dev@archlinux.org +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see http://www.gnu.org/licenses/. +#

+[[ -n "$LIBMAKEPKG_UTIL_COMPRESS_SH" ]] && return +LIBMAKEPKG_UTIL_COMPRESS_SH=1

+LIBRARY=${LIBRARY:-'@libmakepkgdir@'}

+source "$LIBRARY/util/message.sh"

+# Wrapper around many stream compression formats, for use in the middle of a +# pipeline. A tar archive is passed on stdin and compressed to stdout. +compress_as() {

•   # $1: final archive filename extension for compression type detection
  

• local filename="$1"
• case "$filename" in

• *tar.gz)  ${COMPRESSGZ[@]:-gzip -c -f -n} ;;
  

• *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;;
  

• *tar.xz)  ${COMPRESSXZ[@]:-xz -c -z -} ;;
  

• *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;;
  

• *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;;
  

• *tar.Z)   ${COMPRESSZ[@]:-compress -c -f} ;;
  

• *tar)     cat ;;
  

• *) warning "$(gettext "'%s' is not a valid archive extension.")" \
  

• 	"$ext"; cat ;;
  

• esac

+} diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index fdc8bb6a..6449accd 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -754,21 +754,8 @@ create_package() { msg2 "$(gettext "Compressing package...")" # TODO: Maybe this can be set globally for robustness shopt -s -o pipefail

• # bsdtar's gzip compression always saves the time stamp, making one
• # archive created using the same command line distinct from another.
• # Disable bsdtar compression and use gzip -n for now. list_package_files | LANG=C bsdtar -cnf - --null --files-from - |
• case "$PKGEXT" in

• *tar.gz)  ${COMPRESSGZ[@]:-gzip -c -f -n} ;;
  

• *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;;
  

• *tar.xz)  ${COMPRESSXZ[@]:-xz -c -z -} ;;
  

• *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;;
  

• *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;;
  

• *tar.Z)   ${COMPRESSZ[@]:-compress -c -f} ;;
  

• *tar)     cat ;;
  

• *) warning "$(gettext "'%s' is not a valid archive extension.")" \
  

• 	"$PKGEXT"; cat ;;
  

• esac > "${pkg_file}" || ret=$?

• compress_as "$PKGEXT" > "${pkg_file}" || ret=$?
  
  

  shopt -u -o pipefail

@@ -849,26 +836,20 @@ create_srcpackage() { done done

• local TAR_OPT

• case "$SRCEXT" in

• *tar.gz)  TAR_OPT="-z" ;;
  

• *tar.bz2) TAR_OPT="-j" ;;
  

• *tar.xz)  TAR_OPT="-J" ;;
  

• *tar.lrz) TAR_OPT="--lrzip" ;;
  

• *tar.lzo) TAR_OPT="--lzop" ;;
  

• *tar.Z)   TAR_OPT="-Z" ;;
  

• *tar)     TAR_OPT=""  ;;
  

• *) warning "$(gettext "'%s' is not a valid archive extension.")" \
  

• "$SRCEXT" ;;
  

• esac

• local fullver=$(get_full_version) local pkg_file="$SRCPKGDEST/${pkgbase}-${fullver}${SRCEXT}"

  # tar it up msg2 "$(gettext "Compressing source package...")" cd_safe "${srclinks}"

• if ! LANG=C bsdtar -cL ${TAR_OPT} -f "$pkg_file" ${pkgbase}; then

• # TODO: Maybe this can be set globally for robustness
• shopt -s -o pipefail
• LANG=C bsdtar -cLf - ${pkgbase} | compress_as "$SRCEXT" > "${pkg_file}" || ret=$?
• shopt -u -o pipefail
• if (( ret )); then error "$(gettext "Failed to create source package file.")" exit 1 # TODO: error code fi

This allows for more easily extending the list of allowed compression methods, as it has to be modified in only one place.

Also allow the user to specify their own preferred command + options for source packages in addition to compiled packages. Currently, makepkg.conf(5) erroneously claims this is already possible.

Signed-off-by: Eli Schwartz eschwartz@archlinux.org ---

v3: .gitignore separated into own commit

scripts/Makefile.am | 1 + scripts/libmakepkg/util/compress.sh.in | 47 ++++++++++++++++++++++++++++++++++ scripts/makepkg.sh.in | 37 +++++++------------------- 3 files changed, 57 insertions(+), 28 deletions(-) create mode 100644 scripts/libmakepkg/util/compress.sh.in

diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 86e44cb6..75e439e8 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -96,6 +96,7 @@ LIBMAKEPKG_IN = \ libmakepkg/tidy/strip.sh \ libmakepkg/tidy/zipman.sh \ libmakepkg/util.sh \ + libmakepkg/util/compress.sh \ libmakepkg/util/message.sh \ libmakepkg/util/option.sh \ libmakepkg/util/parseopts.sh \ diff --git a/scripts/libmakepkg/util/compress.sh.in b/scripts/libmakepkg/util/compress.sh.in new file mode 100644 index 00000000..4e020e7f --- /dev/null +++ b/scripts/libmakepkg/util/compress.sh.in @@ -0,0 +1,47 @@ +#!/bin/bash +# +# compress.sh - functions to compress archives in a uniform manner +# +# Copyright (c) 2017 Pacman Development Team pacman-dev@archlinux.org +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see http://www.gnu.org/licenses/. +# + +[[ -n "$LIBMAKEPKG_UTIL_COMPRESS_SH" ]] && return +LIBMAKEPKG_UTIL_COMPRESS_SH=1 + +LIBRARY=${LIBRARY:-'@libmakepkgdir@'} + +source "$LIBRARY/util/message.sh" + + +# Wrapper around many stream compression formats, for use in the middle of a +# pipeline. A tar archive is passed on stdin and compressed to stdout. +compress_as() { + # $1: final archive filename extension for compression type detection + + local filename="$1" + + case "$filename" in + *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; + *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; + *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; + *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; + *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; + *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; + *tar) cat ;; + *) warning "$(gettext "'%s' is not a valid archive extension.")" \ + "$ext"; cat ;; + esac +} diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 54998fdb..462ee4bb 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -754,21 +754,8 @@ create_package() { msg2 "$(gettext "Compressing package...")" # TODO: Maybe this can be set globally for robustness shopt -s -o pipefail - # bsdtar's gzip compression always saves the time stamp, making one - # archive created using the same command line distinct from another. - # Disable bsdtar compression and use gzip -n for now. list_package_files | LANG=C bsdtar -cnf - --null --files-from - | - case "$PKGEXT" in - *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; - *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; - *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; - *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; - *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; - *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; - *tar) cat ;; - *) warning "$(gettext "'%s' is not a valid archive extension.")" \ - "$PKGEXT"; cat ;; - esac > "${pkg_file}" || ret=$? + compress_as "$PKGEXT" > "${pkg_file}" || ret=$?

shopt -u -o pipefail

@@ -849,26 +836,20 @@ create_srcpackage() { done done

- local TAR_OPT - case "$SRCEXT" in - *tar.gz) TAR_OPT="-z" ;; - *tar.bz2) TAR_OPT="-j" ;; - *tar.xz) TAR_OPT="-J" ;; - *tar.lrz) TAR_OPT="--lrzip" ;; - *tar.lzo) TAR_OPT="--lzop" ;; - *tar.Z) TAR_OPT="-Z" ;; - *tar) TAR_OPT="" ;; - *) warning "$(gettext "'%s' is not a valid archive extension.")" \ - "$SRCEXT" ;; - esac - local fullver=$(get_full_version) local pkg_file="$SRCPKGDEST/${pkgbase}-${fullver}${SRCEXT}"

# tar it up msg2 "$(gettext "Compressing source package...")" cd_safe "${srclinks}" - if ! LANG=C bsdtar -cL ${TAR_OPT} -f "$pkg_file" ${pkgbase}; then + + # TODO: Maybe this can be set globally for robustness + shopt -s -o pipefail + LANG=C bsdtar -cLf - ${pkgbase} | compress_as "$SRCEXT" > "${pkg_file}" || ret=$? + + shopt -u -o pipefail + + if (( ret )); then error "$(gettext "Failed to create source package file.")" exit 1 # TODO: error code fi

On 31/10/17 04:15, Eli Schwartz wrote:

Signed-off-by: Eli Schwartz eschwartz@archlinux.org

This and the repo-add patch look fine, but will not be pulled until libarchive supporting these formats is released.

v3: fix typo

doc/makepkg.conf.5.txt | 3 ++- etc/makepkg.conf.in | 1 + scripts/libmakepkg/util/compress.sh.in | 1 + 3 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/doc/makepkg.conf.5.txt b/doc/makepkg.conf.5.txt index 5bd3472f..d01c13b6 100644 --- a/doc/makepkg.conf.5.txt +++ b/doc/makepkg.conf.5.txt @@ -246,6 +246,7 @@ Options **COMPRESSGZ=**"(gzip -c -f -n)":: **COMPRESSBZ2=**"(bzip2 -c -f)":: **COMPRESSXZ=**"(xz -c -z -)":: +**COMPRESSZST=**"(zstd -c -z -)":: **COMPRESSLZO**"(lzop -q)":: **COMPRESSLRZ=**"(lrzip -q)":: **COMPRESSZ=**"(compress -c -f)"::

There has got to be a better way than continuously extending this variable list...

A