[arch-dev-public] RFC: (devtools) Changing default compression method to zstd

[Evangelos Foutras]

On Mon, 25 Mar 2019 at 01:22, Jan Alexander Steffens via
arch-dev-public <arch-dev-public at archlinux.org> wrote:
>
> On Sun, Mar 24, 2019 at 7:35 PM Robin Broda via arch-dev-public
> <arch-dev-public at archlinux.org> wrote:
> > The required changeset is, i think:
> > PKGEXT='.pkg.tar.zst'
> > COMPRESSZST=(zstd -c -T0 -18 -)
>
> When we implement this, I would say we go with "zstd -c -T0 -" in
> pacman's makepkg.conf and "zstd -C -T0 -18 -" in the configs shipped
> with devtools.
>
> I think users that build their own local packages are more likely to
> benefit from fast compression. Anyone building with makechrootpkg for
> distribution gets the high compression level.

That's actually really smart! We can also leave out "-T0" since the
default compression level is very fast anyway. Plus, it's already
implemented in this way in pacman git so we don't have to touch
anything there. (But, if it were to be multithreaded there as well, I
would replace zstd with zstdmt; same for devtools.)

As far as compression level goes, I believe we should select the
highest one that doesn't have increased memory requirements during
decompression. So that would be -19. Robin makes a good case about
-18; looking at upstream's "Compression Speed vs Ratio" graph [1] I
would say -18 is preferable to -19 if we are concerned about
compression speed and memory usage (my totally unscientific
measurements show a 25% memory increase and 20% speed decrease going
from -18 to -19 when using -T4). That said, I might still opt for -19
due to the slightly higher compression ratio; memory usage isn't too
big of an issue and the slower speed is mitigated by multithreading
(i.e.: it will still be much faster than xz).

Assuming .zst packages have been installable as far back as September
2018 when libarchive 3.3.3 was released, it seems to me that the
following steps can be taken:

1) Check if repo-add and dbscripts recognize .zst packages.
2) Add "COMPRESSZST=(zstdmt -19 -c -z -q -)" to devtools' makepkg-x86_64.conf.
3) Release a test package and confirm that it's installable. (Possibly
also test with an old installation from September 2018.)
4) Announce the transition to the new compression algorithm and
provide a date-stamped mirror URL [2] for really old installations
without libarchive 3.3.3.

[1] https://facebook.github.io/zstd/
[2] https://archive.archlinux.org/repos/2019/xx/xx/$repo/os/$arch