On Mon, 25 Mar 2019 at 01:22, Jan Alexander Steffens via arch-dev-public <arch-dev-public@archlinux.org> wrote:
On Sun, Mar 24, 2019 at 7:35 PM Robin Broda via arch-dev-public <arch-dev-public@archlinux.org> wrote:
The required changeset is, i think: PKGEXT='.pkg.tar.zst' COMPRESSZST=(zstd -c -T0 -18 -)
When we implement this, I would say we go with "zstd -c -T0 -" in pacman's makepkg.conf and "zstd -C -T0 -18 -" in the configs shipped with devtools.
I think users that build their own local packages are more likely to benefit from fast compression. Anyone building with makechrootpkg for distribution gets the high compression level.
That's actually really smart! We can also leave out "-T0" since the default compression level is very fast anyway. Plus, it's already implemented in this way in pacman git so we don't have to touch anything there. (But, if it were to be multithreaded there as well, I would replace zstd with zstdmt; same for devtools.) As far as compression level goes, I believe we should select the highest one that doesn't have increased memory requirements during decompression. So that would be -19. Robin makes a good case about -18; looking at upstream's "Compression Speed vs Ratio" graph [1] I would say -18 is preferable to -19 if we are concerned about compression speed and memory usage (my totally unscientific measurements show a 25% memory increase and 20% speed decrease going from -18 to -19 when using -T4). That said, I might still opt for -19 due to the slightly higher compression ratio; memory usage isn't too big of an issue and the slower speed is mitigated by multithreading (i.e.: it will still be much faster than xz). Assuming .zst packages have been installable as far back as September 2018 when libarchive 3.3.3 was released, it seems to me that the following steps can be taken: 1) Check if repo-add and dbscripts recognize .zst packages. 2) Add "COMPRESSZST=(zstdmt -19 -c -z -q -)" to devtools' makepkg-x86_64.conf. 3) Release a test package and confirm that it's installable. (Possibly also test with an old installation from September 2018.) 4) Announce the transition to the new compression algorithm and provide a date-stamped mirror URL [2] for really old installations without libarchive 3.3.3. [1] https://facebook.github.io/zstd/ [2] https://archive.archlinux.org/repos/2019/xx/xx/$repo/os/$arch