On 3/24/19 8:38 PM, Evangelos Foutras via arch-dev-public wrote:
On Mon, 25 Mar 2019 at 01:22, Jan Alexander Steffens via arch-dev-public email@example.com wrote:
On Sun, Mar 24, 2019 at 7:35 PM Robin Broda via arch-dev-public firstname.lastname@example.org wrote:
The required changeset is, i think: PKGEXT='.pkg.tar.zst' COMPRESSZST=(zstd -c -T0 -18 -)
When we implement this, I would say we go with "zstd -c -T0 -" in pacman's makepkg.conf and "zstd -C -T0 -18 -" in the configs shipped with devtools.
I think users that build their own local packages are more likely to benefit from fast compression. Anyone building with makechrootpkg for distribution gets the high compression level.
That's actually really smart! We can also leave out "-T0" since the default compression level is very fast anyway. Plus, it's already implemented in this way in pacman git so we don't have to touch anything there. (But, if it were to be multithreaded there as well, I would replace zstd with zstdmt; same for devtools.)
As far as compression level goes, I believe we should select the highest one that doesn't have increased memory requirements during decompression. So that would be -19. Robin makes a good case about -18; looking at upstream's "Compression Speed vs Ratio" graph  I would say -18 is preferable to -19 if we are concerned about compression speed and memory usage (my totally unscientific measurements show a 25% memory increase and 20% speed decrease going from -18 to -19 when using -T4). That said, I might still opt for -19 due to the slightly higher compression ratio; memory usage isn't too big of an issue and the slower speed is mitigated by multithreading (i.e.: it will still be much faster than xz).
Assuming .zst packages have been installable as far back as September 2018 when libarchive 3.3.3 was released, it seems to me that the following steps can be taken:
- Check if repo-add and dbscripts recognize .zst packages.
repo-add checks to see if bsdtar/libarchive recognizes the file as a compressed archive containing a .PKGINFO file, and therefore, repo-add will work whenever pacman works.
dbscripts whitelists the known package extensions, and I will be adding new extensions to dbscripts in tandem with a stable pacman release that contains makepkg support, so that should not be a problem either.
- Add "COMPRESSZST=(zstdmt -19 -c -z -q -)" to devtools' makepkg-x86_64.conf.
Or just sync it with the pacman package. :p
- Release a test package and confirm that it's installable. (Possibly
also test with an old installation from September 2018.)
You can confirm that today if you build using makepkg from pacman-git. Alternatively, try this test package I built:
- Announce the transition to the new compression algorithm and
provide a date-stamped mirror URL  for really old installations without libarchive 3.3.3.
We've had such transitions before, e.g. when adding hook support. IMO the transition does not need to be longer than a month, as we are substantially saying "your libarchive must be from the last six months". Arch Linux does not, AFAIK, have a general policy to support systems that have failed to update in 6 months.
If anyone manages to break their system by having a very old libarchive version after a warning period provided as a news entry, we do not need to officially support anything... but I provide fully static recovery binaries for pacman, here: https://aur.archlinux.org/packages/pacman-static This should be sufficient without recourse to legacy compat packages for libarchive, or requiring users to reset their system to any give ALA date. (Notwithstanding this, many things might break in that time frame, and general advice usually seems to be to upgrade in stages using the ALA anyway.)
They are available both as the (prebuilt custom repo) package "pacman-static", and as extracted binaries that can be downloaded and run directly without the need to run pacman or even decompress the file. They are verified with my PGP key.
These binaries are based on a libarchive.a which is compiled with libzstd.a support, so that works too.