[arch-dev-public] RFC: (devtools) Changing default compression method to zstd

[Eli Schwartz]

On 3/24/19 8:38 PM, Evangelos Foutras via arch-dev-public wrote:
> On Mon, 25 Mar 2019 at 01:22, Jan Alexander Steffens via
> arch-dev-public <arch-dev-public at archlinux.org> wrote:
>>
>> On Sun, Mar 24, 2019 at 7:35 PM Robin Broda via arch-dev-public
>> <arch-dev-public at archlinux.org> wrote:
>>> The required changeset is, i think:
>>> PKGEXT='.pkg.tar.zst'
>>> COMPRESSZST=(zstd -c -T0 -18 -)
>>
>> When we implement this, I would say we go with "zstd -c -T0 -" in
>> pacman's makepkg.conf and "zstd -C -T0 -18 -" in the configs shipped
>> with devtools.
>>
>> I think users that build their own local packages are more likely to
>> benefit from fast compression. Anyone building with makechrootpkg for
>> distribution gets the high compression level.
> 
> That's actually really smart! We can also leave out "-T0" since the
> default compression level is very fast anyway. Plus, it's already
> implemented in this way in pacman git so we don't have to touch
> anything there. (But, if it were to be multithreaded there as well, I
> would replace zstd with zstdmt; same for devtools.)
> 
> As far as compression level goes, I believe we should select the
> highest one that doesn't have increased memory requirements during
> decompression. So that would be -19. Robin makes a good case about
> -18; looking at upstream's "Compression Speed vs Ratio" graph [1] I
> would say -18 is preferable to -19 if we are concerned about
> compression speed and memory usage (my totally unscientific
> measurements show a 25% memory increase and 20% speed decrease going
> from -18 to -19 when using -T4). That said, I might still opt for -19
> due to the slightly higher compression ratio; memory usage isn't too
> big of an issue and the slower speed is mitigated by multithreading
> (i.e.: it will still be much faster than xz).
> 
> Assuming .zst packages have been installable as far back as September
> 2018 when libarchive 3.3.3 was released, it seems to me that the
> following steps can be taken:
> 
> 1) Check if repo-add and dbscripts recognize .zst packages.

repo-add checks to see if bsdtar/libarchive recognizes the file as a
compressed archive containing a .PKGINFO file, and therefore, repo-add
will work whenever pacman works.

dbscripts whitelists the known package extensions, and I will be adding
new extensions to dbscripts in tandem with a stable pacman release that
contains makepkg support, so that should not be a problem either.

> 2) Add "COMPRESSZST=(zstdmt -19 -c -z -q -)" to devtools' makepkg-x86_64.conf.

Or just sync it with the pacman package. :p

> 3) Release a test package and confirm that it's installable. (Possibly
> also test with an old installation from September 2018.)

You can confirm that today if you build using makepkg from pacman-git.
Alternatively, try this test package I built:

https://pkgbuild.com/~eschwartz/repo/x86_64/testpkg-foobar-1-1-any.pkg.tar.zst

> 4) Announce the transition to the new compression algorithm and
> provide a date-stamped mirror URL [2] for really old installations
> without libarchive 3.3.3.

We've had such transitions before, e.g. when adding hook support. IMO
the transition does not need to be longer than a month, as we are
substantially saying "your libarchive must be from the last six months".
Arch Linux does not, AFAIK, have a general policy to support systems
that have failed to update in 6 months.

If anyone manages to break their system by having a very old libarchive
version after a warning period provided as a news entry, we do not need
to officially support anything... but I provide fully static recovery
binaries for pacman, here: https://aur.archlinux.org/packages/pacman-static
This should be sufficient without recourse to legacy compat packages for
libarchive, or requiring users to reset their system to any give ALA date.
(Notwithstanding this, many things might break in that time frame, and
general advice usually seems to be to upgrade in stages using the ALA
anyway.)

They are available both as the (prebuilt custom repo) package
"pacman-static", and as extracted binaries that can be downloaded and
run directly without the need to run pacman or even decompress the file.
They are verified with my PGP key.

These binaries are based on a libarchive.a which is compiled with
libzstd.a support, so that works too.

-- 
Eli Schwartz
Bug Wrangler and Trusted User

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 1601 bytes
Desc: OpenPGP digital signature
URL: <https://lists.archlinux.org/pipermail/arch-dev-public/attachments/20190325/d50727e4/attachment.sig>