[arch-dev-public] RFC: (devtools) Changing default compression method to zstd

[Robin Broda]

Hello everyone,

Now that Zstd 1.4.4 has been out, and released into our repos as well, i think it's time for a new status report on this.

I re-ran the benchmarks with the new zstd, and we are hitting marginally better times in compression & decompression in all scenarios.

In the past few weeks, other team members and I have taken a look at our own projects, and what would be needed to finally transition to zstd.
Notable progress:
- A news post was made by eworm, indicating that users should upgrade if they haven't done so since September 2018 (https://www.archlinux.org/news/required-update-to-recent-libarchive/)
- dbscripts has been updated by eschwartz to support zstd

What still needs to be done:
- infrastructure:
  - nginx rewrite rules hardcode xz PKGEXT https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc4916629ac3a74d1c49/roles/archive/templates/nginx.d.conf.j2#L32-L35
  - archive config(?) hardcodes PKGEXT https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc4916629ac3a74d1c49/roles/archive/templates/archive.conf.j2#L15
  - potentially more things, someone from devops should take a look
- archivetools: https://github.com/archlinux/archivetools/issues/6
- namcap? some files refer to a hardcoded .xz, though jelle concluded that this is irrelevant
  - someone with namcap knowledge should look into this
- srcpac (is this still used?)
  - hardcodes 'pkg.tar.?z'
- kde-build hardcodes pkg.tar.xz https://git.archlinux.org/kde-build.git/tree/build-packages?id=cda04698f4064cb87d427ccb3bdf954f127c65f7#n35
- devtools:
  - hardcodes 'pkg.tar?(.?z)' https://git.archlinux.org/devtools.git/tree/lib/common.sh?id=2c611d20bdd04feae6ab32a7ef8947aeb4118604#n155
  - more than one occurrence of this iirc

There might be things I've missed.
I encourage every Arch Linux project maintainer to check their own code for hardcoded xz extensions, you probably know your code best.


As soon as these things are out of the way, we can proceed with the proposal.

The changeset proposal remains on these settings:
PKGEXT='.pkg.tar.zst'
COMPRESSZST=(zstd -c -T0 --ultra -20 -)

-- 
Rob (coderobe)

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.archlinux.org/pipermail/arch-dev-public/attachments/20191208/24366808/attachment.sig>