[arch-dev-public] RFC: (devtools) Changing default compression method to zstd

[Eli Schwartz]

On 12/8/19 7:39 AM, Robin Broda via arch-dev-public wrote:
> Hello everyone,
> 
> Now that Zstd 1.4.4 has been out, and released into our repos as well, i think it's time for a new status report on this.
> 
> I re-ran the benchmarks with the new zstd, and we are hitting marginally better times in compression & decompression in all scenarios.
> 
> In the past few weeks, other team members and I have taken a look at our own projects, and what would be needed to finally transition to zstd.
> Notable progress:
> - A news post was made by eworm, indicating that users should upgrade if they haven't done so since September 2018 (https://www.archlinux.org/news/required-update-to-recent-libarchive/)
> - dbscripts has been updated by eschwartz to support zstd
> 
> What still needs to be done:
> - infrastructure:
>   - nginx rewrite rules hardcode xz PKGEXT https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc4916629ac3a74d1c49/roles/archive/templates/nginx.d.conf.j2#L32-L35
>   - archive config(?) hardcodes PKGEXT https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc4916629ac3a74d1c49/roles/archive/templates/archive.conf.j2#L15
>   - potentially more things, someone from devops should take a look
> - archivetools: https://github.com/archlinux/archivetools/issues/6
> - namcap? some files refer to a hardcoded .xz, though jelle concluded that this is irrelevant
>   - someone with namcap knowledge should look into this

namcap is indeed irrelevant. It hardcodes 'xz' in three places:
- the README
- the makepkg.conf used for building packages within the namcap
  self-test suite
- the case statement in the main script, which detects files ending in
  .xz, .zst, and other valid extensions, in order to choose the right
  decompressor before finally handing the uncompressed .tar to namcap
  itself

> - srcpac (is this still used?)
>   - hardcodes 'pkg.tar.?z'

Definitely not still used. :p

It hasn't been developed in 5 years. It also still hardcodes the `abs`
program.

> - kde-build hardcodes pkg.tar.xz https://git.archlinux.org/kde-build.git/tree/build-packages?id=cda04698f4064cb87d427ccb3bdf954f127c65f7#n35
> - devtools:
>   - hardcodes 'pkg.tar?(.?z)' https://git.archlinux.org/devtools.git/tree/lib/common.sh?id=2c611d20bdd04feae6ab32a7ef8947aeb4118604#n155
>   - more than one occurrence of this iirc
> 
> There might be things I've missed.
> I encourage every Arch Linux project maintainer to check their own code for hardcoded xz extensions, you probably know your code best.
> 
> 
> As soon as these things are out of the way, we can proceed with the proposal.
> 
> The changeset proposal remains on these settings:
> PKGEXT='.pkg.tar.zst'
> COMPRESSZST=(zstd -c -T0 --ultra -20 -)
> 


-- 
Eli Schwartz
Bug Wrangler and Trusted User

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 1601 bytes
Desc: OpenPGP digital signature
URL: <https://lists.archlinux.org/pipermail/arch-dev-public/attachments/20191208/55d96705/attachment.sig>