[arch-dev-public] RFC: (devtools) Changing default compression method to zstd

[Robin Broda]

On 3/25/19 1:38 AM, Evangelos Foutras via arch-dev-public wrote:
> On Mon, 25 Mar 2019 at 01:22, Jan Alexander Steffens via
> arch-dev-public <arch-dev-public at archlinux.org> wrote:
>>
>> When we implement this, I would say we go with "zstd -c -T0 -" in
>> pacman's makepkg.conf and "zstd -C -T0 -18 -" in the configs shipped
>> with devtools.
>>
>> I think users that build their own local packages are more likely to
>> benefit from fast compression. Anyone building with makechrootpkg for
>> distribution gets the high compression level.
> 
> That's actually really smart! We can also leave out "-T0" since the
> default compression level is very fast anyway. Plus, it's already
> implemented in this way in pacman git so we don't have to touch
> anything there. (But, if it were to be multithreaded there as well, I
> would replace zstd with zstdmt; same for devtools.)

That's why the subject line is prefixed with '(devtools)' :)
What's unclear to me is the difference between zstd -T0 and zstdmt, however.


> 
> As far as compression level goes, I believe we should select the
> highest one that doesn't have increased memory requirements during
> decompression. So that would be -19. Robin makes a good case about
> -18; looking at upstream's "Compression Speed vs Ratio" graph [1] I
> would say -18 is preferable to -19 if we are concerned about
> compression speed and memory usage (my totally unscientific
> measurements show a 25% memory increase and 20% speed decrease going
> from -18 to -19 when using -T4). That said, I might still opt for -19
> due to the slightly higher compression ratio; memory usage isn't too
> big of an issue and the slower speed is mitigated by multithreading
> (i.e.: it will still be much faster than xz).
> 
I do think that at -19+, memory usage becomes a bigger issue.
The difference between -18 and -19 on cuda is almost a gigabyte!

While not really a problem for our beefy build boxes, some 4- or even 8-GB developer
machines could really suffer from such an incline in memory usage.
Thus i stand by -18 being the more sensible choice.


Rob

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.archlinux.org/pipermail/arch-dev-public/attachments/20190325/c2f4fe1f/attachment.sig>