On 20.02.21 11:31 mike lojkovic via arch-general wrote:

zstd is even faster at decompressing at boot than lz4 now?

I wonder about this, too. Even the zstd guys don't claim that: https://github.com/facebook/zstd#benchmarks According to their benchmarks lz4 decompression is way faster. That's why I don't really understand the hype about zstd.

Am Sat, 20 Feb 2021 07:50:58 -0300 schrieb Giancarlo Razzolini via arch-general :

Em fevereiro 20, 2021 2:52 mike lojkovic via arch-general escreveu:

...

zstd is even faster at decompressing at boot than lz4 now?

I don't think you'll notice much difference with either one.

Regards, Giancarlo Razzolini

Depends on the system cpu power. On slow cpu systems lz4 is much faster when compressing the image and also decompressing. Though you hardly notice the decompression time difference. On slow discs with a fast cpu a better compression may turn this around. In many cases zstd compression gives a better compromise nowadays with a powerful enough cpu. On my older systems with fast discs but slow cpus I keep using lz4. -Andy

On 20/02/2021 16:17, NicoHood via arch-general wrote:

I was also wondering what gives the best speed. I got a boot partition with 250mb, wouldn't it make sense to use CAT aka no compression?

It's a balance between CPU usage and disk IO. cat requires more data to be read from disk, zstd requires more CPU time. Slow CPU and fast SSD? cat. Fast CPU and slow HDD? zstd.

On Fri, Feb 19, 2021 at 16:45:02 -0300, Giancarlo Razzolini via arch-general wrote:

If you're using lz4 now, you might want to try zstd. It's faster to generate images now.

I benchmarked it on my mkinitcpio image, and zstd with mkinitcpio's default compression level 19 (standard COMPRESSION_OPTIONS for zstd) is several orders of magnitude slower than lz4, only to gain ~30% extra compression. It is slower to decompress too, but less noticeably so. $ time lz4 -l initramfs-linux.img Compressed filename will be : initramfs-linux.img.lz4 Compressed 11714560 bytes into 6595816 bytes ==> 56.30% real 0m0.065s user 0m0.043s sys 0m0.019s $ time zstd -19 -T0 initramfs-linux.img initramfs-linux.img : 33.76% (11714560 => 3954632 bytes, initramfs-linux.img.zst) real 0m8.173s user 0m6.167s sys 0m0.187s $ rm initramfs-linux.img.zst $ time zstd -T0 initramfs-linux.img initramfs-linux.img : 40.92% (11714560 => 4793496 bytes, initramfs-linux.img.zst) real 0m0.127s user 0m0.130s sys 0m0.027s Sticking to lz4 here. Geert

I benchmarked it on my mkinitcpio image, and zstd with mkinitcpio's […] Though you have benchmarked a wrong thing. It’s decompression time that matters here, not compression. The image is compressed to make it load faster during boot and that’s the important metric here.

I did my own benchmarks, though. On my decade-old system lz4 is still faster than zstd during decompression: 0.117415s (s=0.005948) vs 0.229915s (s=0.027075s). However, looking deeper, the time spent in syscalls suggests a different thing. lz4, having a bigger file, spent 0.049630s (s=0.007560s), while zstd 0.038125s (s=0.011196). That is not enough to claim they are different or zstd is better than lz4. But you can see in which direction it goes, in particular if one assumes a faster CPU+RAM. Methodology: image compressed using lz4 and zstd (same options as Geert Hendrickx has used). Copied to 50 files to create a batch, flushed to disk. Decompression of each file into /dev/null timed separately with caches dropped between each batch. Each batch ran 4 times for the total sample size of 200 for each group.

Le 21/02/2021 à 02:56, Geert Hendrickx via arch-general a écrit :

On Sat, Feb 20, 2021 at 21:25:21 +0100, mpan via arch-general wrote:

...

...

I benchmarked it on my mkinitcpio image, and zstd with mkinitcpio's […] Though you have benchmarked a wrong thing. It’s decompression time that matters here, not compression. The image is compressed to make it load faster during boot and that’s the important metric here.

Compression time matters also, since on many systems an initramfs gets booted only once or a few times for every time one gets generated, and zstd -19 is ridiculously slow for larger files, However mkinitcpio v30 reverted to zstd's default compression level 3, which is much better.

Excepted that decompression is a blocking event for boot, while compression is async. Anyway, based on the benchmark published on the PR that reverted to the default level, using -8 seems the best tradeoff for fast compression and small size, and −18 if space really is an issue (or you’re reading the file from a floppy…). Also, on most systems nowadays we are talking about changes in the order of hundreds of milliseconds, while I take ~5s to type my entire LUKS passphrase on boot. So…

Le 21/02/2021 à 17:34, mike lojkovic via arch-general a écrit :

Those milliseconds matter. When you have a thousand processes starting up shaving off a couple fractions of a second on each helps you hit the less than 10 second boot time.

Well I don’t boot often enough for even say 30s to make a difference (and my boot time is actually dominated by 1. UEFI and 2. Passphrase input). Outside of being impressive, what does a <10s boot time actually achieves? It likely takes you more time to adopt a good posture in front of your computer while it is booting up. If we were talking 10s vs 2 min I’d understand, but we are barely talking 20s vs 10s…

A 10 second improvement is _large_. You may think that you don't need it because you're sitting at your computer, but think about servers, vms, automated systems... Every second makes things quicker. Moving from 20 seconds to 10 seconds is halving its boot time. That is a good thing. On Sun, Feb 21, 2021 at 08:17:28PM +0400, Archange via arch-general wrote:

Le 21/02/2021 à 17:34, mike lojkovic via arch-general a écrit :

...

Those milliseconds matter. When you have a thousand processes starting up shaving off a couple fractions of a second on each helps you hit the less than 10 second boot time.

Well I don’t boot often enough for even say 30s to make a difference (and my boot time is actually dominated by 1. UEFI and 2. Passphrase input). Outside of being impressive, what does a <10s boot time actually achieves? It likely takes you more time to adopt a good posture in front of your computer while it is booting up. If we were talking 10s vs 2 min I’d understand, but we are barely talking 20s vs 10s…

-- Kevin Morris

---------------------------------------- From: Kevin Morris via arch-general Sent: Mon Feb 22 02:12:20 CET 2021 To: General Discussion about Arch Linux Cc: Kevin Morris Subject: Re: [arch-general] [arch-announce] Moving to Zstandard images by default on mkinitcpio

A 10 second improvement is _large_. You may think that you don't need it because you're sitting at your computer, but think about servers, vms, automated systems... Every second makes things quicker. Moving from 20 seconds to 10 seconds is halving its boot time.

That is a good thing.

I agree however someone benchmark showed it's 0.1s not 10s difference (0.117415s vs 0.229915s). Yours sincerely G. K.

Em fevereiro 22, 2021 10:23 JustKidding via arch-general escreveu:

Do I have to wait for kernel 5.11 in order to use zstd?

No, zstd is supported since kernel 5.9.

Em fevereiro 22, 2021 10:53 Damjan Georgievski via arch-general escreveu:

...

Em fevereiro 22, 2021 10:23 JustKidding via arch-general escreveu:

...

Do I have to wait for kernel 5.11 in order to use zstd?

No, zstd is supported since kernel 5.9.

The unfortunate consequence of this is that currently Arch only has 5.10 kernels available, and a mkinitcpio that by default only works with >5.9.

But people on Haswell CPUs had to have 5.4 pinned, since any later kernel was crashing the GPU, so the mkinitcpio change will break their boot unless they are extra/doubly careful.

Well, this is why this announcement was made as well as some buffer period for people to accommodate the needed changes. Regards, Giancarlo Razzolini