On 3/16/19 1:30 AM, Adam Fontenot via arch-general wrote:

On Fri, Mar 15, 2019 at 11:10 PM Darren Wu via arch-general wrote:

...

I have 4Mbps (512KBytes/s) 'broad'band and i7-6500U CPU. I wanna cry.

Even in a worst-case scenario like this one, the Squash compression test I linked to shows an increase from 26 secs (xz -6) to 29 secs (zstd -19), so you wouldn't be significantly impacted by this change.

I hope your local authorities decide to give you real broadband in the near future, however. :-)

My situation is similar to Darren's: My primary connection to the internet is through my cell phone carrier and a mobile WiFi hot spot. In urban areas, I can get as much as 50 megabits per second, but presently, due to my remote location, it's around 5 or 6. I also have a monthly data cap, which I share with my wife, and only WiFi (i.e., no wires; that nice 300 megabits from hot spot to device is shared by all devices, and there's a per device limit, too). FWIW, I have an i7-7700HQ CPU. In the old days (when large files were a megabyte or two and network bandwidth was measured in kilibits per second), we assumed that the network was the bottleneck. I think what Adam is propsing is that things are different now, and that the CPU is the bottleneck. As always, it depends. :-) My vote, whether it has any weight or not, is for higher compression ratios at the expense of CPU cycles when decompressing; i.e., xz rather than zstd. Also, consider that the 10% increase in archive size is suffered repeatedly as servers store and propagate new releases, but that the increase in decompression time is only suffered by the end user once, likely during a manual update operation or an automated background process, where it doesn't matter much. I used to have this argument with coworkers over build times and wake-from-sleep times. Is the extra time to decompress archives really killing anyone's productivity? Are users choosing OS distros based on how long it takes do install Open Office? Are Darren and I dinosaurs, doomed to live in a world where everyone else has a multi-gigabit per second internet connection and a cell phone class CPU? Jokingly, but not as much as you think, Dan

On 16/03/2019 13:24, Ralph Corderoy wrote:

suggest replacing xz with lzip as xz has quite a few flaws in its file format.

I seem to remember that this has been debunked/ruled out as irrelevant to package distribution every time it has been proposed to Debian, e.g. it even has worse decompression times then xz. https://lists.debian.org/debian-devel/2015/07/msg00377.html https://www.reddit.com/r/linux/comments/58swzc/xz_format_inadequate_for_long...

On Sat, Mar 16, 2019 at 5:01 AM Dan Sommers <2QdxY4RzWzUUiLuE@potatochowder.com> wrote:

My situation is similar to Darren's: My primary connection to the internet is through my cell phone carrier and a mobile WiFi hot spot. In urban areas, I can get as much as 50 megabits per second, but presently, due to my remote location, it's around 5 or 6. I also have a monthly data cap, which I share with my wife, and only WiFi (i.e., no wires; that nice 300 megabits from hot spot to device is shared by all devices, and there's a per device limit, too). FWIW, I have an i7-7700HQ CPU.

In the old days (when large files were a megabyte or two and network bandwidth was measured in kilibits per second), we assumed that the network was the bottleneck. I think what Adam is propsing is that things are different now, and that the CPU is the bottleneck. As always, it depends. :-)

My vote, whether it has any weight or not, is for higher compression ratios at the expense of CPU cycles when decompressing; i.e., xz rather than zstd. Also, consider that the 10% increase in archive size is suffered repeatedly as servers store and propagate new releases, but that the increase in decompression time is only suffered by the end user once, likely during a manual update operation or an automated background process, where it doesn't matter much.

I used to have this argument with coworkers over build times and wake-from-sleep times. Is the extra time to decompress archives really killing anyone's productivity? Are users choosing OS distros based on how long it takes do install Open Office? Are Darren and I dinosaurs, doomed to live in a world where everyone else has a multi-gigabit per second internet connection and a cell phone class CPU?

Jokingly, but not as much as you think, Dan

I think you're overstating your case a little bit. In the United States, nothing less than 25 Mbps can legally be called broadband, and the average download speed is approaching 100 Mbps (90% of us have access to 25 Mbps or better internet). Zstd -19 is faster overall than xz -6 starting at around 20 Mbps, so it's a better choice even on some sub-broadband connections. Your PassMark score is only about 50% better than that used on the Squash compression test, so I don't know that the computer speed element is significant. Furthermore, if space saving is the primary concern, why are we using the default xz -6 option, rather than something stronger like -9? I support using zstd because even in the absolute worst case (instant decompression), you're looking at less than a 10% increase in upgrade time, while for most users, a reduction of 50% would not be atypical (lzma is slow!). I'm not suggesting throwing out all concerns about disk space and transfer time, I'm just suggesting that times have changed *somewhat*, and that for most users zstd may provide a better trafe-off. In my case (100 Mbit connection), which is close to the US average, downloading and decompressing the latest Firefox package would take less than 1/3 the time it currently takes if we switched to zstd. Adam

On Sat, 16 Mar 2019 07:11:31 -0500 Dan Sommers <2QdxY4RzWzUUiLuE@potatochowder.com> wrote:

On 3/16/19 7:03 AM, pete via arch-general wrote:

...

What would be the Reason for the change to a slower less efficient method apart from it may be someones pet toy ..

It's only less efficient because networks are more faster than CPUs. If I had a *multi-terabit* per second network, and multi-*kilohertz* 8-bit CPUs on both ends, then zstd might be more efficient for the whole compress, transmit, decompress cycle.

Arch can potentially control the CPU at the servers, but neither the network(s) nor the user's CPUs, which means that I'm agreeing with your conclusion rather than your premise. :-)

Yeah, it's the CPU processing itself that's the bottle neck, and more importantly, one CPU cache miss and it's taking that flight from Pluto to the center of the sun and back. All to retrieve some forgotten data.