On 3/24/19 9:18 PM, Baptiste Jonglez wrote:

Just one detail: your results for -19, -20 and -21 are identical because apparently zstd needs an additional flag (--ultra) to "unlock" the higher compression levels:

zstd -c -T0 -20 - Warning : compression level higher than max, reduced to 19

Also, I see you did not test zstd with a small number of cores: can you add e.g. -T1, -T2 and -T4 to the comparison? It would give a more realistic idea of what to expect when building on a typical machine, as opposed to dragon ;) In my tests, using less threads also decreased memory usage when compressing (35% less memory when switching from -T2 to -T1).

Damn, i knew i must've missed something - archange had already mentioned on IRC that these results look weird, but i shrugged it off. Should've double-checked. I'll get you a new table with the higher levels fixed and a second set with -T2 for comparison later. Regardless, IIRC preliminary testing showed that these gains are not worth it, as they were quite small in the tests we ran a while ago.

For decompression, it seems that both xz and zstd run single-threaded, so there's not much to think about (zstd is just incredibly fast).

Correct Rob

On Sun, 24 Mar 2019 at 23:45, Allan McRae via arch-dev-public wrote:

On 25/3/19 4:34 am, Robin Broda via arch-dev-public wrote:

...

This change requires a new pacman release, as as of writing this, zstd support is in master but hasn't landed in a release yet.

Which is a complete blocker for quite a period of time.

We need to assume every system has a copy of pacman-5.2+ before we can switch packages to zstd.

Why is pacman support needed here? I can already install .zstd packages using pacman 5.1.3. The crucial part seems to be libarchive support, which was added in v3.3.3 (~ September 2018).

On 03/25/19 at 12:15am, Robin Broda via arch-dev-public wrote:

On 3/24/19 11:20 PM, Evangelos Foutras via arch-dev-public wrote:

...

On Sun, 24 Mar 2019 at 23:45, Allan McRae via arch-dev-public wrote:

...

On 25/3/19 4:34 am, Robin Broda via arch-dev-public wrote:

...

This change requires a new pacman release, as as of writing this, zstd support is in master but hasn't landed in a release yet.

Which is a complete blocker for quite a period of time.

We need to assume every system has a copy of pacman-5.2+ before we can switch packages to zstd.

Why is pacman support needed here? I can already install .zstd packages using pacman 5.1.3.

The crucial part seems to be libarchive support, which was added in v3.3.3 (~ September 2018).

Yes, installing zstd packages works - the pacman release is merely required for makepkg. Unless that has already landed too, which would be news to me :)

Thus i don't think we need a hold-off period like this, Allan.

We still need a hold-off period, we're just waiting to make sure people have libarchive v3.3.3 instead of pacman v5.2.0.

On 25/3/19 9:28 am, Robin Broda via arch-dev-public wrote:

On 3/25/19 12:22 AM, Andrew Gregory wrote:

...

On 03/25/19 at 12:15am, Robin Broda via arch-dev-public wrote:

...

On 3/24/19 11:20 PM, Evangelos Foutras via arch-dev-public wrote:

...

On Sun, 24 Mar 2019 at 23:45, Allan McRae via arch-dev-public wrote:

...

We need to assume every system has a copy of pacman-5.2+ before we can switch packages to zstd.

Why is pacman support needed here? I can already install .zstd packages using pacman 5.1.3.

The crucial part seems to be libarchive support, which was added in v3.3.3 (~ September 2018).

Yes, installing zstd packages works - the pacman release is merely required for makepkg. Unless that has already landed too, which would be news to me :)

Thus i don't think we need a hold-off period like this, Allan.

We still need a hold-off period, we're just waiting to make sure people have libarchive v3.3.3 instead of pacman v5.2.0.

That update happened half a year ago, i'm sure that most people with an installation that old will already have to fetch other packages, like the keyring, separately for it to go through.

Fetching a keyring does not potentially bump sonames.

Plus, with libarchives' release cycle, i don't think that libarchive itself is gonna be rebuilt immediately after the change is implemented - providing extra time to upgrade libarchive without having to download a release packed as xz separately.

And if openssl gets and soname bump? A

On Sun, Mar 24, 2019 at 04:39:54PM -0700, Andrew Gregory via arch-dev-public wrote:

I don't consider hoping that libarchive doesn't need a rebuild in the near future a great strategy. That being said, this is really a question of how long of a period we need between libarchive v3.3.3 and us making the switch. I'm not a packager, so I don't have much of an opinion on that.

Well, we pride ourselves with having competent users. I think waiting a year is conservative and safe. However, personally I think we can wait for the next pacman release and write an announcment. Then we give everyone a month to update and we can have a smooth transition. Assuming of course that everyone is on-board with this change. I would like to get some opinions from packaging devs with experiences. -- Morten Linderud PGP: 9C02FF419FECBE16

[2019-03-25 00:46:15 +0100] Morten Linderud via arch-dev-public:

On Sun, Mar 24, 2019 at 04:39:54PM -0700, Andrew Gregory via arch-dev-public wrote:

...

I don't consider hoping that libarchive doesn't need a rebuild in the near future a great strategy. That being said, this is really a question of how long of a period we need between libarchive v3.3.3 and us making the switch. I'm not a packager, so I don't have much of an opinion on that.

Well, we pride ourselves with having competent users. I think waiting a year is conservative and safe. However, personally I think we can wait for the next pacman release and write an announcment. Then we give everyone a month to update and we can have a smooth transition. Assuming of course that everyone is on-board with this change.

So far we all seem to agree it's a change for the better. However your timeline is confused: we only need to wait for a new pacman release to start building zstd-compressed packages; we can then push them to the repos straight away, assuming users have had enough time to update libarchive-3.3.3. It's already been in [core] for more than six months. Traditionally we wait a year before pushing changes that break backward compatibility. That always seemed a bit extreme to me so I'd personally be fine with doing the switch in a month or two with certain precautions: - Post an announcement warning users they'll need libarchive-3.3.3 or higher a month from now and telling them to update if they haven't done so in the last six months. - Prepare a static build of libarchive-3.3.3 compressed with xz and write a wiki page with detailed instructions on how to manually switch from an old system (for users who might want to switch even later). Cheers. -- Gaetan

Morten Linderud via arch-dev-public on Mon, 2019/03/25 00:46:

On Sun, Mar 24, 2019 at 04:39:54PM -0700, Andrew Gregory via arch-dev-public wrote:

...

I don't consider hoping that libarchive doesn't need a rebuild in the near future a great strategy. That being said, this is really a question of how long of a period we need between libarchive v3.3.3 and us making the switch. I'm not a packager, so I don't have much of an opinion on that.

Well, we pride ourselves with having competent users. I think waiting a year is conservative and safe. However, personally I think we can wait for the next pacman release and write an announcment. Then we give everyone a month to update and we can have a smooth transition. Assuming of course that everyone is on-board with this change.

I am in with this. Who ever runs a rolling release distribution and does not update within half a year did a bad decision. So let's go for it. -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}

On 3/25/19 1:38 AM, Evangelos Foutras via arch-dev-public wrote:

On Mon, 25 Mar 2019 at 01:22, Jan Alexander Steffens via arch-dev-public wrote:

...

When we implement this, I would say we go with "zstd -c -T0 -" in pacman's makepkg.conf and "zstd -C -T0 -18 -" in the configs shipped with devtools.

I think users that build their own local packages are more likely to benefit from fast compression. Anyone building with makechrootpkg for distribution gets the high compression level.

That's actually really smart! We can also leave out "-T0" since the default compression level is very fast anyway. Plus, it's already implemented in this way in pacman git so we don't have to touch anything there. (But, if it were to be multithreaded there as well, I would replace zstd with zstdmt; same for devtools.)

That's why the subject line is prefixed with '(devtools)' :) What's unclear to me is the difference between zstd -T0 and zstdmt, however.

As far as compression level goes, I believe we should select the highest one that doesn't have increased memory requirements during decompression. So that would be -19. Robin makes a good case about -18; looking at upstream's "Compression Speed vs Ratio" graph [1] I would say -18 is preferable to -19 if we are concerned about compression speed and memory usage (my totally unscientific measurements show a 25% memory increase and 20% speed decrease going from -18 to -19 when using -T4). That said, I might still opt for -19 due to the slightly higher compression ratio; memory usage isn't too big of an issue and the slower speed is mitigated by multithreading (i.e.: it will still be much faster than xz).

I do think that at -19+, memory usage becomes a bigger issue. The difference between -18 and -19 on cuda is almost a gigabyte! While not really a problem for our beefy build boxes, some 4- or even 8-GB developer machines could really suffer from such an incline in memory usage. Thus i stand by -18 being the more sensible choice. Rob

On 3/24/19 8:38 PM, Evangelos Foutras via arch-dev-public wrote:

On Mon, 25 Mar 2019 at 01:22, Jan Alexander Steffens via arch-dev-public wrote:

...

On Sun, Mar 24, 2019 at 7:35 PM Robin Broda via arch-dev-public wrote:

...

The required changeset is, i think: PKGEXT='.pkg.tar.zst' COMPRESSZST=(zstd -c -T0 -18 -)

When we implement this, I would say we go with "zstd -c -T0 -" in pacman's makepkg.conf and "zstd -C -T0 -18 -" in the configs shipped with devtools.

I think users that build their own local packages are more likely to benefit from fast compression. Anyone building with makechrootpkg for distribution gets the high compression level.

That's actually really smart! We can also leave out "-T0" since the default compression level is very fast anyway. Plus, it's already implemented in this way in pacman git so we don't have to touch anything there. (But, if it were to be multithreaded there as well, I would replace zstd with zstdmt; same for devtools.)

As far as compression level goes, I believe we should select the highest one that doesn't have increased memory requirements during decompression. So that would be -19. Robin makes a good case about -18; looking at upstream's "Compression Speed vs Ratio" graph [1] I would say -18 is preferable to -19 if we are concerned about compression speed and memory usage (my totally unscientific measurements show a 25% memory increase and 20% speed decrease going from -18 to -19 when using -T4). That said, I might still opt for -19 due to the slightly higher compression ratio; memory usage isn't too big of an issue and the slower speed is mitigated by multithreading (i.e.: it will still be much faster than xz).

Assuming .zst packages have been installable as far back as September 2018 when libarchive 3.3.3 was released, it seems to me that the following steps can be taken:

1) Check if repo-add and dbscripts recognize .zst packages.

repo-add checks to see if bsdtar/libarchive recognizes the file as a compressed archive containing a .PKGINFO file, and therefore, repo-add will work whenever pacman works. dbscripts whitelists the known package extensions, and I will be adding new extensions to dbscripts in tandem with a stable pacman release that contains makepkg support, so that should not be a problem either.

2) Add "COMPRESSZST=(zstdmt -19 -c -z -q -)" to devtools' makepkg-x86_64.conf.

Or just sync it with the pacman package. :p

3) Release a test package and confirm that it's installable. (Possibly also test with an old installation from September 2018.)

You can confirm that today if you build using makepkg from pacman-git. Alternatively, try this test package I built: https://pkgbuild.com/~eschwartz/repo/x86_64/testpkg-foobar-1-1-any.pkg.tar.z...

4) Announce the transition to the new compression algorithm and provide a date-stamped mirror URL [2] for really old installations without libarchive 3.3.3.

We've had such transitions before, e.g. when adding hook support. IMO the transition does not need to be longer than a month, as we are substantially saying "your libarchive must be from the last six months". Arch Linux does not, AFAIK, have a general policy to support systems that have failed to update in 6 months. If anyone manages to break their system by having a very old libarchive version after a warning period provided as a news entry, we do not need to officially support anything... but I provide fully static recovery binaries for pacman, here: https://aur.archlinux.org/packages/pacman-static This should be sufficient without recourse to legacy compat packages for libarchive, or requiring users to reset their system to any give ALA date. (Notwithstanding this, many things might break in that time frame, and general advice usually seems to be to upgrade in stages using the ALA anyway.) They are available both as the (prebuilt custom repo) package "pacman-static", and as extracted binaries that can be downloaded and run directly without the need to run pacman or even decompress the file. They are verified with my PGP key. These binaries are based on a libarchive.a which is compiled with libzstd.a support, so that works too. -- Eli Schwartz Bug Wrangler and Trusted User

Hello everyone, Now that Zstd 1.4.4 has been out, and released into our repos as well, i think it's time for a new status report on this. I re-ran the benchmarks with the new zstd, and we are hitting marginally better times in compression & decompression in all scenarios. In the past few weeks, other team members and I have taken a look at our own projects, and what would be needed to finally transition to zstd. Notable progress: - A news post was made by eworm, indicating that users should upgrade if they haven't done so since September 2018 (https://www.archlinux.org/news/required-update-to-recent-libarchive/) - dbscripts has been updated by eschwartz to support zstd What still needs to be done: - infrastructure: - nginx rewrite rules hardcode xz PKGEXT https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc49166... - archive config(?) hardcodes PKGEXT https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc49166... - potentially more things, someone from devops should take a look - archivetools: https://github.com/archlinux/archivetools/issues/6 - namcap? some files refer to a hardcoded .xz, though jelle concluded that this is irrelevant - someone with namcap knowledge should look into this - srcpac (is this still used?) - hardcodes 'pkg.tar.?z' - kde-build hardcodes pkg.tar.xz https://git.archlinux.org/kde-build.git/tree/build-packages?id=cda04698f4064... - devtools: - hardcodes 'pkg.tar?(.?z)' https://git.archlinux.org/devtools.git/tree/lib/common.sh?id=2c611d20bdd04fe... - more than one occurrence of this iirc There might be things I've missed. I encourage every Arch Linux project maintainer to check their own code for hardcoded xz extensions, you probably know your code best. As soon as these things are out of the way, we can proceed with the proposal. The changeset proposal remains on these settings: PKGEXT='.pkg.tar.zst' COMPRESSZST=(zstd -c -T0 --ultra -20 -) -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

Hello again, We are on our way to getting zstd merged. The planned merge window for this is **2019/12/27** at around 20:00 Europe/Berlin time. Several team members will be meeting at the 36c3, so we figured this is a good opportunity to do this, as it makes sure that several people are aware, ready, and together for this - should anything immediately explode. This has been ACK'd by anthraxx. Foxboron is also aware. I have been mentioning this on IRC, and discussed it with anthraxx via e-mail. This mail serves to formalize the plans. **There is one hard requirement on the merge happening:** All critical Arch Linux production infrastructure needs to be ready. The known-unsolved problems are labelled below, please see to them if any of you can. I will also take a look myself, but I do not have much involvement with these parts of the puzzle, so I would really appreciate someone with more in-depth knowledge taking a look. If the known issues are solved and nothing major comes up, the deployment will happen as planned. On 12/8/19 1:39 PM, Robin Broda via arch-dev-public wrote:

What still needs to be done: - infrastructure: - nginx rewrite rules hardcode xz PKGEXT https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc49166... - archive config(?) hardcodes PKGEXT https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc49166...

Unsolved: This needs fixing. I would *really* like someone from devtools to take a look at this, but if unsolved i will also attempt to fix it.

- archivetools: https://github.com/archlinux/archivetools/issues/6

Unsolved: This needs fixing. If nobody else gets to it, I'll see to it myself.

- namcap? some files refer to a hardcoded .xz, though jelle concluded that this is irrelevant

Solved: This is indeed irrelevant (see Eli's reply)

- srcpac (is this still used?)

Solved: This is indeed a dead project (see Eli's reply)

- kde-build hardcodes pkg.tar.xz https://git.archlinux.org/kde-build.git/tree/build-packages?id=cda04698f4064...

Solved: arojas said this will be fixed after zstd is deployed. This is also non-critical.

- devtools: Solved: Patches ready, will be merged with the switcharoo.

I encourage every Arch Linux project maintainer to check their own code for hardcoded xz extensions, you probably know your code best. Please -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

On 12/14/19 7:58 PM, Robin Broda via arch-dev-public wrote:

On 12/8/19 1:39 PM, Robin Broda via arch-dev-public wrote:

...

What still needs to be done: - infrastructure: - nginx rewrite rules hardcode xz PKGEXT https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc49166... - archive config(?) hardcodes PKGEXT https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc49166...

Unsolved: This needs fixing. I would *really* like someone from devtools to take a look at this, but if unsolved i will also attempt to fix it.

Gah, s/devtools/devops/ -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

Em dezembro 15, 2019 15:22 Robin Broda via arch-dev-public escreveu:

All known issues have been dealt with. We are waiting on the patches to land, thus: - https://www.archlinux.org/packages/extra/any/namcap/ being upgraded to 3.2.9 - https://github.com/archlinux/archivetools/pull/8 being merged - https://github.com/coderobe/infrastructure/commit/7155521ed6f823ac6c6d06d1b9... being picked into infrastructure.git - this *depends on* the archivetools patch - archive-cleaner is not mission-critical, it does not need to be patched before the scheduled deployment - Foxboron is working on a patch

These can all be dealt with during the deployment if they do not happen beforehand anyways. This means the deployment will happen as planned.

Hi All, We seem to be getting .zst packages just fine. The following packages are on zstd format already: git kdevelop remind ipython chromium exim libva perl-xml-sax clang llvm llvm lib32-llvm nodejs ldc youtube-viewer cuda xfce4-terminal qt5-tools llvm lib32-llvm mpv qtcreator jenkins texstudio python-keyring ldc lldb you-get openmp python-keyrings-alt log4cplus acorn libfm-qt min matrix-synapse lld udftools v2ray compiler-rt trojan netcdf-openmpi shiboken2 shiboken2 python-parameterized parity-ethereum cbindgen ibm-sw-tpm2 golang-gopkg-check.v1 golang-golang-x-sys golang-golang-x-crypto golang-golang-x-image python-cfn-lint python-dephell mill python-gdspy tpm2-totp python-poetry ruby-public_suffix ruby-ffi devtools python-zstandard terraform-provider-keycloak wishbone-utils s-nail python-rdflib-jsonld libarchive grub openvpn Also, this is the order they were added to the repositories, so, kudos Christian for sending our very first .zst package. We haven't had any complaints either, so I think everything is good. Regards, Giancarlo Razzolini

Em dezembro 31, 2019 12:08 Andrew Crerar escreveu:

On 12/31/19 10:04 AM, Giancarlo Razzolini via arch-dev-public wrote:

Just an addendum, the ordering on my previous email was wrong. The first zst package we got (after devtools, ofc) was texstudio. So, it's Sven that should get the kudos. Regards, Giancarlo Razzolini