On 29/3/20 8:52 pm, Evangelos Foutras wrote:

If I see a SIGILL on my AMD Phenom II X6 1090T then Arch will have failed me. 😜

I believe your proposal should only be discussed as co-existing optimized port(s) and even then I'm not sure it's worth the trouble. Performance-critical applications can and frequently are optimized for the running processor (I'm thinking of stuff like glibc and ffmpeg here).

AVX2 was a bold choice, and really a place to get a discussion started. We are currently supporting processors from 2003. We can be better than that. A

On 29/3/20 11:17 pm, Filipe Laíns wrote:

I would also like to note that rebuilding everything with forced support for AVX2 or whatever won't have much effect. Most packages do not have workloads where it would make use sense to use these CPU extensions, and as such, GCC would not use them.

That assumes we just add AVX2. Whereas, requiring a CPU supporting AVX2 would bring other optimizations that would be used. As I replied earlier, AVX2 may be going too far. But is a good starting point for discussion. If that is too far, what could we accept? SSE4.2? AVX? Surely we can do better than pure x86_64. To have a separate architecture would require automated builds, which requires being able to sign packages automatically. And we have not achieved database signing in 9 years.... I'm looking for a boost that could be achieved now. Allan

On Sun, 2020-03-29 at 15:39 +0100, Filipe Laíns via arch-dev-public
wrote:
> I make the PKGBUILD build for 2
* I can make

Sorry, I am a little distracted today.
 
Filipe Laíns

I want to clarify what I am proposing. I would not be an entirely new architecture in the sense of i686, CPU extensions are not different architectures and shouldn't be treated as such. What I would for us to do is to create a x86-64-axv2, etc. that would complement x86-64. We would not add it as a target for all packages, just for the ones that make sense. For this pacman would have to support architecture priority. We could have something like this: Architecture = x86-64-axv2 x86-64 This means if a x86-64-axv2 package is available, it would be selected over the x86-64 one. That way we don't need to rebuild all packages. My point here is that to me it does not really make sense to drop support for older CPUs. We will have little benefit in newer CPUs. Projects that need the performance already dynamically choose the CPU extensions to use in the runtime -- they will work on all x86-64 CPUs. If this did in fact bring a relevant performance improvement, like the original mail let's me to believe (but this may be just me), I would be all for it, but that's not the case. On Sun, 2020-03-29 at 16:51 +0200, Morten Linderud via arch-dev-public wrote:

Well, how do you think we supported two architectures? Why do you think `extra-x86_64-build` is named the way it is?

The "problem" is that we have no intentions of building 1 package 4 times and keep things in sync by hand, it was tedious enough with i686, which was part of why it was dropped in the first place. Thus we want build-servers to do this for us.

Then automate it? Is there any reason why we can't have the tooling build all architectures for us? Why not have an `extra-build` helper that will call extra-$arch-build for all every architecture? This will have practically the same affect as my SIMD packages not. The only difference would be how people are consuming them. It would just work out of the box instead of them having to install the -avx2 variant.

Allan is going to have a hard time argueing that the minimal improvements is going to justify the absurd time we'll end up building things by hand, it's the crux of the problem essentially. I'm also sure he knows this.

Surely we can bikeshed about which architectures to support, what we should discuss is how we should accomplish the task in general.

...

Furthermore, if you do indeed whish to move this forward please present us with reasonable data. Take a few packages that would benefit from this, build them with the proposed architecture and show us benchmarks. I think it's gonna be very hard for you to find packages with considerable improvement but I might be wrong, please show me.

See last paragraph.

Which paragraph are you referring to? I will be taking a step back now. I will probably wait 1 or 2 days before replying. Feel free to reach me privately for a direct discussion if you want to. Cheers, Filipe Laíns

On Sun, Mar 29, 2020 at 04:25:48PM +0100, Filipe Laíns via arch-dev-public wrote:

I want to clarify what I am proposing.

I would not be an entirely new architecture in the sense of i686, CPU extensions are not different architectures and shouldn't be treated as such.

What I would for us to do is to create a x86-64-axv2, etc. that would complement x86-64. We would not add it as a target for all packages, just for the ones that make sense.

For this pacman would have to support architecture priority. We could have something like this:

Architecture = x86-64-axv2 x86-64

Couldn't this be as simple as having a package with avx2 (or whatever) extensions compiled in in a separate repository that takes precedence in pacman.conf? Thanks, -Santiago

[2020-03-29 16:25:48 +0100] Filipe Laíns via arch-dev-public:

What I would for us to do is to create a x86-64-axv2, etc. that would complement x86-64. We would not add it as a target for all packages, just for the ones that make sense.

For this pacman would have to support architecture priority. We could have something like this:

Architecture = x86-64-axv2 x86-64

I'd like to say why not but everything remains to be done, here. Whereas pacman and our toolchain have mature support for multiple architectures, and they have it today.

My point here is that to me it does not really make sense to drop support for older CPUs. We will have little benefit in newer CPUs.

Nothing is being dropped. Every CPU that does not support the new architecture can keep running the x86_64 packages they currently do.

Then automate it? Is there any reason why we can't have the tooling build all architectures for us? Why not have an `extra-build` helper that will call extra-$arch-build for all every architecture?

That would be awesome but the tooling does not yet exist. Personally I do not consider it terribly bothersome to build packages for multiple architectures like we did for i686 and x86_64. And I think it would be preferable to introduce a new architecture tomorrow than wait a few more months in the hope someone implements your proposed scheme. Cheers. -- Gaetan

On Mon, 2020-03-30 at 09:07 +1000, Allan McRae via arch-dev-public wrote:

SSE4.2 is 2008 for Intel, 2011 for AMD. Though I guess some processors were released without it for some time after that. AVX was released by both in 2011.

So why is one too far and the other not?

I was looking at some edge cases where the CPUs had AVX but no SSE4.2. Intel's website is also a bit unreliable with older CPUs, a lot of the cases it is not listing extensions when it should be.

No need for pacman support. Just add higher instruction set to a new repo and set that repo with higher priority.

Right, that works too. Although, we can't have the word "auto" in the arch definition automatically identifying and enabling the supported extensions.

But that involves developers choosing which packages to build with higher instruction sets, which requires extra developer time.

Well. The thing about these CPU extensions is that they require very specific use workloads to be useful. They are more efficient when performing the tasks they were design to do, but they have an higher power consumption, which generally results in the CPU lowering clocks. To make proper use of them, you usually need to build your algorithm around them. Just enabling it in GCC might bring some performance improvements but they are usually *very* minimal. Upstreams that benefit from these extensions will most of the time write their own kernels and provide runtime detection or add a switch in the build system. I don't believe this is too much of burden for packagers. 99.9% of the time, just building a package with SEE/AVX natively in GCC will not bring any relevant performance enhancement. Furthermore, building an extra optimized version makes it much more reasonable for us to choose AVX2, which will have a much higher real world impact than just building everything with SSE4.2 instead. With that said, if you still want to bump the minimum requirements, could please consider taking just a little bit of time to run some benchmarks on packages you think should get an improvement?

Ideally, we would just autobuild for more optimized architectures, but this requires auto-signing packages, which has not happened in the last decade (but may in this one...).

Picking an instruction set that is ~10 years old and making it the default for the distro seems a reasonable approach to me.

The instruction set might be 10 years old, but you have to look at the discontinuation date for CPU families which don't support it, and maybe add 1 or 2 years for the bulk of the inventory to be cleared out from stores. Cheers, Filipe Laíns