[arch-dev-public] Starting x86_64_v3 port

newer
[arch-dev-public] On the toolchain...

older
[arch-dev-public] [RFC] archweb...

Allan McRae

29 Jan 2022 29 Jan '22

12:12 a.m.

Hi all, You may remember long in the past we discussed adding an x86_64_v3 port. From memory, pkgstats shows this will benefit ~2/3 (or 3/4?) of our users! So lets get this underway! Apart from tooling (devtools/dbscripts), we need to make some decisions. My plan is to seed the x86_64_v3 repos with x86_64 packages and then they get replaced as updates/rebuilds happen (potentially supplemented with a rebuild party). Pacman 6.0 can handle this. This is not a perfectly clean new port, but is substantially less of a burden than releasing a pure x86_64_v3 port. Our .BUILDINFO files do record the package architecture, so this mixture should not affect (e.g.) our reproducible builds etc. The decision to be made is who will package for this repo? I think these are the options: A) packagers upload both x86_64 and x86_64_v3 to the repos. Our build server will help those without x86_64_v3 machines. B) we recruit some packagers to build the x86_64_v3 packages. C) Some combination of A+B. My understanding is our x86_64 port started with B, then C, then A. I think with our build infrastructure now, we can start with A, but that is more of a burden for packagers. I doubt it will be much of a burden as x86_64_v3 specific build issues are unlikely. Is there any particular objection to requiring packagers upload both architectures? Allan

Show replies by date

Morten Linderud

29 Jan 29 Jan

1:13 a.m.

On Sat, Jan 29, 2022 at 10:12:30AM +1000, Allan McRae via arch-dev-public wrote:

...

Is there any particular objection to requiring packagers upload both architectures?

I'm personally not really motivated doing the required builds. We have an underdeveloped infrastructure which hasn't changed since we abandoned i686 5 years ago. I'd personally like to see more work on our build tooling before we commit to new architectures. -- Morten Linderud PGP: 9C02FF419FECBE16

Allan McRae

1:22 a.m.

On 29/1/22 11:13, Morten Linderud via arch-dev-public wrote:

...

On Sat, Jan 29, 2022 at 10:12:30AM +1000, Allan McRae via arch-dev-public wrote:

...
Is there any particular objection to requiring packagers upload both architectures?

I'm personally not really motivated doing the required builds. We have an underdeveloped infrastructure which hasn't changed since we abandoned i686 5 years ago.

I'd personally like to see more work on our build tooling before we commit to new architectures.

FYI, it is a single extra command. Either of these will work... offload-build --arch x86_64_v3 extra-x86_64_v3-build Nothing else changes for the packager. Allan

Morten Linderud

1:28 a.m.

On Sat, Jan 29, 2022 at 11:22:32AM +1000, Allan McRae via arch-dev-public wrote:

...

On 29/1/22 11:13, Morten Linderud via arch-dev-public wrote:

...
On Sat, Jan 29, 2022 at 10:12:30AM +1000, Allan McRae via arch-dev-public wrote:

...
Is there any particular objection to requiring packagers upload both architectures?

I'm personally not really motivated doing the required builds. We have an underdeveloped infrastructure which hasn't changed since we abandoned i686 5 years ago.

I'd personally like to see more work on our build tooling before we commit to new architectures.

FYI, it is a single extra command. Either of these will work...

offload-build --arch x86_64_v3 extra-x86_64_v3-build

Nothing else changes for the packager.

Allan

I'll spend twice as long waiting for a package to build which increases the time spent packaging. Which again requires me to spend more time watching stuff fly by. This also assumes people are capable of using the buildserver which is not always the case either. This wasn't great with i686, and I'm not sure why we'd find this acceptable today? -- Morten Linderud PGP: 9C02FF419FECBE16

Allan McRae

1:45 a.m.

On 29/1/22 11:28, Morten Linderud via arch-dev-public wrote:

...

On Sat, Jan 29, 2022 at 11:22:32AM +1000, Allan McRae via arch-dev-public wrote:

...
On 29/1/22 11:13, Morten Linderud via arch-dev-public wrote:

...
On Sat, Jan 29, 2022 at 10:12:30AM +1000, Allan McRae via arch-dev-public wrote:

...
Is there any particular objection to requiring packagers upload both architectures?

I'm personally not really motivated doing the required builds. We have an underdeveloped infrastructure which hasn't changed since we abandoned i686 5 years ago.

I'd personally like to see more work on our build tooling before we commit to new architectures.

FYI, it is a single extra command. Either of these will work...

offload-build --arch x86_64_v3 extra-x86_64_v3-build

Nothing else changes for the packager.

Allan

I'll spend twice as long waiting for a package to build which increases the time spent packaging. Which again requires me to spend more time watching stuff fly by.

I have concerns if that is how you spend your time... :)

...

This also assumes people are capable of using the buildserver which is not always the case either.

This wasn't great with i686, and I'm not sure why we'd find this acceptable today?

What was not great with i686? We managed two architectures for many, many years. The reason for removing i686 was to do with outdated technology, not to do with build times or infrastructure. Do you have objections beyond not wanting to package for both repos? i.e. do you object to option C in my original email, where we have a team to keep the repos in sync when package maintainers do not build both? Allan

Filipe Laíns

3:39 a.m.

On Sat, 2022-01-29 at 11:45 +1000, Allan McRae via arch-dev-public wrote:

...

On 29/1/22 11:28, Morten Linderud via arch-dev-public wrote:

...
On Sat, Jan 29, 2022 at 11:22:32AM +1000, Allan McRae via arch-dev-public wrote:

...
On 29/1/22 11:13, Morten Linderud via arch-dev-public wrote:

...
On Sat, Jan 29, 2022 at 10:12:30AM +1000, Allan McRae via arch-dev- public wrote:

...
Is there any particular objection to requiring packagers upload both architectures?

I'm personally not really motivated doing the required builds. We have an underdeveloped infrastructure which hasn't changed since we abandoned i686 5 years ago.

I'd personally like to see more work on our build tooling before we commit to new architectures.

FYI, it is a single extra command. Either of these will work...

offload-build --arch x86_64_v3 extra-x86_64_v3-build

Nothing else changes for the packager.

Allan

I'll spend twice as long waiting for a package to build which increases the time spent packaging. Which again requires me to spend more time watching stuff fly by.

I have concerns if that is how you spend your time... :)

...
This also assumes people are capable of using the buildserver which is not always the case either.

This wasn't great with i686, and I'm not sure why we'd find this acceptable today?

What was not great with i686? We managed two architectures for many, many years. The reason for removing i686 was to do with outdated technology, not to do with build times or infrastructure.

Do you have objections beyond not wanting to package for both repos? i.e. do you object to option C in my original email, where we have a team to keep the repos in sync when package maintainers do not build both?

Allan

I think the issue is not that we don't want to package for both repos, but the increased workload that places on us. For example, the most painful packages to update for me personally are the cross compilers (and other elements of the cross toolchains, but mainly the compilers). This is something I am already struggling with, and adding a new architecture will make it twice as painful. https://archlinux.org/packages/?sort=&q=gcc&maintainer=FFY00&flagged=Flagged This could have been implemented by making x86_64_v3 kind of an overlay over the normal x86_64, but instead it is its hole new architecture requiring duplicated builds for all packages, and not just the ones where it actually makes a difference and/or the maintainer is willing to do it. I don't think this proposal considers the impact it will have on our already struggling staff properly, but alas, there's nothing I can do as the RFC was accepted. It will certainly deter me from adding a couple new packages to the repos, and perhaps drop some, FWIW. The best option seems B, or C, but I don't really understand how B is supposed to work in practice. Isn't the new architecture is supposed to be in sync with the current one? Because not doing so seems a little bit problematic, and I don't see a way for this to work otherwise. Anyway, I thank you for your time working on this, it is certainly valuable, I am just unsure, and, admittedly, perhaps a little bit pessimistic, about how this implementation impacts my commitment as an Arch packager. My two cents are that adding a x86_64_v3 would be amazing if had already reached the, still far, goal of having automated package builders. But we are not there yet, so this introduces a lot of manual labor :/ Cheers, Filipe Laíns

Morten Linderud

12:11 p.m.

On Sat, Jan 29, 2022 at 11:45:57AM +1000, Allan McRae via arch-dev-public wrote:

...

On 29/1/22 11:28, Morten Linderud via arch-dev-public wrote:

...
On Sat, Jan 29, 2022 at 11:22:32AM +1000, Allan McRae via arch-dev-public wrote:

...
On 29/1/22 11:13, Morten Linderud via arch-dev-public wrote:

...
On Sat, Jan 29, 2022 at 10:12:30AM +1000, Allan McRae via arch-dev-public wrote:

...
Is there any particular objection to requiring packagers upload both architectures?

I'm personally not really motivated doing the required builds. We have an underdeveloped infrastructure which hasn't changed since we abandoned i686 5 years ago.

I'd personally like to see more work on our build tooling before we commit to new architectures.

FYI, it is a single extra command. Either of these will work...

offload-build --arch x86_64_v3 extra-x86_64_v3-build

Nothing else changes for the packager.

Allan

I'll spend twice as long waiting for a package to build which increases the time spent packaging. Which again requires me to spend more time watching stuff fly by.

I have concerns if that is how you spend your time... :)

I have plenty time left over to ask for pacman backports :) DO NOT WORRY!

...

...
This also assumes people are capable of using the buildserver which is not always the case either.

This wasn't great with i686, and I'm not sure why we'd find this acceptable today?

What was not great with i686? We managed two architectures for many, many years. The reason for removing i686 was to do with outdated technology, not to do with build times or infrastructure.

Do you have objections beyond not wanting to package for both repos? i.e. do you object to option C in my original email, where we have a team to keep the repos in sync when package maintainers do not build both?

I'm simply not sure where you are going to get the people for that and how you want to deal with it? A *lot* has changed since the x86_64 port. Bringing on people to do these rebuild implies they need access to infrastructure, keyring and so on. And we already have a staff shortage. I'd like to see some details on how you envision C should be working first. Generally my thoughts is that we shouldn't *need* to have a more manhours to deal with a x86_64_v3. We should instead strenghten our staff and work on the following: * Signing enclave * Better rebuilding tools * Build automation * Git migration It would make discussions like these completely obsolete. Do we want v2, v3, v4, v5, v90001? Enable it in a setting and we'd have the repos. It is a lot of work but it would modernize and make things a lot simpler for us. -- Morten Linderud PGP: 9C02FF419FECBE16

Allan McRae

12:53 p.m.

On 29/1/22 22:11, Morten Linderud via arch-dev-public wrote:

...

On Sat, Jan 29, 2022 at 11:45:57AM +1000, Allan McRae via arch-dev-public wrote:

...
On 29/1/22 11:28, Morten Linderud via arch-dev-public wrote:

...
On Sat, Jan 29, 2022 at 11:22:32AM +1000, Allan McRae via arch-dev-public wrote:

...
On 29/1/22 11:13, Morten Linderud via arch-dev-public wrote:

...
On Sat, Jan 29, 2022 at 10:12:30AM +1000, Allan McRae via arch-dev-public wrote:

...
Is there any particular objection to requiring packagers upload both architectures?

I'm personally not really motivated doing the required builds. We have an underdeveloped infrastructure which hasn't changed since we abandoned i686 5 years ago.

I'd personally like to see more work on our build tooling before we commit to new architectures.

FYI, it is a single extra command. Either of these will work...

offload-build --arch x86_64_v3 extra-x86_64_v3-build

Nothing else changes for the packager.

Allan

I'll spend twice as long waiting for a package to build which increases the time spent packaging. Which again requires me to spend more time watching stuff fly by.

I have concerns if that is how you spend your time... :)

I have plenty time left over to ask for pacman backports :) DO NOT WORRY!

...
...
This also assumes people are capable of using the buildserver which is not always the case either.

This wasn't great with i686, and I'm not sure why we'd find this acceptable today?

What was not great with i686? We managed two architectures for many, many years. The reason for removing i686 was to do with outdated technology, not to do with build times or infrastructure.

Do you have objections beyond not wanting to package for both repos? i.e. do you object to option C in my original email, where we have a team to keep the repos in sync when package maintainers do not build both?

I'm simply not sure where you are going to get the people for that and how you want to deal with it?

There is community demand for this port - it will provide benefit for the majority of our users. I'm sure I can find interested parties to join.

...

A *lot* has changed since the x86_64 port. Bringing on people to do these rebuild implies they need access to infrastructure, keyring and so on. And we already have a staff shortage.

I'd like to see some details on how you envision C should be working first.

Exactly the same as it did in the i686/x86_64 days. Some packagers will upload both variants, some will not. There was a webpage that showed the package differences between the architectures and a group of people built and uploaded packages to keep x86_64 in sync. This was particularly important when many devs did not have x86_64 hardware yet. We may/will need to recruit some people to do these rebuilds. The number of people needed depends on how many packagers would package for both architectures.

...

Generally my thoughts is that we shouldn't *need* to have a more manhours to deal with a x86_64_v3. We should instead strenghten our staff and work on the following:

* Signing enclave * Better rebuilding tools * Build automation * Git migration

It would make discussions like these completely obsolete. Do we want v2, v3, v4, v5, v90001? Enable it in a setting and we'd have the repos. It is a lot of work but it would modernize and make things a lot simpler for us.

It has been over a decade since pacman allowed signing of repo databases, and we still have not managed to get those signed. And a git migration has been discussed almost as long. I don't think we should wait for this. Though, given there is several years between hardware releases for each x86_64 version so far, we may be in that situation for automated builds by v90001! But if developing this infrastructure does occur, everyone recruited for packaging x86_64_v3 will be able to help keep packages updated! Win-win. Allan

Morten Linderud

1:02 p.m.

On Sat, Jan 29, 2022 at 10:53:41PM +1000, Allan McRae via arch-dev-public wrote:

...

On 29/1/22 22:11, Morten Linderud via arch-dev-public wrote:

...
On Sat, Jan 29, 2022 at 11:45:57AM +1000, Allan McRae via arch-dev-public wrote:

...
On 29/1/22 11:28, Morten Linderud via arch-dev-public wrote:

...
On Sat, Jan 29, 2022 at 11:22:32AM +1000, Allan McRae via arch-dev-public wrote:

...
On 29/1/22 11:13, Morten Linderud via arch-dev-public wrote:

...
On Sat, Jan 29, 2022 at 10:12:30AM +1000, Allan McRae via arch-dev-public wrote: > Is there any particular objection to requiring packagers upload both > architectures?

I'm personally not really motivated doing the required builds. We have an underdeveloped infrastructure which hasn't changed since we abandoned i686 5 years ago.

I'd personally like to see more work on our build tooling before we commit to new architectures.

FYI, it is a single extra command. Either of these will work...

offload-build --arch x86_64_v3 extra-x86_64_v3-build

Nothing else changes for the packager.

Allan

I'll spend twice as long waiting for a package to build which increases the time spent packaging. Which again requires me to spend more time watching stuff fly by.

I have concerns if that is how you spend your time... :)

I have plenty time left over to ask for pacman backports :) DO NOT WORRY!

...
...
This also assumes people are capable of using the buildserver which is not always the case either.

This wasn't great with i686, and I'm not sure why we'd find this acceptable today?

What was not great with i686? We managed two architectures for many, many years. The reason for removing i686 was to do with outdated technology, not to do with build times or infrastructure.

Do you have objections beyond not wanting to package for both repos? i.e. do you object to option C in my original email, where we have a team to keep the repos in sync when package maintainers do not build both?

I'm simply not sure where you are going to get the people for that and how you want to deal with it?

There is community demand for this port - it will provide benefit for the majority of our users. I'm sure I can find interested parties to join.

...
A *lot* has changed since the x86_64 port. Bringing on people to do these rebuild implies they need access to infrastructure, keyring and so on. And we already have a staff shortage.

I'd like to see some details on how you envision C should be working first.

Exactly the same as it did in the i686/x86_64 days. Some packagers will upload both variants, some will not. There was a webpage that showed the package differences between the architectures and a group of people built and uploaded packages to keep x86_64 in sync. This was particularly important when many devs did not have x86_64 hardware yet.

We may/will need to recruit some people to do these rebuilds. The number of people needed depends on how many packagers would package for both architectures.

This doesn't explain what I wanted to know though. Are you expecting we find 5-10-20 people and then onboard them as developers or TUs? Are you envioning a seperate signing keyring, or are you planning on adding them to the archlinux-keyring? How are we dealing with access? Do they get full access to our package repos as devs, or are you planning on a seperate role which has access to the required repositories? This is relevant because of how dbscripts is deployed on gemini. All of this requires a fair bit of planning and thought before it's a feasable option to mention. -- Morten Linderud PGP: 9C02FF419FECBE16

Allan McRae

1:49 p.m.

On 29/1/22 23:02, Morten Linderud wrote:

...

On Sat, Jan 29, 2022 at 10:53:41PM +1000, Allan McRae via arch-dev-public wrote:

...
Exactly the same as it did in the i686/x86_64 days. Some packagers will upload both variants, some will not. There was a webpage that showed the package differences between the architectures and a group of people built and uploaded packages to keep x86_64 in sync. This was particularly important when many devs did not have x86_64 hardware yet.

We may/will need to recruit some people to do these rebuilds. The number of people needed depends on how many packagers would package for both architectures.

This doesn't explain what I wanted to know though.

Are you expecting we find 5-10-20 people and then onboard them as developers or TUs?

Are you envioning a seperate signing keyring, or are you planning on adding them to the archlinux-keyring?

How are we dealing with access? Do they get full access to our package repos as devs, or are you planning on a seperate role which has access to the required repositories? This is relevant because of how dbscripts is deployed on gemini.

All of this requires a fair bit of planning and thought before it's a feasable option to mention.

Assuming we need people to help the x86_64_v3 port, I would post a news item and have people apply. We have advertised developer positions in the past and received dozens of applications, and readily filled the available positions with quality applicants. They would be brought on as Package Maintainers (once approved on the staff list) with access to [extra] and [community], and have packaging privileges including being added to the keyring. While advertising for x86_64_v3 specific packagers, we should make a list of other packaging areas needing help and recruit for those too. Allan

Pierre Schmitz

2:16 p.m.

On Sat, Jan 29, 2022 at 2:49 PM Allan McRae via arch-dev-public wrote:

...

Assuming we need people to help the x86_64_v3 port, I would post a news item and have people apply. We have advertised developer positions in the past and received dozens of applications, and readily filled the available positions with quality applicants. They would be brought on as Package Maintainers (once approved on the staff list) with access to [extra] and [community], and have packaging privileges including being added to the keyring.

While advertising for x86_64_v3 specific packagers, we should make a list of other packaging areas needing help and recruit for those too.

While I would have preferred to gradually have raised the CPU requirements of our main repo (e.g. v2 right now and v3 in a few years), maintaining two x86_64 variants for a transition period might work. Nonetheless we should in work on how to get Arch back to be bleeding edge regardless of this. One aspect might be to reduce the overall amount of packages and get rid of unmaintained software (either by us or upstream). As the vast majority of hardware is v3 already we should consider x86_64 (https://pierre-schmitz.com

Brett Cornwall

5:35 p.m.

On 2022-01-29 15:16, Pierre Schmitz via arch-dev-public wrote:

...

On Sat, Jan 29, 2022 at 2:49 PM Allan McRae via arch-dev-public wrote:

...
Assuming we need people to help the x86_64_v3 port, I would post a news item and have people apply. We have advertised developer positions in the past and received dozens of applications, and readily filled the available positions with quality applicants. They would be brought on as Package Maintainers (once approved on the staff list) with access to [extra] and [community], and have packaging privileges including being added to the keyring.

While advertising for x86_64_v3 specific packagers, we should make a list of other packaging areas needing help and recruit for those too.

While I would have preferred to gradually have raised the CPU requirements of our main repo (e.g. v2 right now and v3 in a few years), maintaining two x86_64 variants for a transition period might work. Nonetheless we should in work on how to get Arch back to be bleeding edge regardless of this. One aspect might be to reduce the overall amount of packages and get rid of unmaintained software (either by us or upstream).

As the vast majority of hardware is v3 already we should consider x86_64 (

Indeed, I would rather that we just move everything to v3 instead of supporting different versions. This seems the simplest way forward.

Morten Linderud

5:40 p.m.

On Sat, Jan 29, 2022 at 09:35:07AM -0800, Brett Cornwall via arch-dev-public wrote:

...

On 2022-01-29 15:16, Pierre Schmitz via arch-dev-public wrote:

...
On Sat, Jan 29, 2022 at 2:49 PM Allan McRae via arch-dev-public wrote:

...
Assuming we need people to help the x86_64_v3 port, I would post a news item and have people apply. We have advertised developer positions in the past and received dozens of applications, and readily filled the available positions with quality applicants. They would be brought on as Package Maintainers (once approved on the staff list) with access to [extra] and [community], and have packaging privileges including being added to the keyring.

While advertising for x86_64_v3 specific packagers, we should make a list of other packaging areas needing help and recruit for those too.

While I would have preferred to gradually have raised the CPU requirements of our main repo (e.g. v2 right now and v3 in a few years), maintaining two x86_64 variants for a transition period might work. Nonetheless we should in work on how to get Arch back to be bleeding edge regardless of this. One aspect might be to reduce the overall amount of packages and get rid of unmaintained software (either by us or upstream).

As the vast majority of hardware is v3 already we should consider x86_64 (
Indeed, I would rather that we just move everything to v3 instead of supporting different versions.

This seems the simplest way forward.

This was already discussed in our RFC and rejected. I'd prefer if we stick to what was agreed on there. Multiple points was raised which is worth reading up on. https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/2 -- Morten Linderud PGP: 9C02FF419FECBE16

Sven-Hendrik Haase

7:17 p.m.

On Sat, 29 Jan 2022 at 15:17, Pierre Schmitz via arch-dev-public < arch-dev-public@lists.archlinux.org> wrote:

...

On Sat, Jan 29, 2022 at 2:49 PM Allan McRae via arch-dev-public wrote:

...
Assuming we need people to help the x86_64_v3 port, I would post a news item and have people apply. We have advertised developer positions in the past and received dozens of applications, and readily filled the available positions with quality applicants. They would be brought on as Package Maintainers (once approved on the staff list) with access to [extra] and [community], and have packaging privileges including being added to the keyring.

While advertising for x86_64_v3 specific packagers, we should make a list of other packaging areas needing help and recruit for those too.

While I would have preferred to gradually have raised the CPU requirements of our main repo (e.g. v2 right now and v3 in a few years), maintaining two x86_64 variants for a transition period might work. Nonetheless we should in work on how to get Arch back to be bleeding edge regardless of this. One aspect might be to reduce the overall amount of packages and get rid of unmaintained software (either by us or upstream).

As the vast majority of hardware is v3 already we should consider x86_64 (
Greetings,

Pierre

-- Pierre Schmitz, https://pierre-schmitz.com

This wouldn't really be too much of an issue if we had proper automation. With automation, this exact problem solves itself to a degree. Surely there will still be specific breakages now and then but the bulk of the burden will go away. We'd even be able to support other targets with ease. However, I realize this will require a lot of upfront infra work before we're there and I'm not sure we should block this proposal on that work. If we don't eventually get good automation (and packages in git), this kinda problem will keep reoccurring. Sadly I don't really have time to work on this right now though I'd love to. Sven

Maxime Gauduin

30 Jan 30 Jan

1 p.m.

On Sat, 2022-01-29 at 20:17 +0100, Sven-Hendrik Haase via arch-dev- public wrote:

...

On Sat, 29 Jan 2022 at 15:17, Pierre Schmitz via arch-dev-public < arch-dev-public@lists.archlinux.org> wrote:

...
On Sat, Jan 29, 2022 at 2:49 PM Allan McRae via arch-dev-public wrote:

...
Assuming we need people to help the x86_64_v3 port, I would post a news item and have people apply. We have advertised developer positions in the past and received dozens of applications, and readily filled the available positions with quality applicants. They would be brought on as Package Maintainers (once approved on the staff list) with access to [extra] and [community], and have packaging privileges including being added to the keyring.

While advertising for x86_64_v3 specific packagers, we should make a list of other packaging areas needing help and recruit for those too.

While I would have preferred to gradually have raised the CPU requirements of our main repo (e.g. v2 right now and v3 in a few years), maintaining two x86_64 variants for a transition period might work. Nonetheless we should in work on how to get Arch back to be bleeding edge regardless of this. One aspect might be to reduce the overall amount of packages and get rid of unmaintained software (either by us or upstream).

As the vast majority of hardware is v3 already we should consider x86_64 (
Greetings,

Pierre

-- Pierre Schmitz, https://pierre-schmitz.com

This wouldn't really be too much of an issue if we had proper automation. With automation, this exact problem solves itself to a degree. Surely there will still be specific breakages now and then but the bulk of the burden will go away. We'd even be able to support other targets with ease.

However, I realize this will require a lot of upfront infra work before we're there and I'm not sure we should block this proposal on that work.

If we don't eventually get good automation (and packages in git), this kinda problem will keep reoccurring. Sadly I don't really have time to work on this right now though I'd love to.

Sven

We already have 2 working automated build tools, that I know of, Evangelos' and mine [0]. I'm sure we can figure something out fairly quickly, unless we'd rather go with some Gitlab CI now that we have one. It would probably make more sense to go that route, but I've already fitted several Gitlab instances with a Buildbot CI, I find it more flexible and it also works wonders. [0] https://github.com/alucryd/archbuild Cheers, -- Maxime

Morten Linderud

3:12 p.m.

On Sun, Jan 30, 2022 at 02:00:51PM +0100, Maxime Gauduin via arch-dev-public wrote:

...

On Sat, 2022-01-29 at 20:17 +0100, Sven-Hendrik Haase via arch-dev-

...
This wouldn't really be too much of an issue if we had proper automation. With automation, this exact problem solves itself to a degree. Surely there will still be specific breakages now and then but the bulk of the burden will go away. We'd even be able to support other targets with ease.

However, I realize this will require a lot of upfront infra work before we're there and I'm not sure we should block this proposal on that work.

If we don't eventually get good automation (and packages in git), this kinda problem will keep reoccurring. Sadly I don't really have time to work on this right now though I'd love to.

Sven

We already have 2 working automated build tools, that I know of, Evangelos' and mine [0]. I'm sure we can figure something out fairly quickly, unless we'd rather go with some Gitlab CI now that we have one. It would probably make more sense to go that route, but I've already fitted several Gitlab instances with a Buildbot CI, I find it more flexible and it also works wonders.

I think having buildbot would be more flexible for such an infrastructure then piggy back on the pipeline system for gitlab. It sometimes feels too restrictive being tied to git repositories and buildbot is a *lot* more flexible when it comes to it. It might also be worth a consideration that the build infrastructure would be completely seperate from our git forge. This would make it easier if we want to move VCS/forge in the future. -- Morten Linderud PGP: 9C02FF419FECBE16

Thore Bödecker

29 Jan 29 Jan

2:26 p.m.

Shouldn't we rather recruit people to help with build automation, git migration and the other topics that keep biting us whenever we try adopting new stuff in a timely manner and/or with less manual stuff to do? I fully agree with Morten on this one and we should definitely get our infra and build automation sorted out first, before thinking about trying to "compete" with new features that other distros might offer. Admittedly I've been intrigued by the x86_64_v3/x86_64_v4 targets for a while but no matter how I look at it, having better build automation and tooling around it is an absolute blocker for this. Don't get me wrong, I'm totally in favour of this proposal in the hopefully not too distant future, just not right now until the aforementioned blockers have been resolved. On 29.01.22 23:49, Allan McRae via arch-dev-public wrote:

...

On 29/1/22 23:02, Morten Linderud wrote:

...
On Sat, Jan 29, 2022 at 10:53:41PM +1000, Allan McRae via arch-dev-public wrote:

...
Exactly the same as it did in the i686/x86_64 days. Some packagers will upload both variants, some will not. There was a webpage that showed the package differences between the architectures and a group of people built and uploaded packages to keep x86_64 in sync. This was particularly important when many devs did not have x86_64 hardware yet.

We may/will need to recruit some people to do these rebuilds. The number of people needed depends on how many packagers would package for both architectures.

This doesn't explain what I wanted to know though.

Are you expecting we find 5-10-20 people and then onboard them as developers or TUs?

Are you envioning a seperate signing keyring, or are you planning on adding them to the archlinux-keyring?

How are we dealing with access? Do they get full access to our package repos as devs, or are you planning on a seperate role which has access to the required repositories? This is relevant because of how dbscripts is deployed on gemini.

All of this requires a fair bit of planning and thought before it's a feasable option to mention.

Assuming we need people to help the x86_64_v3 port, I would post a news item and have people apply. We have advertised developer positions in the past and received dozens of applications, and readily filled the available positions with quality applicants. They would be brought on as Package Maintainers (once approved on the staff list) with access to [extra] and [community], and have packaging privileges including being added to the keyring.

While advertising for x86_64_v3 specific packagers, we should make a list of other packaging areas needing help and recruit for those too.

Allan

Grüße, Thore -- Thore Bödecker GPG ID: 0xD622431AF8DB80F3 GPG FP: 0F96 559D 3556 24FC 2226 A864 D622 431A F8DB 80F3

David Runge

6:19 p.m.

On 2022-01-29 13:11:05 (+0100), Morten Linderud via arch-dev-public wrote:

...

* Signing enclave * Better rebuilding tools * Build automation * Git migration

It would make discussions like these completely obsolete. Do we want v2, v3, v4, v5, v90001? Enable it in a setting and we'd have the repos. It is a lot of work but it would modernize and make things a lot simpler for us.

FWIW: I fully agree with the above. As tempting as it may seem to "just do it": The time spent on the tooling that would allow us to handle this amount of packages in an automatable fashion lacks greatly. I do not see that "we have had feature X for Y years" can serve as reason for introducing a lot more work or for adding outsiders to the packaging context. The tooling - instead of improving - will further stagnate and be cemented this way. This is something we have to consider and work on. I am afraid that otherwise we are maneuvering ourselves into an unmanagable corner with this. To be more specific: I believe our current best way forward is to continuously propagate our pain point projects that are our main blockers to achieve adding more architectures easily (and in a manageable fashion, without burning out packagers). We need more contributors to improve and focus on these projects and this can and should be done by advertising them more specifically and more regularly. I am not against introducing one or many new architectures, but I am certain that it will take a negative toll on our current group of packagers (even with additional packagers) due to the current set of tooling at hand (as it diverts the attention from improving the tooling). Best, David -- https://sleepmap.de

Andreas Radke

30 Jan 30 Jan

11:02 a.m.

1) -1 for bringing new packagers onboard just due to the lack of automatic rebuilds. 2) For me 2 of my 3 systems are only x86-64-v2 so I won't be able to test anything seriously. So I'm out of the game if v3 is the new default. Both systems are fast enough for many more years. So there's no plan to replace them. 3) Anything that just needs simple rebuilds should be done in 2022 without any manpower. What's the plan with the new v3 port if I build only v1 packages? Will v3 stay at the old version or will the new v1 package also be pushed to v3 just lacking the better optimization until some new v3 rebuild brings it back? This way v3 would go forth and back all the time and should be avoided. But falling behind v1 for some time is also not desired. 4) Is anybody actually running ALHP or working on it? https://alhp.harting.dev/packages.html [5) Why don't we care about ARM and OpenRISC at all? Aren't we riding pretty dead horses here just for very minor improvements?] -Andy

Morten Linderud

11:13 a.m.

On Sun, Jan 30, 2022 at 12:02:46PM +0100, Andreas Radke via arch-dev-public wrote:

...

[5) Why don't we care about ARM and OpenRISC at all? Aren't we riding pretty dead horses here just for very minor improvements?]

ALARM has refused to join the project because of our ancient tooling. RISC is being worked on by Felix. https://github.com/felixonmars/archriscv-packages -- Morten Linderud PGP: 9C02FF419FECBE16

David Runge

11:18 a.m.

On 2022-01-30 12:13:45 (+0100), Morten Linderud via arch-dev-public wrote:

...

ALARM has refused to join the project because of our ancient tooling.

Case in point. :) -- https://sleepmap.de

Allan McRae

12:04 p.m.

On 30/1/22 21:13, Morten Linderud via arch-dev-public wrote:

...

On Sun, Jan 30, 2022 at 12:02:46PM +0100, Andreas Radke via arch-dev-public wrote:

...
[5) Why don't we care about ARM and OpenRISC at all? Aren't we riding pretty dead horses here just for very minor improvements?]

ALARM has refused to join the project because of our ancient tooling. RISC is being worked on by Felix.

Not 100% accurate. When I asked there was significant issues around sponsorship and ownership of the arm machines used as a buildfarm.

Giancarlo Razzolini

8 Feb 8 Feb

2:17 p.m.

Em janeiro 29, 2022 9:11 Morten Linderud via arch-dev-public escreveu:

...

Generally my thoughts is that we shouldn't *need* to have a more manhours to deal with a x86_64_v3. We should instead strenghten our staff and work on the following:

* Signing enclave * Better rebuilding tools * Build automation * Git migration

It would make discussions like these completely obsolete. Do we want v2, v3, v4, v5, v90001? Enable it in a setting and we'd have the repos. It is a lot of work but it would modernize and make things a lot simpler for us.

I wholeheartedly agree that we should improve our automation and our tooling. However, I don't think it should be a blocker for -v3. When we did i686, I simply built the package and I was not able to test it. With -v3, the situation is different, most here can test both packages. The distro as a whole will benefit from this new architecture. We *do* have issues when it comes to the tooling, that's a given. I don't know what else we can do, other than recruiting people interested in helping with it. Regards, Giancarlo Razzolini

Johannes Löthberg

29 Jan 29 Jan

3:46 a.m.

Excerpts from Morten Linderud via arch-dev-public's message of January 29, 2022 2:28:

...

On Sat, Jan 29, 2022 at 11:22:32AM +1000, Allan McRae via arch-dev-public wrote:

...
On 29/1/22 11:13, Morten Linderud via arch-dev-public wrote:

...
On Sat, Jan 29, 2022 at 10:12:30AM +1000, Allan McRae via arch-dev-public wrote:

...
Is there any particular objection to requiring packagers upload both architectures?

I'm personally not really motivated doing the required builds. We have an underdeveloped infrastructure which hasn't changed since we abandoned i686 5 years ago.

I'd personally like to see more work on our build tooling before we commit to new architectures.

FYI, it is a single extra command. Either of these will work...

offload-build --arch x86_64_v3 extra-x86_64_v3-build

Nothing else changes for the packager.

Allan

I'll spend twice as long waiting for a package to build which increases the time spent packaging. Which again requires me to spend more time watching stuff fly by.

You *can* run the builds concurrently though since they're different architectures. So at least if you're using the build server it shouldn't be significantly slower unless you're building something that is heavy enough to build where you use ~all of the resources it has. -- Sincerely, Johannes Löthberg :: SA0DEM

Allan McRae

1:29 a.m.

On 29/1/22 10:12, Allan McRae via arch-dev-public wrote:

...

Apart from tooling (devtools/dbscripts), we need to make some decisions.

FYI - I just submitted merge requests for both devtools and dbscripts. These should be committed in tandem with work at devops end preparing the initial state of the x86_64_v3 repos. There may be more work needed on dbscripts to cleanup x86_64 packages from the x86_64_v3 repos, assuming we seed the repos that way. Allan

Felix Yan

3:12 a.m.

On 1/29/22 02:12, Allan McRae via arch-dev-public wrote:

...

The decision to be made is who will package for this repo? I think these are the options:

A) packagers upload both x86_64 and x86_64_v3 to the repos. Our build server will help those without x86_64_v3 machines.

B) we recruit some packagers to build the x86_64_v3 packages.

C) Some combination of A+B.

My understanding is our x86_64 port started with B, then C, then A.

I am fine with either and could happily help with B in long term. For me the issue with either B or C is that our packages are often FTBFS and we are slow to fix them, generally. To make the port really usable for a B/C workflow, we need a way to fill in the time gap (because the old package could be unusable for the time, like missing a so-name bump etc). Do you find it acceptable if the x86_64_v3 rebuilders put back in the new x86_64 package until the build was fixed and probably a point pkgrel was added for the real x86_64_v3 rebuild, as long as we use B/C to build for x86_64_v3, in the long term? -- Regards, Felix Yan

Allan McRae

3:31 a.m.

On 29/1/22 13:12, Felix Yan via arch-dev-public wrote:

...

On 1/29/22 02:12, Allan McRae via arch-dev-public wrote:

...
The decision to be made is who will package for this repo? I think these are the options:

A) packagers upload both x86_64 and x86_64_v3 to the repos. Our build server will help those without x86_64_v3 machines.

B) we recruit some packagers to build the x86_64_v3 packages.

C) Some combination of A+B.

My understanding is our x86_64 port started with B, then C, then A.

I am fine with either and could happily help with B in long term.

For me the issue with either B or C is that our packages are often FTBFS and we are slow to fix them, generally. To make the port really usable for a B/C workflow, we need a way to fill in the time gap (because the old package could be unusable for the time, like missing a so-name bump etc).

Do you find it acceptable if the x86_64_v3 rebuilders put back in the new x86_64 package until the build was fixed and probably a point pkgrel was added for the real x86_64_v3 rebuild, as long as we use B/C to build for x86_64_v3, in the long term?

I envisioned the x86_64_v3 repos were initially seeded with x86_64 packages. Then x86_64_v3 packages gradually replaced them as updates/rebuilds happen in x86_64. FTBFS for new packages should be less of an issue as long as the two repos do not get out of sync for long - I was considering the i686/x86_64 days where the sync with i686 was within days if not hours. But yes, a workaround could be adding x86_64 packages into x86_64_v3 if really needed. I'm not sure the workflow for doing that in terms of devtools/dbscripts. Allan

Tobias Powalowski

31 Jan 31 Jan

8:01 a.m.

Hi, A) I can do both with my server, but I don't package a lot. B) I agree with Andy here it should be automated. When the automatic setup is realized, we should definitly consider to support more architectures, like aarch64 for the new showing up arm servers and Apple Macs. greetings tpowa Am Sa., 29. Jan. 2022 um 01:12 Uhr schrieb Allan McRae via arch-dev-public < arch-dev-public@lists.archlinux.org>:

...

Hi all,

You may remember long in the past we discussed adding an x86_64_v3 port. From memory, pkgstats shows this will benefit ~2/3 (or 3/4?) of our users! So lets get this underway!

Apart from tooling (devtools/dbscripts), we need to make some decisions.

My plan is to seed the x86_64_v3 repos with x86_64 packages and then they get replaced as updates/rebuilds happen (potentially supplemented with a rebuild party). Pacman 6.0 can handle this. This is not a perfectly clean new port, but is substantially less of a burden than releasing a pure x86_64_v3 port. Our .BUILDINFO files do record the package architecture, so this mixture should not affect (e.g.) our reproducible builds etc.

The decision to be made is who will package for this repo? I think these are the options:

A) packagers upload both x86_64 and x86_64_v3 to the repos. Our build server will help those without x86_64_v3 machines.

B) we recruit some packagers to build the x86_64_v3 packages.

C) Some combination of A+B.

My understanding is our x86_64 port started with B, then C, then A.

I think with our build infrastructure now, we can start with A, but that is more of a burden for packagers. I doubt it will be much of a burden as x86_64_v3 specific build issues are unlikely.

Is there any particular objection to requiring packagers upload both architectures?

Allan

-- Tobias Powalowski Archlinux Developer & Package Maintainer (tpowa) https://www.archlinux.org http://www.archlinux.org tpowa@archlinux.org St. Martin-Apotheke Herzog-Georg-Str. 25 89415 Lauingen https://www.st-martin-apo.de info@st-martin-apo.de

821

Age (days ago)

831

Last active (days ago)

List overview

Download

27 comments

14 participants

participants (14)

Allan McRae
Andreas Radke
Brett Cornwall
David Runge
Felix Yan
Filipe Laíns
Giancarlo Razzolini
Johannes Löthberg
Maxime Gauduin
Morten Linderud
Pierre Schmitz
Sven-Hendrik Haase
Thore Bödecker
Tobias Powalowski

[arch-dev-public] Starting x86_64_v3 port

tags

participants (14)