Hello everybody, for some time I used to push rc release for some packages (namely systemd & util-linux) to core-testing. This broke once, when a package was built against that rc release and thus did depend on new symbols, then finally was moved before the package it was built against. After that I pushed all the rc releases to a public personal repository, anybody interested could test from there. But I think the audience was a lot smaller, and last systemd release revealed some unexpected regressions *after* final release. That situation is not any better, especially when rc releases in Arch were a major factor in finding regressions early. At Arch Summit 2025 we had a chat about that situation and discussed several ideas. In the end we came up with one reasonable solution: We should introduce new repositories [core-unstable] and [extra-unstable] for this kind of testing. People would still have to enable these repositories, but I guess chances are higher than for my personal repository. I guess we have three places that need to be touched: * create repositories on servers (-> DevOps) * dbscripts * devtools Anything more? Feedback is welcome, same for other ideas - this is not set in stone and I am open for anything that works for me/us. Thanks! -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}
On 07/11/2025 14:48, Christian Hesse wrote:
Hello everybody,
for some time I used to push rc release for some packages (namely systemd & util-linux) to core-testing. This broke once, when a package was built against that rc release and thus did depend on new symbols, then finally was moved before the package it was built against.
After that I pushed all the rc releases to a public personal repository, anybody interested could test from there. But I think the audience was a lot smaller, and last systemd release revealed some unexpected regressions *after* final release. That situation is not any better, especially when rc releases in Arch were a major factor in finding regressions early.
At Arch Summit 2025 we had a chat about that situation and discussed several ideas. In the end we came up with one reasonable solution: We should introduce new repositories [core-unstable] and [extra-unstable] for this kind of testing. People would still have to enable these repositories, but I guess chances are higher than for my personal repository.
Short feedback from my side, how does this work in terms of hierarchy? I assume [extra-unstable] would be above [extra-testing] and not [extra-staging]?
Jelle van der Waa <jelle@vdwaa.nl> on Fri, 2025/11/07 14:59:
Short feedback from my side, how does this work in terms of hierarchy? I assume [extra-unstable] would be above [extra-testing]
Yes.
and not [extra-staging]?
Hmm, would these ever be used in one configuration? I think end users (even testers) should not have [extra-staging] in their configuration as it is used for rebuilds and thus breaks by design. On the other hand [extra-unstable] should never be used for building, at least not any packages that are supposed to go to a different repository than [extra-unstable]. So any situation we have to define a correct order? -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}
On November 7, 2025 8:48:07 AM EST, Christian Hesse <list@eworm.de> wrote:
Hello everybody,
for some time I used to push rc release for some packages (namely systemd & util-linux) to core-testing. This broke once, when a package was built against that rc release and thus did depend on new symbols, then finally was moved before the package it was built against.
After that I pushed all the rc releases to a public personal repository, anybody interested could test from there. But I think the audience was a lot smaller, and last systemd release revealed some unexpected regressions *after* final release. That situation is not any better, especially when rc releases in Arch were a major factor in finding regressions early.
At Arch Summit 2025 we had a chat about that situation and discussed several ideas. In the end we came up with one reasonable solution: We should introduce new repositories [core-unstable] and [extra-unstable] for this kind of testing. People would still have to enable these repositories, but I guess chances are higher than for my personal repository.
I guess we have three places that need to be touched:
* create repositories on servers (-> DevOps) * dbscripts * devtools
Anything more?
Would -unstable also go through the sign off process? If so: - arch-signoff - archweb
Feedback is welcome, same for other ideas - this is not set in stone and I am open for anything that works for me/us. Thanks!
-- Best, Daniel <https://danielcapella.com>
"Daniel M. Capella" <polyzen@archlinux.org> on Fri, 2025/11/07 09:20:
I guess we have three places that need to be touched:
* create repositories on servers (-> DevOps) * dbscripts * devtools
Anything more?
Would -unstable also go through the sign off process? If so:
- arch-signoff - archweb
As these packages are not intended to be moved to [core] or [extra]... No. Once the rc phase is done the rc packages would be dropped from [core-unstable] and [extra-unstable], and the final packages would go their way through [core-testing] and [extra-testing] to finally land in [core] and [extra]. -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}
Am Freitag, dem 07.11.2025 um 14:48 +0100 schrieb Christian Hesse:
Hello everybody,
Hey 👋️
At Arch Summit 2025 we had a chat about that situation and discussed several ideas. In the end we came up with one reasonable solution: We should introduce new repositories [core-unstable] and [extra- unstable] for this kind of testing. People would still have to enable these repositories, but I guess chances are higher than for my personal repository.
Is this also supposed to replace {gnome,kde}-unstable? I have a hard time imagine how testing this should be done if multiple packages are in alpha/beta that might even depend on each other in some way. To be clear - Most of the time I consider gnome-unstable completely broken. Just a dumping ground for new pre-releases and for users who deal themself with this mess until it becomes more reliable. 😂️ I'd like to keep it this way. Would -unstable depend on core- and extra-testing or on the "stable" repos?
Fabian Bornschein <fabiscafe@mailbox.org> on Fri, 2025/11/07 16:04:
Is this also supposed to replace {gnome,kde}-unstable? I have a hard time imagine how testing this should be done if multiple packages are in alpha/beta that might even depend on each other in some way.
This is to be discussed, but I think I would tend to keep it separated. Someone who wants to test rc releases of systemd or util-linux is not necessarily in search for trouble with unstable gnome. :-p Though someone could enable [core-unstable], but skip [extra-unstable]...
To be clear - Most of the time I consider gnome-unstable completely broken. Just a dumping ground for new pre-releases and for users who deal themself with this mess until it becomes more reliable. 😂️ I'd like to keep it this way.
I have not tested anything from these repositories... Can't tell.
Would -unstable depend on core- and extra-testing or on the "stable" repos?
At least my packages would be in a state the having stable und unstable, but skipping testing, would be just fine. -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}
Christian Hesse <list@eworm.de> on Fri, 2025/11/07 20:40:
Would -unstable depend on core- and extra-testing or on the "stable" repos?
At least my packages would be in a state the having stable und unstable, but skipping testing, would be just fine.
Looks like this is a bad idea. Let's assume we have something in testing with an soname bump - for example openssl, and systemd is rebuilt as well. With systemd in [core-unstable] we want it to work in any case, so yes - depending on [core-testing] and [extra-testing]. (Though it may work without most of the time.) Thanks to Mike Yuan for giving the hint! -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}
El viernes, 7 de noviembre de 2025 a las 14:48 Christian Hesse escribió:
At Arch Summit 2025 we had a chat about that situation and discussed several ideas. In the end we came up with one reasonable solution: We should introduce new repositories [core-unstable] and [extra-unstable] for this kind of testing. People would still have to enable these repositories, but I guess chances are higher than for my personal repository.
I guess we have three places that need to be touched:
* create repositories on servers (-> DevOps) * dbscripts * devtools
Anything more?
How would that work on the git repo level? Do you update to the unstable version in main? In that case, what happens if the package needs a rebuild in the stable repos?
Antonio Rojas <arojas@archlinux.org> on Fri, 2025/11/07 16:11:
How would that work on the git repo level? Do you update to the unstable version in main? In that case, what happens if the package needs a rebuild in the stable repos?
Ah, really good question... Have not thought about that. How is this done for Gnome and KDE at the moment? Anybody involved there can elaborate? For the packages in my personal repository I use a separate branch and the changes are just dropped once the final packages go to official repositories. For sure we do not want that when packages go to official unstable repositories. -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}
El viernes, 7 de noviembre de 2025 a las 20:44 Christian Hesse escribió:
Antonio Rojas <arojas@archlinux.org> on Fri, 2025/11/07 16:11:
How would that work on the git repo level? Do you update to the unstable version in main? In that case, what happens if the package needs a rebuild in the stable repos?
Ah, really good question... Have not thought about that.
How is this done for Gnome and KDE at the moment? Anybody involved there can elaborate?
We use separate branches. This workflow was broken by the move to git though, and hasn't been fixed yet, so the procedure is quite hackish now as we need to bypass the limitations of the official tools. I'm hoping this proposal will make this finally be sorted out on the devtools side.
Quoting Christian Hesse (2025-11-07 08:48:07)
At Arch Summit 2025 we had a chat about that situation and discussed several ideas. In the end we came up with one reasonable solution: We should introduce new repositories [core-unstable] and [extra-unstable] for this kind of testing. People would still have to enable these repositories, but I guess chances are higher than for my personal repository. [...] Feedback is welcome, same for other ideas - this is not set in stone and I am open for anything that works for me/us. Thanks!
Thanks for the idea! Even though I don't think this isn't the "right" "full" solution to a broader underlying challenge (e.g., it doesn't really address closely related issue of overlapping rebuilds), I love taking a simple and pragmatic approach here. Having read through the thread, I am still slightly unsure I understand the proposed layout. Is it (i) a linear chain [extra] <- [extra-unstable] <- [extra-testing] <- [extra-staging], where the core principle that packages earlier in the chain (i.e., further right in the illustration above) are "newer" packages and correspond to "more recent" Git commits? This is a layout that'd work well with the current Git structure but has other implications, such as unstable releases always "blocking" overlapping rebuilds; i.e., the rebuild adopts the unstable release and can only move once the unstable release is ready to move too. I believe it also wouldn't address the example you brought up in the initial email. Or is a better way to think of it as (ii) a branched repo [extra] <- [extra-testing] <- [extra-staging], ^ `--- [extra-unstable] where the order in the pacman configuration would probably still be the same as above but there's now branched package history/evolution. This second layout would most likely need a related branching concept in Git and brings up some other questions; e.g., * Can I enable -testing and -unstable repos at the same time or are they mutually exclusive? * How should I, as a user, choose between -unstable and -testing repos? * If -testing and -unstable can be enabled at the same time, what happens if there's an -unstable version of the package while the package is part of some ongoing rebuild at the same time? Does the rebuild or the -unstable version "win"? Or do we need to rebuild the -unstable version as well when it moves to -testing (which seems to imply that -testing and -unstable now *must* be enabled together to avoid breakage)? * In either case, what happens when a rebuild enters the stable repos; are we expected to rebuild all potentially affected -unstable packages as well? It'd be good to have provisional answers here and flesh them out in an RFC if we'd like to move forward. Thanks again! Lukas
Hello everybody, sorry for the late reply... I have been quite busy lately, and I guess this will not change until next year. Lukas Fleischer <lfleischer@archlinux.org> on Sat, 2025/11/08 13:10:
Having read through the thread, I am still slightly unsure I understand the proposed layout. Is it (i) a linear chain
[extra] <- [extra-unstable] <- [extra-testing] <- [extra-staging],
where the core principle that packages earlier in the chain (i.e., further right in the illustration above) are "newer" packages and correspond to "more recent" Git commits?
This is a layout that'd work well with the current Git structure but has other implications, such as unstable releases always "blocking" overlapping rebuilds; i.e., the rebuild adopts the unstable release and can only move once the unstable release is ready to move too. I believe it also wouldn't address the example you brought up in the initial email.
No, I think this would not help here.
Or is a better way to think of it as (ii) a branched repo
[extra] <- [extra-testing] <- [extra-staging], ^ `--- [extra-unstable]
where the order in the pacman configuration would probably still be the same as above but there's now branched package history/evolution.
Similar, but I think we need to branch at a different point, between testing and staging: [extra] <- [extra-testing] <- [extra-staging], ^ `--- [extra-unstable] So order in pacman.conf would be: * extra-unstable * extra-testing * extra
This second layout would most likely need a related branching concept in Git and brings up some other questions; e.g.,
Exactly. I think that is our main issue at the moment. This has has to be solved before we can proceed.
* Can I enable -testing and -unstable repos at the same time or are they mutually exclusive?
* How should I, as a user, choose between -unstable and -testing repos?
* If -testing and -unstable can be enabled at the same time, what happens if there's an -unstable version of the package while the package is part of some ongoing rebuild at the same time? Does the rebuild or the -unstable version "win"? Or do we need to rebuild the -unstable version as well when it moves to -testing (which seems to imply that -testing and -unstable now *must* be enabled together to avoid breakage)?
* In either case, what happens when a rebuild enters the stable repos; are we expected to rebuild all potentially affected -unstable packages as well?
It'd be good to have provisional answers here and flesh them out in an RFC if we'd like to move forward.
As the layout should look different I will not answer each point above, but give my answer here: * Users can enable -testing just they do now - no change. * Enabling -unstable requires enable -testing as well. If a package is both repositories the one in -testing is covered by the one in -unstable. * The choice is up to the user: -testing for "fresh" stable releases that are supposed to move to stable, -unstable (and -testing) for possibly unstable pre-releases. * With an soname rebuild in -testing the package would have to be rebuilt for -testing and -unstable. That's why -testing is required: Without it could work, but may fail with breakage in dynamic linking. Did that cover all your questions? -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}
Quoting Christian Hesse (2025-12-05 07:47:23)
Similar, but I think we need to branch at a different point, between testing and staging:
[extra] <- [extra-testing] <- [extra-staging], ^ `--- [extra-unstable]
So order in pacman.conf would be:
* extra-unstable * extra-testing * extra
This second layout would most likely need a related branching concept in Git and brings up some other questions; e.g.,
Exactly. I think that is our main issue at the moment. This has has to be solved before we can proceed. [...] * With an soname rebuild in -testing the package would have to be rebuilt for -testing and -unstable. That's why -testing is required: Without it could work, but may fail with breakage in dynamic linking.
Did that cover all your questions?
Thanks! For the most part, yes, I think so. Two follow-ups: 1. Are we going to have [extra-unstable-staging] for rebuilds, or do we plan to have a different mechanism to handle soname rebuilds across the new repos? 2. I believe we'll also need some additional constraints to make this work well; e.g., I'm assuming we'd want to ensure that -unstable must always have more recent package versions than -testing and -staging. We may also want to build some of those into devtools as checks. Does that make sense? Best, Lukas
Lukas Fleischer <lfleischer@archlinux.org> on Fri, 2025/12/05 18:00:
Quoting Christian Hesse (2025-12-05 07:47:23)
Did that cover all your questions?
Thanks! For the most part, yes, I think so. Two follow-ups:
1. Are we going to have [extra-unstable-staging] for rebuilds, or do we plan to have a different mechanism to handle soname rebuilds across the new repos?
Oh, good question... I have not yet had that case. We would hit it when the pre-release bumps a soname, right? So something like systemd pushing a pre-release where libudev is bumped from libudev.so.1 to libudev.so.2... At least for the packages that I care about here these cases should be pretty rare... :-p But that might be different for others.
2. I believe we'll also need some additional constraints to make this work well; e.g., I'm assuming we'd want to ensure that -unstable must always have more recent package versions than -testing and -staging.
That's the idea, yes.
We may also want to build some of those into devtools as checks. Does that make sense?
Hmm, not sure. Depending on package (and impact of rebuild when required) this may vary from over-complicating to required. After all this has much greater impact than expected when we want to make it complete and correctly. -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}
Quoting Christian Hesse (2025-12-06 16:45:29)
Lukas Fleischer <lfleischer@archlinux.org> on Fri, 2025/12/05 18:00:
Quoting Christian Hesse (2025-12-05 07:47:23)
Did that cover all your questions?
Thanks! For the most part, yes, I think so. Two follow-ups:
1. Are we going to have [extra-unstable-staging] for rebuilds, or do we plan to have a different mechanism to handle soname rebuilds across the new repos?
Oh, good question... I have not yet had that case. We would hit it when the pre-release bumps a soname, right? So something like systemd pushing a pre-release where libudev is bumped from libudev.so.1 to libudev.so.2...
At least for the packages that I care about here these cases should be pretty rare... :-p
Based on your earlier replies, if I understand correctly, it'd also be required whenever there's a soname bump (for any package, not necessarily in -unstable) while any dependent package is in -unstable.
[...]
We may also want to build some of those into devtools as checks. Does that make sense?
Hmm, not sure. Depending on package (and impact of rebuild when required) this may vary from over-complicating to required.
FWIW, I don't think those are extremely rare corner cases once we start using -unstable more. Specifically, I'm thinking of cases where we push a new package to -testing or stable repos and forget to drop the (now older) package from -unstable. That said, I think it's reasonable to monitor and quickly fix those issues when they appear; as long as we have a good understanding of what potential issues may appear and as long as they break -unstable only.
After all this has much greater impact than expected when we want to make it complete and correctly.
El sábado, 6 de diciembre de 2025 a las 22:45 Christian Hesse escribió:
Lukas Fleischer <lfleischer@archlinux.org> on Fri, 2025/12/05 18:00:
Quoting Christian Hesse (2025-12-05 07:47:23)
Did that cover all your questions?
Thanks! For the most part, yes, I think so. Two follow-ups:
1. Are we going to have [extra-unstable-staging] for rebuilds, or do we plan to have a different mechanism to handle soname rebuilds across the new repos?
Oh, good question... I have not yet had that case. We would hit it when the pre-release bumps a soname, right? So something like systemd pushing a pre-release where libudev is bumped from libudev.so.1 to libudev.so.2...
At least for the packages that I care about here these cases should be pretty rare... :-p But that might be different for others.
I don't think this new unstable repo should add more burden for other PMs when they need to do so some rebuild. Whenever there is a soname bump that affects a package in [unstable], the maintainer will be notified (since they are also maintaining the stable package), and it should be their responsibility to coordinate with the rebuild author to make sure the unstable packages are rebuild asap when the rebuild moves to testing. And if that doesn't happen soon enough, [unstable] users should be able to temporarily endure the pain (it's called [unstable] for a reason). That's the way it has worked for decades for the [kde-unstable] repo, which is often affected by this sort of issue.
Antonio Rojas <arojas@archlinux.org> on Sun, 2025/12/07 09:26:
El sábado, 6 de diciembre de 2025 a las 22:45 Christian Hesse escribió:
Lukas Fleischer <lfleischer@archlinux.org> on Fri, 2025/12/05 18:00:
Quoting Christian Hesse (2025-12-05 07:47:23)
Did that cover all your questions?
Thanks! For the most part, yes, I think so. Two follow-ups:
1. Are we going to have [extra-unstable-staging] for rebuilds, or do we plan to have a different mechanism to handle soname rebuilds across the new repos?
Oh, good question... I have not yet had that case. We would hit it when the pre-release bumps a soname, right? So something like systemd pushing a pre-release where libudev is bumped from libudev.so.1 to libudev.so.2...
At least for the packages that I care about here these cases should be pretty rare... :-p But that might be different for others.
I don't think this new unstable repo should add more burden for other PMs when they need to do so some rebuild. Whenever there is a soname bump that affects a package in [unstable], the maintainer will be notified (since they are also maintaining the stable package), and it should be their responsibility to coordinate with the rebuild author to make sure the unstable packages are rebuild asap when the rebuild moves to testing. And if that doesn't happen soon enough, [unstable] users should be able to temporarily endure the pain (it's called [unstable] for a reason).
That's the way it has worked for decades for the [kde-unstable] repo, which is often affected by this sort of issue.
Yes, full ACK. This can be used, but nobody should be forced to. Packages by other packagers involved in rebuild should be handle by those actually using the unstable repositories. -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}
On 11/7/25 2:48 PM, Christian Hesse wrote:
Hello everybody,
for some time I used to push rc release for some packages (namely systemd & util-linux) to core-testing. This broke once, when a package was built against that rc release and thus did depend on new symbols, then finally was moved before the package it was built against.
After that I pushed all the rc releases to a public personal repository, anybody interested could test from there. But I think the audience was a lot smaller, and last systemd release revealed some unexpected regressions *after* final release. That situation is not any better, especially when rc releases in Arch were a major factor in finding regressions early.
At Arch Summit 2025 we had a chat about that situation and discussed several ideas. In the end we came up with one reasonable solution: We should introduce new repositories [core-unstable] and [extra-unstable] for this kind of testing. People would still have to enable these repositories, but I guess chances are higher than for my personal repository.
I guess we have three places that need to be touched:
* create repositories on servers (-> DevOps) * dbscripts * devtools
Anything more?
Feedback is welcome, same for other ideas - this is not set in stone and I am open for anything that works for me/us. Thanks!
Hi, Any estimates on how big (in size on disk) would the new unstable repos be? Regards, Arun
pitastrudl <pitastrudl@archlinux.org> on Sun, 2025/12/07 00:54:
On 11/7/25 2:48 PM, Christian Hesse wrote:
Hello everybody,
for some time I used to push rc release for some packages (namely systemd & util-linux) to core-testing. This broke once, when a package was built against that rc release and thus did depend on new symbols, then finally was moved before the package it was built against.
After that I pushed all the rc releases to a public personal repository, anybody interested could test from there. But I think the audience was a lot smaller, and last systemd release revealed some unexpected regressions *after* final release. That situation is not any better, especially when rc releases in Arch were a major factor in finding regressions early.
At Arch Summit 2025 we had a chat about that situation and discussed several ideas. In the end we came up with one reasonable solution: We should introduce new repositories [core-unstable] and [extra-unstable] for this kind of testing. People would still have to enable these repositories, but I guess chances are higher than for my personal repository.
I guess we have three places that need to be touched:
* create repositories on servers (-> DevOps) * dbscripts * devtools
Anything more?
Feedback is welcome, same for other ideas - this is not set in stone and I am open for anything that works for me/us. Thanks!
Hi,
Any estimates on how big (in size on disk) would the new unstable repos be?
That would depend on who uses it, and for what package. Nobody is forced to do, and it is not forced for all packages. So for myself this is primarily interesting for system and util-linux, possibly adding mkinitcpio on top every now and then. A release comes every few month to half a year, with roughly three or four pre-releases. I think gromit wanted to push linux-mainline (as a separate package, all the rc releases by Linus) - that would consume a lot more of disk space. No idea who would use it as well. -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}
participants (7)
-
Antonio Rojas
-
Christian Hesse
-
Daniel M. Capella
-
Fabian Bornschein
-
Jelle van der Waa
-
Lukas Fleischer
-
pitastrudl