[arch-dev-public] Redefining [testing] and a new [staging] repo

newer
[arch-dev-public] [toolchain] gcc...

Pierre Schmitz

7 May 2010 7 May '10

4:44 p.m.

Hi all, here is another crazy idea I was thinking about for some days. This should improve our overall package quality and simplify our work flow. The problem with out current [testing] repository are these: * it is only used by very few people. Most of the times we don't get much feedback until packages are moved to core or extra * it is often in a inconsistent state; especially during incomplete so name bump rebuilds * sometimes packages are known to be broken or unstable On the other side we are sometimes in need of some intermediate repository. For example single people stack a pile of packages in their home dirs until they can be moved to a repo at once. We have also seen temporary repos like jpng to manage larger rebuilds. My idea is to redefine the testing repo and introduce a new staging one. (remember the good old days? ;-)) With the implementation of the package pooling moving packages between repos can be done with nearly no overhead. So ideally we should be able to use testing more often, even for a very short period like a day. proposal for [testing] * never push any known-to-be-broken packages here (for example incomplete rebuilds) * candidates for core are put here * ideally new builds of important/critical packages or major rebuilds can be put here to test them by a larger audience. * don't put any package in here which are never meant to be moved to core/extra. (like experimental alpha software etc.) proposal for [staging] * this repo should only used by devs and tus * it is not meant to be used on a production system * it should only be enabled in a packages build environment (chroot) * this could be even excluded from outbound rsync. Due to package pooling packages would still be propagated. And moving those packages to other repos will be "instantly". Summing things up more packages should be passed through testing, more users should be able to use testing without breaking their systems and we don't have to make them reading the high traffic arch-dev-public list. We also should be able to collaborate more in packaging: Dev A puts a new lib into [staging] and another one can start rebuild other packages using that lib. Another common use case would be large builds like KDE we usually start packaging one week before release. Till now those were put into users' staging dirs and the other devs had to manually download them, create a local repo and install them. So, what do you think about this idea? Greetings, Pierre -- Pierre Schmitz, https://users.archlinux.de/~pierre

Show replies by date

Aaron Griffin

7 May 7 May

4:50 p.m.

On Fri, May 7, 2010 at 11:44 AM, Pierre Schmitz wrote:

...

Hi all,

here is another crazy idea I was thinking about for some days. This should improve our overall package quality and simplify our work flow. The problem with out current [testing] repository are these: * it is only used by very few people. Most of the times we don't get much feedback until packages are moved to core or extra * it is often in a inconsistent state; especially during incomplete so name bump rebuilds * sometimes packages are known to be broken or unstable

On the other side we are sometimes in need of some intermediate repository. For example single people stack a pile of packages in their home dirs until they can be moved to a repo at once. We have also seen temporary repos like jpng to manage larger rebuilds.

My idea is to redefine the testing repo and introduce a new staging one. (remember the good old days? ;-)) With the implementation of the package pooling moving packages between repos can be done with nearly no overhead. So ideally we should be able to use testing more often, even for a very short period like a day.

proposal for [testing] * never push any known-to-be-broken packages here (for example incomplete rebuilds) * candidates for core are put here * ideally new builds of important/critical packages or major rebuilds can be put here to test them by a larger audience. * don't put any package in here which are never meant to be moved to core/extra. (like experimental alpha software etc.)

proposal for [staging] * this repo should only used by devs and tus * it is not meant to be used on a production system * it should only be enabled in a packages build environment (chroot) * this could be even excluded from outbound rsync. Due to package pooling packages would still be propagated. And moving those packages to other repos will be "instantly".

Summing things up more packages should be passed through testing, more users should be able to use testing without breaking their systems and we don't have to make them reading the high traffic arch-dev-public list. We also should be able to collaborate more in packaging: Dev A puts a new lib into [staging] and another one can start rebuild other packages using that lib. Another common use case would be large builds like KDE we usually start packaging one week before release. Till now those were put into users' staging dirs and the other devs had to manually download them, create a local repo and install them.

So, what do you think about this idea?

I don't like it. It adds a lot of complexity. For large rebuilds, the dbscripts DO allow us to make on-the-fly repos, though currently the directory needs to be created by an admin (that can be changed). We've done this before with some releases, and I think it's a good idea to use this capability, rather than define a testing-testing repo (that's what this is - a testing repo before packages go to testing...)

Pierre Schmitz

5:02 p.m.

On Fri, 7 May 2010 11:50:11 -0500, Aaron Griffin wrote:

...

I don't like it. It adds a lot of complexity.

For large rebuilds, the dbscripts DO allow us to make on-the-fly repos, though currently the directory needs to be created by an admin (that can be changed).

Having a repo for just this purpose will make things easier and I don't see where it would increase complexity.

...

We've done this before with some releases, and I think it's a good idea to use this capability, rather than define a testing-testing repo (that's what this is - a testing repo before packages go to testing...)

No you got me wrong here. It's not meant to be to test packages before moving them to testing. Most packages will still be moved directly to extra or testing. The staging repo would be just a place where we assemble larger builds or could coordinate building with other devs without destroying a real repo like testing. So, staging is no real repo, its not in a chain like (staging->testing->extra) and our current work flow of directly putting packages into testing or extra wouldn't be touched. I hope I made my point this time; though you are still allowed to dislike it. :P -- Pierre Schmitz, https://users.archlinux.de/~pierre

Aaron Griffin

5:13 p.m.

On Fri, May 7, 2010 at 12:02 PM, Pierre Schmitz wrote:

...

On Fri, 7 May 2010 11:50:11 -0500, Aaron Griffin wrote:

...
I don't like it. It adds a lot of complexity.

For large rebuilds, the dbscripts DO allow us to make on-the-fly repos, though currently the directory needs to be created by an admin (that can be changed).

Having a repo for just this purpose will make things easier and I don't see where it would increase complexity.

...
We've done this before with some releases, and I think it's a good idea to use this capability, rather than define a testing-testing repo (that's what this is - a testing repo before packages go to testing...)

No you got me wrong here. It's not meant to be to test packages before moving them to testing. Most packages will still be moved directly to extra or testing. The staging repo would be just a place where we assemble larger builds or could coordinate building with other devs without destroying a real repo like testing.

So, staging is no real repo, its not in a chain like (staging->testing->extra) and our current work flow of directly putting packages into testing or extra wouldn't be touched.

I hope I made my point this time; though you are still allowed to dislike it. :P

So in effect, it seems we're saying the same things, except I am using REPO_NAME="foo-rebuild" for each rebuild "foo", and you are using REPO_NAME="staging" for *all* rebuilds. This can already be done, the staging dir just needs to be added to the server. It would, however, need to be excluded from rsync if you want that.

Pierre Schmitz

5:40 p.m.

On Fri, 7 May 2010 12:13:33 -0500, Aaron Griffin wrote:

...

So in effect, it seems we're saying the same things, except I am using REPO_NAME="foo-rebuild" for each rebuild "foo", and you are using REPO_NAME="staging" for *all* rebuilds.

Somehow yes, but the detail is quite important. But a already existing repo will make things a lot easier. Maybe its worth to manually create a separate repo but probably not for just a few. We also sometimes upload packages somewhere to share or let others grab them from our home dirs on gerolde. Such a repo would simplify this and induce a general work flow for e.g. rebuilds. But still, the main intention is to make testing more usable to get more feedback and in turn have less bugs in core/extra.

...

This can already be done, the staging dir just needs to be added to the server.

It would, however, need to be excluded from rsync if you want that.

Sure, the implementation of this is a peace of cake: mostly a mkdir and maybe adding some shortcuts to devtools and dbscripts to have something like stagingpkg and staging2extra etc.. -- Pierre Schmitz, https://users.archlinux.de/~pierre

Andreas Radke

8 May 8 May

7:58 a.m.

I don't see the need for any additional repo. I wouldn't use it. -Andy

Allan McRae

9 May 9 May

6:49 a.m.

On 08/05/10 02:44, Pierre Schmitz wrote:

...

proposal for [staging] * this repo should only used by devs and tus * it is not meant to be used on a production system * it should only be enabled in a packages build environment (chroot) * this could be even excluded from outbound rsync. Due to package pooling packages would still be propagated. And moving those packages to other repos will be "instantly".

So, things like the kde-unstable, gnome-unstable, xorg18 repos should go in [staging]? Or should they go straight into [testing]? Because people actually use them so [staging] may be too polluted for them, but they are not ready for [testing]... do we need another repo in-between? :P How about we just make the group owner of the root of repo to allow people with push access to [core] to create repos and have db-scripts only fail if [community{,-testing}] is used on gerolde? Allan

Pierre Schmitz

10:09 a.m.

The biggest issue about this idea seems to be my bad description. ;-) So please ask questions and I'll try to make my point a little more obvious. I was asked if this concept would delay the release of new packages. The answer is of course not. It would even speed things up due to easier collaboration between devs. And if due to this new work flow more people would use testing we'll also get more feedback and could move packages sooner to core/extra. On Sun, 09 May 2010 16:49:55 +1000, Allan McRae wrote:

...

So, things like the kde-unstable, gnome-unstable, xorg18 repos should go

...

in [staging]? Or should they go straight into [testing]? Because people actually use them so [staging] may be too polluted for them, but they are not ready for [testing]... do we need another repo in-between?

...

:P

No, you got me wrong here. The staging repo will only include packages that could be moved to core/extra or testing later. Those *-unstable repos contain beta versions that will never be pushed into any other repo. So those special repos could still be used and wont be affected. My goal was to have more people using testing. We could only achieve this if we make this repo more reliable. The main problem are incomplete rebuilds and those should be assembled in staging. So, most of the time staging will probably empty and packages that will be pushed there will stay for a quite short time. And there is no active testing or even sign-offs needed to move them to another repo. An example: Let's say I build openssl 1.0.1 with a so name bump. I wont have to rebuild at least core packages and no warning to not use testing atm has to be issued. I would just say "Hey, I just put new openssl into staging; please help rebuilding". Once we finished our rebuild list all those packages should be moved to testing and we can actually start using them. Using this approach the testing repo will never be inconsistent and more people could use it without the need to watch arch-dev-public for warnings. Of course this doesn't mean there wont be broken packages in testing; it just means that we don'T break things intentionally any more.

...

How about we just make the group owner of the root of repo to allow people with push access to [core] to create repos and have db-scripts only fail if [community{,-testing}] is used on gerolde?

That's probably something we want to do anyway. But it's not the solution to this problem here. I don't believe anybody will create a new repo for just a couple of rebuilds, adjust their pacman.conf etc.. They will just keep pushing unfinished rebuilds straight to testing. That's why I think this step needs to be made easy and without any overhead. -- Pierre Schmitz, https://users.archlinux.de/~pierre

Eric Bélanger

5:48 p.m.

On Sun, May 9, 2010 at 6:09 AM, Pierre Schmitz wrote:

...

The biggest issue about this idea seems to be my bad description. ;-) So please ask questions and I'll try to make my point a little more obvious.

I was asked if this concept would delay the release of new packages. The answer is of course not. It would even speed things up due to easier collaboration between devs. And if due to this new work flow more people would use testing we'll also get more feedback and could move packages sooner to core/extra.

On Sun, 09 May 2010 16:49:55 +1000, Allan McRae wrote:

...
So, things like the kde-unstable, gnome-unstable, xorg18 repos should go

...
in [staging]? Or should they go straight into [testing]? Because people actually use them so [staging] may be too polluted for them, but they are not ready for [testing]... do we need another repo in-between?

...
:P

No, you got me wrong here. The staging repo will only include packages that could be moved to core/extra or testing later. Those *-unstable repos contain beta versions that will never be pushed into any other repo. So those special repos could still be used and wont be affected.

My goal was to have more people using testing. We could only achieve this if we make this repo more reliable. The main problem are incomplete rebuilds and those should be assembled in staging. So, most of the time staging will probably empty and packages that will be pushed there will stay for a quite short time. And there is no active testing or even sign-offs needed to move them to another repo.

An example: Let's say I build openssl 1.0.1 with a so name bump. I wont have to rebuild at least core packages and no warning to not use testing atm has to be issued. I would just say "Hey, I just put new openssl into staging; please help rebuilding". Once we finished our rebuild list all those packages should be moved to testing and we can actually start using them.

Using this approach the testing repo will never be inconsistent and more people could use it without the need to watch arch-dev-public for warnings. Of course this doesn't mean there wont be broken packages in testing; it just means that we don'T break things intentionally any more.

I like this idea. It will be easier to collaborate on rebuilds. And you would only to add the staging repository once to your chroot instead of adding/removing a repository for every rebuild (e.g. jpng).

...

...
How about we just make the group owner of the root of repo to allow people with push access to [core] to create repos and have db-scripts only fail if [community{,-testing}] is used on gerolde?

That's probably something we want to do anyway. But it's not the solution to this problem here. I don't believe anybody will create a new repo for just a couple of rebuilds, adjust their pacman.conf etc.. They will just keep pushing unfinished rebuilds straight to testing. That's why I think this step needs to be made easy and without any overhead.

-- Pierre Schmitz, https://users.archlinux.de/~pierre

5110

Age (days ago)

5112

Last active (days ago)

List overview

Download

8 comments

5 participants

participants (5)

Aaron Griffin
Allan McRae
Andreas Radke
Eric Bélanger
Pierre Schmitz