First I'll go over the basic design of pacbuild, then I'll highlight how far I am, and then I'll explain the the problems I see so far.
My original goal with pacbuild was to write a program that would scan abs for package discrepencies between different architecture's repos (well, between i686 and everything else), build the new package for that architecture, and stick it in the appropriate repo.
I broke the design into four pieces: peach, cherry, strawberry, and waka.
Peach is the web interface for package validation.
Cherry is the main build manager, it scans the abs repo, queues packages, sends them to remote strawberry instances, sets them to be validated, and adds them to the repo after validation.
Strawberry runs on each remote build machine. It gets builds from cherry and sends them to waka to be built.
Waka takes a source package (I can explain this later if needed), builds it in a clean chroot, and spits out a binary package.
Currently cherry does everything up to the validation part and strawberry and waka are complete. Peach is still but a twinkle in my eye.
Now the fun part: problems. I started pacbuild for i586. It worked really well, because packages are very rarely modified from i686 to i586. There was no real need for a cvsup tree and whatnot.
I also designed cherry to do too much. I should probably make another one or two daemons in there. One, at least, to scan the repos and queue packages. That way cherry can just worry about managing all the remote builds. The other daemon would be for putting the packages back into the repos. The only reason I would develop this seperately is then we could use it for i686 as well.
The way pacbuild was designed originally, it needed the single-PKGBUILD-for-all-architectures method. It depended on all PKGBUILDs working for all architectures and tried to build them for all architectures without those architecture developers even having looked at an update. This way architecture developers become more of a manager than an active maintainer.
The problem here is that each architecture's version of a package (if the newer version isn't in the ppc repo, for example) isn't accessible in abs. To do the abs thing for each individual architecture, we need to tag them at the very least (have seperate repos at the most). Tagging them pretty much means they'd need someone looking at them. It's possible that the build script could tag all the ones that were verified, but developers would still need to fix problems that showed up and tag those.
As far as I know, Debian does it with one source package for many architectures. They also have different source sharing methods (I don't know exactly how they work). Frugalware does it with multiple repos, I believe. Gentoo has people sign off on each package on each architecture, but they sort of have to in gentoo...
Comments?
Jason
Jason Chu wrote:
Now the fun part: problems. I started pacbuild for i586. It worked really well, because packages are very rarely modified from i686 to i586. There was no real need for a cvsup tree and whatnot.
I hear you! Cvsup it a overhead only, if you x86 guys had'nt set a password for eradonly anonymous not even bypassing a password prompt would have been necessary. ;-) Not to speak of the fact that Emz3 only runs on a platform where there is already a bootstrapped Emz3 compiler, which I haven't found for PPC yet.
If it were possible I'd like to see everyone move away from it and use plain CVS or Subversion at best.
I also designed cherry to do too much. I should probably make another one or two daemons in there. One, at least, to scan the repos and queue packages. That way cherry can just worry about managing all the remote builds. The other daemon would be for putting the packages back into the repos. The only reason I would develop this seperately is then we could use it for i686 as well.
The way pacbuild was designed originally, it needed the single-PKGBUILD-for-all-architectures method. It depended on all PKGBUILDs working for all architectures and tried to build them for all architectures without those architecture developers even having looked at an update. This way architecture developers become more of a manager than an active maintainer.
I cannot agree more, all of the maintainers should always try to keep the PKGBUILDs exactly the same wherevere possible. Debian does it by the lowest deminor I think, stripping features off the configure until it works everywhere. Archlinux users might not agree with that, so do I. So smaller if[ $CARCH ] sections are totally ok. Unless some really obscure platforms such as ARM32, m68K or even SH-* are planned we won't have a problem. And even then, we could still have like a Archembedded or whatever.
The problem here is that each architecture's version of a package (if the newer version isn't in the ppc repo, for example) isn't accessible in abs. To do the abs thing for each individual architecture, we need to tag them at the very least (have seperate repos at the most). Tagging them pretty much means they'd need someone looking at them. It's possible that the build script could tag all the ones that were verified, but developers would still need to fix problems that showed up and tag those.
Do you mean for instance something like this?
2 kinds of tags in CVS for the latest stable: CURRENT-I686 CURRENT-PPC
Then I do not see a problem with the fact that a new PKGBUILD might not yet be available for all platforms as soon as one platform's maintainer updated it. I think reviewing and kind of signing the package for other architectures is a must, not a problem in itself.
If a user want's stuff that has not necessarily been tested or even been uploaded to be tested, he checks out the HEAD or whatever which always works for getting *all* the latest PKGBULDs, unlike TESTING-I686 on ppc f.e.
- kth5
On Wed, Aug 31, 2005 at 11:58:24PM +0200, Alexander Baldeck wrote:
Jason Chu wrote:
Now the fun part: problems. I started pacbuild for i586. It worked really well, because packages are very rarely modified from i686 to i586. There was no real need for a cvsup tree and whatnot.
I hear you! Cvsup it a overhead only, if you x86 guys had'nt set a password for eradonly anonymous not even bypassing a password prompt would have been necessary. ;-) Not to speak of the fact that Emz3 only runs on a platform where there is already a bootstrapped Emz3 compiler, which I haven't found for PPC yet.
If it were possible I'd like to see everyone move away from it and use plain CVS or Subversion at best.
Well, cvsup is just sort of a seperate daemon for sharing cvs trees. I understand that cactus (I believe that's who it was) wrote some patches for abs to work with subversion. I wrote something similar for the old TUR... it's not hard technically, it's finding acceptance that's so difficult.
I also designed cherry to do too much. I should probably make another one or two daemons in there. One, at least, to scan the repos and queue packages. That way cherry can just worry about managing all the remote builds. The other daemon would be for putting the packages back into the repos. The only reason I would develop this seperately is then we could use it for i686 as well. The way pacbuild was designed originally, it needed the single-PKGBUILD-for-all-architectures method. It depended on all PKGBUILDs working for all architectures and tried to build them for all architectures without those architecture developers even having looked at an update. This way architecture developers become more of a manager than an active maintainer.
I cannot agree more, all of the maintainers should always try to keep the PKGBUILDs exactly the same wherevere possible. Debian does it by the lowest deminor I think, stripping features off the configure until it works everywhere. Archlinux users might not agree with that, so do I. So smaller if[ $CARCH ] sections are totally ok. Unless some really obscure platforms such as ARM32, m68K or even SH-* are planned we won't have a problem. And even then, we could still have like a Archembedded or whatever.
Hehehe, Archembedded.
The problem here is that each architecture's version of a package (if the newer version isn't in the ppc repo, for example) isn't accessible in abs. To do the abs thing for each individual architecture, we need to tag them at the very least (have seperate repos at the most). Tagging them pretty much means they'd need someone looking at them. It's possible that the build script could tag all the ones that were verified, but developers would still need to fix problems that showed up and tag those.
Do you mean for instance something like this?
2 kinds of tags in CVS for the latest stable: CURRENT-I686 CURRENT-PPC
Yeah, basically.
Then I do not see a problem with the fact that a new PKGBUILD might not yet be available for all platforms as soon as one platform's maintainer updated it. I think reviewing and kind of signing the package for other architectures is a must, not a problem in itself.
If a user want's stuff that has not necessarily been tested or even been uploaded to be tested, he checks out the HEAD or whatever which always works for getting *all* the latest PKGBULDs, unlike TESTING-I686 on ppc f.e.
By doing this though, we lose one of the major benefits of having an autobuilder (Debian's autobuilder works like this): package releases for multiple architectures. When a Debian maintainer releases a new i686 package, their buildd detects that it's not available for the other architectures and queues it up to be built. The benefit is that building for the other architectures happens all the time, so there's less catchup that needs doing.
If builds have to be verified first, they're already being built on the developer's boxes for that architecture. Ignoring the benefits of a clean build system (even though they are nice benefits...), why not use the developer's package instead of the autobuilt one?
Thing about getting the HEAD specifically, you're never sure if it's for current or for testing. At least the way we're using it.
Ok, what about this... there are a lot of technologies we'd benefit from using (something other than cvsup, different PKGBUILD tree layout, etc). What if we modelled something off the current arch practices, but substituted our own technologies?
Say we wanted to use svn and lay the repo out differently, we distribute a patched abs that can handle the newer layout and different technology. As long as someone keeps the i686 PKGBUILDs in sync with the ones in the official cvs tree on a regular basis (should be scriptable), we'd get all the benefits of the new technology and we might even be able to convince the other developers to move over.
Doing something like this would be more work on the outset, but would probably be very helpful in the long run (especially if the newer system was adopted ;) ).
Jason
Jason Chu said:
Well, cvsup is just sort of a seperate daemon for sharing cvs trees. I understand that cactus (I believe that's who it was) wrote some patches...
Aye. it was me. I used svn update, while you were using wget. Your solution was probably more scalable, quicker, and less dependent upon subversion being on the client.
Mine had the benefit of being damn simple. ;)
it's not hard technically, it's finding acceptance that's so difficult.
yeah.. /me nods head "bobble head" style
arch-ports@lists.archlinux.org