On Thu, Jan 27, 2011 at 4:53 PM, Allan McRae <allan@archlinux.org> wrote:
On 28/01/11 03:36, Thomas S Hatch wrote:
I have been passively working on a similar project called quarters, but I must admit that my motivation is somewhat low not knowing if the project is in demand. So here is my question, do we think that something like this would be a benefit to Arch? Is this the type of project that should merit my attention?
An automated build system, in particular for the big rebuilds we do, has been discussed among the developers many times. So there is definitely interest in such a system.
Also, if we do think that this would be a good thing to have for Arch, I
would like feedback on what types of features the system would have and how it would behave. Right now I am following the idea of supporting a distributed build system so that we can have any number of build servers in the fray working away to produce Arch packages for us. I am also attempting to build it in such a way that a database is not required and that the interface would be amazingly simple (this is Arch after all). This would mean that by mearly checking into svn a package would be built, and then an interface would pop up for the right people to sign it off, and once it has been signed off it would move over.
I have never liked the idea of automatic building after a SVN commit. There a plenty of times that commits are made to SVN in preparation for an update at a later time. Instead, I would prefer a command-line queue submission tool (local or remote...). You could have it flexible enough to take a "makepkg --source" tarball, or allow configuration for updating then building from a SVN/git/etc repo.
I agree with this argument, and I have had many thoughts on how to handle it, the reason is because I really like the idea of managing from SVN commits, I think it would make the whole process much easier.
Thinking about what I do in packaging (and as someone who does not use the AUR), I see the following situations that need to be covered:
1) single package build: This would probably be the majority use case
The big idea here is more that a system can do the chroot build and
What I don't want is an SVN commit to turn into a package push to a production repo, that would be bad. if there is a commit for future use in SVN, and a build happens, thats fine with me, but it could be easily rebuilt when the "future condition" occurs and the process of moving the package from the "build bin" to a production repo should probably always be a manual one. packages checking for the packager, one of the big motivations :)
2) multi-package build: Sometimes packages come in pairs/groups where updating one requires updating the next. e.g. tcl and tk. That requires building tcl, adding the new package to the chroot and then building tk.
Note there could be cases where this is more complicated... e.g. Updating package A,B,C,D. A is required for B, B is required for C and B is required for D. Note that installing C in the chroot after building would be unclean for the build of D. Think updates to desktop environments as an example of this.
makechrootpkg has the facility to add packages to a local repo after being built, so that could be used to implement this.
I have a solution for this, simply that the builders build every package
alone in its own chroot, if it has a dep, then the dep needs to be built and added to a repo that the chroot will be able to get to, so more than one package is never built in a single chroot
3) big rebuild: e.g. for library soname bump. There are two types of builds here. A) the initial builds and B) all the packages requiring rebuild because of those in A. For B, automatic pkgrel bumping would be good. Also, some automatic determination of the build order for B would be very helpful. This would require error handling as if a package low in the dependency chain fails, the rest of that chain should not be built. Again, adding the packages to a local repo as the rebuild progresses will allow to build packages cleanly against those that are already rebuilt. Also not that there could be no packages in list A (e.g. if we just want to rebuild all of [core]).
I don't think that this should be a difficult issue, I am a little
concerned with auto bumping the pkgrel numbers and committing back to svn, but I am sure that is something I can figure out how to do in a clean and safe way
4) flexibility to add makepkg paramaters: I'm not sure how widespread use this would get... e.g. the toolchain is built in the following order: linux-api-headers->glibc->binutils->gcc->binutils->glibc The first glibc and binutils builds will be built with the --nocheck option in the future. The gcc and final binutils and glibc builds are built with -L to save the testsuite build logs. I guess you would make -L the default anyway and have them available at the end of the build for inspection.
This is a tricky problem to solve I think, but doable.
Anyway, #3 is what I consider most important in an automated build system designed to make the job of packaging in Arch easier. It is also the level where I would start using such a system.... Building a single package locally is a two "makechrootpkg" commands (one per architecture) and I do not have to transfer the resultant package to test. So I doubt I would use a build system for that (unless build time was massive) as I can not see the automated build being more efficient for me.
Allan
I really do hope it can make Arch packaging easier, and raise the overall quality of Arch packages (not that I think that it is too terribly needed) As for 3 and 4, Koji can't do those, which makes me all the more excited to give them a try! Thanks for the input Allan, it is nice to hear from a big gun. -Thomas S Hatch