[arch-general] Question about automated builder

[Thomas S Hatch]

On Thu, Jan 27, 2011 at 4:53 PM, Allan McRae <allan at archlinux.org> wrote:

> On 28/01/11 03:36, Thomas S Hatch wrote:
>
>> I have been passively working on a similar project called quarters, but I
>> must admit that my motivation is somewhat low not knowing if the project
>> is
>> in demand. So here is my question, do we think that something like this
>> would be a benefit to Arch? Is this the type of project that should merit
>> my
>> attention?
>>
>
> An automated build system, in particular for the big rebuilds we do, has
> been discussed among the developers many times.  So there is definitely
> interest in such a system.
>
>
>  Also, if we do think that this would be a good thing to have for Arch, I
>> would like feedback on what types of features the system would have and
>> how
>> it would behave. Right now I am following the idea of supporting a
>> distributed build system so that we can have any number of build servers
>> in
>> the fray working away to produce Arch packages for us. I am also
>> attempting
>> to build it in such a way that a database is not required and that the
>> interface would be amazingly simple (this is Arch after all). This would
>> mean that by mearly checking into svn a package would be built, and then
>> an
>> interface would pop up for the right people to sign it off, and once it
>> has
>> been signed off it would move over.
>>
>
> I have never liked the idea of automatic building after a SVN commit. There
> a plenty of times that commits are made to SVN in preparation for an update
> at a later time.  Instead, I would prefer a command-line queue submission
> tool (local or remote...).  You could have it flexible enough to take a
> "makepkg --source" tarball, or allow configuration for updating then
> building from a SVN/git/etc repo.
>
> I agree with this argument, and I have had many thoughts on how to handle
it, the reason is because I really like the idea of managing from SVN
commits, I think it would make the whole process much easier.

What I don't want is an SVN commit to turn into a package push to a
production repo, that would be bad. if there is a commit for future use in
SVN, and a build happens, thats fine with me, but it could be easily rebuilt
when the "future condition" occurs and the process of moving the package
from the "build bin" to a production repo should probably always be a manual
one.


> Thinking about what I do in packaging (and as someone who does not use the
> AUR), I see the following situations that need to be covered:
>
>
> 1) single package build:  This would probably be the majority use case
>
> The big idea here is more that a system can do the chroot build and
packages checking for the packager, one of the big motivations :)

>
> 2) multi-package build:  Sometimes packages come in pairs/groups where
> updating one requires updating the next.  e.g. tcl and tk.  That requires
> building tcl, adding the new package to the chroot and then building tk.
>
> Note there could be cases where this is more complicated...   e.g. Updating
> package A,B,C,D.   A is required for B, B is required for C and B is
> required for D.   Note that installing C in the chroot after building would
> be unclean for the build of D.  Think updates to desktop environments as an
> example of this.
>
> makechrootpkg has the facility to add packages to a local repo after being
> built, so that could be used to implement this.
>
> I have a solution for this, simply that the builders build every package
alone in its own chroot, if it has a dep, then the dep needs to be built and
added to a repo that the chroot will be able to get to, so more than one
package is never built in a single chroot

>
> 3) big rebuild:  e.g. for library soname bump.  There are two types of
> builds here.   A) the initial builds and B) all the packages requiring
> rebuild because of those in A.   For B, automatic pkgrel bumping would be
> good.  Also, some automatic determination of the build order for B would be
> very helpful.  This would require error handling as if a package low in the
> dependency chain fails, the rest of that chain should not be built.  Again,
> adding the packages to a local repo as the rebuild progresses will allow to
> build packages cleanly against those that are already rebuilt.   Also not
> that there could be no packages in list A (e.g. if we just want to rebuild
> all of [core]).
>
> I don't think that this should be a difficult issue, I am a little
concerned with auto bumping the pkgrel numbers and committing back to svn,
but I am sure that is something I can figure out how to do in a clean and
safe way

>
> 4) flexibility to add makepkg paramaters:  I'm not sure how widespread use
> this would get...
> e.g. the toolchain is built in the following order:
> linux-api-headers->glibc->binutils->gcc->binutils->glibc
> The first glibc and binutils builds will be built with the --nocheck option
> in the future.  The gcc and final binutils and glibc builds are built with
> -L to save the testsuite build logs.  I guess you would make -L the default
> anyway and have them available at the end of the build for inspection.
>
> This is a tricky problem to solve I think, but doable.

>
> Anyway, #3 is what I consider most important in an automated build system
> designed to make the job of packaging in Arch easier.  It is also the level
> where I would start using such a system....   Building a single package
> locally is a two "makechrootpkg" commands (one per architecture) and I do
> not have to transfer the resultant package to test.  So I doubt I would use
> a build system for that (unless build time was massive) as I can not see the
> automated build being more efficient for me.
>
> Allan
>

I really do hope it can make Arch packaging easier, and raise the overall
quality of Arch packages (not that I think that it is too terribly needed)

As for 3 and 4, Koji can't do those, which makes me all the more excited to
give them a try!

Thanks for the input Allan, it is nice to hear from a big gun.

-Thomas S Hatch