[arch-dev-public] automatic package builder
Hey guys, Past week I was reading through the buildbot manual, but I decided not to use it. IMO it is far too complex, fat and not something which has the needed functionality easily available (you can extend it with python scripting, but what's the fun of that huh?) So yesterday I decided I will write my own. I'll give it some more though what functionality we need most (ability to prioritize tasks, manually adding tasks, getting changes from svn to name a few). So what do I need from you? - name (what's a tool without a name huh?) - git-repo (not direct, but sometime in the next two weeks or so. Maybe we need to first pick a name too) - ideas. What do you guys want to see first? How do we want to get notified (daily mail, using a webpage which lists success of all completed/pending tasks, something else)? You get the idea, now is the time to speak up. Ronald
On Fri, Sep 25, 2009 at 10:39 AM, Ronald van Haren <pressh@gmail.com> wrote:
Hey guys,
Past week I was reading through the buildbot manual, but I decided not to use it. IMO it is far too complex, fat and not something which has the needed functionality easily available (you can extend it with python scripting, but what's the fun of that huh?)
This buildbot : http://buildbot.net/trac ?
So yesterday I decided I will write my own. I'll give it some more though what functionality we need most (ability to prioritize tasks, manually adding tasks, getting changes from svn to name a few).
So what do I need from you? - name (what's a tool without a name huh?) - git-repo (not direct, but sometime in the next two weeks or so. Maybe we need to first pick a name too) - ideas. What do you guys want to see first? How do we want to get notified (daily mail, using a webpage which lists success of all completed/pending tasks, something else)? You get the idea, now is the time to speak up.
By the way, I am curious about pacbuild : http://wiki.archlinux.org/index.php/Pacbuild_Explained What do people think about it ?
On Fri, Sep 25, 2009 at 11:01 AM, Xavier <shiningxc@gmail.com> wrote:
On Fri, Sep 25, 2009 at 10:39 AM, Ronald van Haren <pressh@gmail.com> wrote:
Hey guys,
Past week I was reading through the buildbot manual, but I decided not to use it. IMO it is far too complex, fat and not something which has the needed functionality easily available (you can extend it with python scripting, but what's the fun of that huh?)
This buildbot : http://buildbot.net/trac ?
yes. That's the one I don't like
So yesterday I decided I will write my own. I'll give it some more though what functionality we need most (ability to prioritize tasks, manually adding tasks, getting changes from svn to name a few).
So what do I need from you? - name (what's a tool without a name huh?) - git-repo (not direct, but sometime in the next two weeks or so. Maybe we need to first pick a name too) - ideas. What do you guys want to see first? How do we want to get notified (daily mail, using a webpage which lists success of all completed/pending tasks, something else)? You get the idea, now is the time to speak up.
By the way, I am curious about pacbuild : http://wiki.archlinux.org/index.php/Pacbuild_Explained What do people think about it ?
On Fri, Sep 25, 2009 at 4:01 AM, Xavier <shiningxc@gmail.com> wrote:
By the way, I am curious about pacbuild : http://wiki.archlinux.org/index.php/Pacbuild_Explained What do people think about it ?
Honest opinion? It was over-engineered from the start and is far to complex to use. You need like 3 daemons running, and have to do manual database insertions (why does it have a database?!) to configure things. - - - - - makechrootpkg was made in such a way that is can and should be used for this purpose. Here's how I would setup a "build machine". a) Create two global chroots, one for "current" and one for "testing". We'll call them /var/archroot and /var/archroot-testing (alternative: append arch on there and create 4 total. Requires a 64bit machine and linux32 usage) b) Scan svn commits SOMEHOW. This could be done by watching the arch-commits list, or simply running "svn up" in a job and parse changes c) For each trunk commit found, do an svn export of the trunk and run "makechrootpkg -c -u -r/var/archroot -l $pkgname" d) Update the read-only layer of the chroot periodically with "mkarchroot -u /var/archroot/root" e) Run namcap on PKGBUILDs and resultant packages f) Place built packages in a specific staging ground we have access to. They still require testing before being promoted to the repos. As for notifications, we webpage AND email notifications would be ideal. Perhaps batched emails every hour or so? "The follow packages work / The following failed hard"?
Aaron Griffin wrote:
On Fri, Sep 25, 2009 at 4:01 AM, Xavier <shiningxc@gmail.com> wrote:
By the way, I am curious about pacbuild : http://wiki.archlinux.org/index.php/Pacbuild_Explained What do people think about it ?
Honest opinion? It was over-engineered from the start and is far to complex to use. You need like 3 daemons running, and have to do manual database insertions (why does it have a database?!) to configure things.
- - - - -
makechrootpkg was made in such a way that is can and should be used for this purpose. Here's how I would setup a "build machine".
a) Create two global chroots, one for "current" and one for "testing". We'll call them /var/archroot and /var/archroot-testing (alternative: append arch on there and create 4 total. Requires a 64bit machine and linux32 usage)
b) Scan svn commits SOMEHOW. This could be done by watching the arch-commits list, or simply running "svn up" in a job and parse changes
c) For each trunk commit found, do an svn export of the trunk and run "makechrootpkg -c -u -r/var/archroot -l $pkgname"
This is the part I really do not like about this. I commit to trunk when 1) I have updated, _built_ and am in the process of committing a package. 2) I have an idea/bug fix for the next release and do not want to make a new release now. In both those cases, a commit to trunk does not need a build. In all honesty, I would think that having downloaded the source and updated the PKGBUILD, you would probably have built the package locally anyway to test it all out. So building for #1 seems annoying as I would then need to download the built package and retest etc, or just not use it... How about, to build a package with the build bot, you send an email to buildbot@archlinux with content "pkgname carch", and if it is from an authorized user, the buildbot starts. Multiple lines = multiple builds? Needs more thought... Anyway, what exactly are the goals of the buildbot? I see these uses: 1) A developer needs to build packages for an arch they do not have 2) Helping out with big rebuilds #1 does not need to monitor trunk as one arch would already be built and tested by the developer and it probably should not start automatically as who commits both arches at the same time... For #2, you need to provide a tree with the dependency chain anyway, and I have yet to strike a rebuild that did not require manual intervention for fixing PKGBUILDs (build fixing patching, update versioned deps). So this is what I would like to see set-up: 1) A way to build individual (or a small group of) packages on request (e.g. a dev does not have access to an arch) 2) A way to handle major rebuilds, including generating a build order (I have a script that somewhat generates a build order and Aaron started a script to do the building) 3) When there are no packages queued to be built, check packages in our repos for being able to build in a clean chroot. I started to write a daemon to do #3 but decided my approach was bad so scrapped most of it. It really needed written in a language other than bash to be useful (needs to store all info about last build time and success in a database and generate web reports). From doing many large rebuilds, I see #3 as being essential to successfully doing #2. Allan
On Fri, Sep 25, 2009 at 6:17 PM, Allan McRae <allan@archlinux.org> wrote:
Aaron Griffin wrote:
On Fri, Sep 25, 2009 at 4:01 AM, Xavier <shiningxc@gmail.com> wrote:
By the way, I am curious about pacbuild : http://wiki.archlinux.org/index.php/Pacbuild_Explained What do people think about it ?
Honest opinion? It was over-engineered from the start and is far to complex to use. You need like 3 daemons running, and have to do manual database insertions (why does it have a database?!) to configure things.
- - - - -
makechrootpkg was made in such a way that is can and should be used for this purpose. Here's how I would setup a "build machine".
a) Create two global chroots, one for "current" and one for "testing". We'll call them /var/archroot and /var/archroot-testing (alternative: append arch on there and create 4 total. Requires a 64bit machine and linux32 usage)
b) Scan svn commits SOMEHOW. This could be done by watching the arch-commits list, or simply running "svn up" in a job and parse changes
c) For each trunk commit found, do an svn export of the trunk and run "makechrootpkg -c -u -r/var/archroot -l $pkgname"
This is the part I really do not like about this. I commit to trunk when 1) I have updated, _built_ and am in the process of committing a package. 2) I have an idea/bug fix for the next release and do not want to make a new release now.
In both those cases, a commit to trunk does not need a build. In all honesty, I would think that having downloaded the source and updated the PKGBUILD, you would probably have built the package locally anyway to test it all out. So building for #1 seems annoying as I would then need to download the built package and retest etc, or just not use it...
How about, to build a package with the build bot, you send an email to buildbot@archlinux with content "pkgname carch", and if it is from an authorized user, the buildbot starts. Multiple lines = multiple builds? Needs more thought...
Anyway, what exactly are the goals of the buildbot? I see these uses: 1) A developer needs to build packages for an arch they do not have 2) Helping out with big rebuilds
#1 does not need to monitor trunk as one arch would already be built and tested by the developer and it probably should not start automatically as who commits both arches at the same time... For #2, you need to provide a tree with the dependency chain anyway, and I have yet to strike a rebuild that did not require manual intervention for fixing PKGBUILDs (build fixing patching, update versioned deps).
So this is what I would like to see set-up: 1) A way to build individual (or a small group of) packages on request (e.g. a dev does not have access to an arch) 2) A way to handle major rebuilds, including generating a build order (I have a script that somewhat generates a build order and Aaron started a script to do the building) 3) When there are no packages queued to be built, check packages in our repos for being able to build in a clean chroot.
I started to write a daemon to do #3 but decided my approach was bad so scrapped most of it. It really needed written in a language other than bash to be useful (needs to store all info about last build time and success in a database and generate web reports). From doing many large rebuilds, I see #3 as being essential to successfully doing #2.
The building from trunk thing is more based on my workflow, but I really don't maintain much these days, so you can ignore it. Here's why I was thinking that way: when I'm at work and see that some package needs something changed, I just update svn considering I'm not on an Arch machine. I don't actually do the build, I just make changes and commit them. Now, your idea with some way to poke the thing to actually do a build (via email, a web page, command line app, etc) would still work here, so it sounds like a better idea.
On Sat, Sep 26, 2009 at 10:35 PM, Aaron Griffin <aaronmgriffin@gmail.com>wrote:
On Fri, Sep 25, 2009 at 6:17 PM, Allan McRae <allan@archlinux.org> wrote:
Aaron Griffin wrote:
On Fri, Sep 25, 2009 at 4:01 AM, Xavier <shiningxc@gmail.com> wrote:
By the way, I am curious about pacbuild : http://wiki.archlinux.org/index.php/Pacbuild_Explained What do people think about it ?
Honest opinion? It was over-engineered from the start and is far to complex to use. You need like 3 daemons running, and have to do manual database insertions (why does it have a database?!) to configure things.
- - - - -
makechrootpkg was made in such a way that is can and should be used for this purpose. Here's how I would setup a "build machine".
a) Create two global chroots, one for "current" and one for "testing". We'll call them /var/archroot and /var/archroot-testing (alternative: append arch on there and create 4 total. Requires a 64bit machine and linux32 usage)
b) Scan svn commits SOMEHOW. This could be done by watching the arch-commits list, or simply running "svn up" in a job and parse changes
c) For each trunk commit found, do an svn export of the trunk and run "makechrootpkg -c -u -r/var/archroot -l $pkgname"
This is the part I really do not like about this. I commit to trunk when 1) I have updated, _built_ and am in the process of committing a package. 2) I have an idea/bug fix for the next release and do not want to make a new release now.
In both those cases, a commit to trunk does not need a build. In all honesty, I would think that having downloaded the source and updated the PKGBUILD, you would probably have built the package locally anyway to test it all out. So building for #1 seems annoying as I would then need to download the built package and retest etc, or just not use it...
How about, to build a package with the build bot, you send an email to buildbot@archlinux with content "pkgname carch", and if it is from an authorized user, the buildbot starts. Multiple lines = multiple builds? Needs more thought...
Anyway, what exactly are the goals of the buildbot? I see these uses: 1) A developer needs to build packages for an arch they do not have 2) Helping out with big rebuilds
#1 does not need to monitor trunk as one arch would already be built and tested by the developer and it probably should not start automatically as who commits both arches at the same time... For #2, you need to provide a tree with the dependency chain anyway, and I have yet to strike a rebuild that did not require manual intervention for fixing PKGBUILDs (build fixing patching, update versioned deps).
So this is what I would like to see set-up: 1) A way to build individual (or a small group of) packages on request (e.g. a dev does not have access to an arch) 2) A way to handle major rebuilds, including generating a build order (I have a script that somewhat generates a build order and Aaron started a script to do the building) 3) When there are no packages queued to be built, check packages in our repos for being able to build in a clean chroot.
I started to write a daemon to do #3 but decided my approach was bad so scrapped most of it. It really needed written in a language other than bash to be useful (needs to store all info about last build time and success in a database and generate web reports). From doing many large rebuilds, I see #3 as being essential to successfully doing #2.
The building from trunk thing is more based on my workflow, but I really don't maintain much these days, so you can ignore it.
Here's why I was thinking that way: when I'm at work and see that some package needs something changed, I just update svn considering I'm not on an Arch machine. I don't actually do the build, I just make changes and commit them.
Now, your idea with some way to poke the thing to actually do a build (via email, a web page, command line app, etc) would still work here, so it sounds like a better idea.
both getting changes from trunk or getting build packages on request should be pretty easy to implement. My initial thought was building from requests and, when no requests available somehow determine what packages from core/extra may be good to have a rebuild. I also would like to be able to add priority flags (high, normal, low, default to normal) so that we can add packages in front of the queue if needed sometime. If no requests are pending, we should do something else usefull, like building old packages against new toolchains and such, but there may be other more important variables (we really should define this better). We should probably also automatically bump the pkgver in these cases. I'd like to have some more dev input on what some of the others like to see so everyone can be happy in the end. Ronald
participants (4)
-
Aaron Griffin
-
Allan McRae
-
Ronald van Haren
-
Xavier