[arch-dev-public] Scripting large rebuilds
Hey guys, Does anyone have any scripts that they use for mass-rebuilds (such as the readline rebuild)? I would like to get more tools of the sort into devtools, so want to see if anyone has anything existing that I could import / use for this purpose. If not, I will write one to do this with makechrootpkg (yay, chroots) to make this easier. Cheers, Aaron
Aaron Griffin wrote:
Hey guys, Does anyone have any scripts that they use for mass-rebuilds (such as the readline rebuild)? I would like to get more tools of the sort into devtools, so want to see if anyone has anything existing that I could import / use for this purpose.
If not, I will write one to do this with makechrootpkg (yay, chroots) to make this easier.
Cheers, Aaron
Not directly but the attached script could be useful. It takes a file of package names and determines the order they need built. At the moment it parses pacman output to get depends so it misses makedeps so that needs fixed. Can makeworld be adjusted to do what you are looking for? Allan #! /usr/bin/env python import os, sys def get_deps( package ): deps = [] pkginfo = os.popen("pacman -Si " + package + " 2> /dev/null").read().split("\n") if pkginfo != ['']: found = False for i in pkginfo: if not found: if i[0:10] == "Depends On": deplist = i.split()[3:] found = True else: if i[0] != " ": break deplist += i.split() for i in deplist: pkg = i.split(">")[0].split("<")[0].split("=")[0] if os.popen("pacman -Si " + pkg + " 2> /dev/null").read().split("\n") != ['']: deps.append(pkg) return deps rebuild_file = open(sys.argv[1], "r") rebuild_list = rebuild_file.read().split() rebuild_deps = {} package_deps = {} for i in rebuild_list: package_deps[i] = get_deps(i) for package in rebuild_list: rebuild_deps[package] = package_deps[package][:] pos = 0 while pos < len(rebuild_deps[package]): dep = rebuild_deps[package][pos] if not package_deps.has_key(dep): package_deps[dep] = get_deps(dep) for i in package_deps[dep]: if not i in rebuild_deps[package]: rebuild_deps[package].append(i) pos += 1 for package in rebuild_list: pos = 0 while pos < len(rebuild_deps[package]): if rebuild_deps[package][pos] in rebuild_list: pos += 1 else: rebuild_deps[package].pop(pos) filtered_deps = {} for package in rebuild_list: filtered_deps[package] = rebuild_deps[package][:] if len(filtered_deps[package]) > 1: pos=0 while pos < len(filtered_deps[package]) and len(filtered_deps[package]) > 1: removed = False for i in filtered_deps[package]: if filtered_deps[package][pos] in rebuild_deps[i]: filtered_deps[package].pop(pos) removed = True break if not removed: pos += 1 for package in rebuild_list: if len(rebuild_deps[package]) == 0: print package else: rebuild_deps[package].sort() print package + " (Requires " + ", ".join(["%s" % pkg for pkg in filtered_deps[package]]) +")"
On Mon, Feb 23, 2009 at 3:18 PM, Allan McRae <allan@archlinux.org> wrote:
Aaron Griffin wrote:
Hey guys, Does anyone have any scripts that they use for mass-rebuilds (such as the readline rebuild)? I would like to get more tools of the sort into devtools, so want to see if anyone has anything existing that I could import / use for this purpose.
If not, I will write one to do this with makechrootpkg (yay, chroots) to make this easier.
Cheers, Aaron
Not directly but the attached script could be useful. It takes a file of package names and determines the order they need built. At the moment it parses pacman output to get depends so it misses makedeps so that needs fixed.
Looks neat - might be useful in devtools on its own.
Can makeworld be adjusted to do what you are looking for?
Hmm probably not. I was thinking something like: $ omg-rebuilds /path/to/svn/checkout /path/to/chroot/dir -- readline bash foo bar baz Or something to that effect, that would not only build all packages in a chroot, but auto-bump the pkgrels and dump the correct results to your svn dir, for a later run of testingpkg...
Am Montag, 23. Februar 2009 22:26:36 schrieb Aaron Griffin:
Or something to that effect, that would not only build all packages in a chroot, but auto-bump the pkgrels and dump the correct results to your svn dir, for a later run of testingpkg...
Would be ncie to have some helper scripts. I use some dumb scripts to manage the kde rebuilds: http://git.archlinux.de/build-scripts/tree/kde Maybe some of those can be extended to be more genereal. Espeically this which sets pkgver, pkgrel and updates the md5sums: http://git.archlinux.de/build- scripts/tree/kde/update-pkgbuilds (the awk scripts is stolen from some post of the forums ;-)) -- Pierre Schmitz Clemens-August-Straße 76 53115 Bonn Telefon 0228 9716608 Mobil 0160 95269831 Jabber pierre@jabber.archlinux.de WWW http://www.archlinux.de
On Mon, Feb 23, 2009 at 3:36 PM, Pierre Schmitz <pierre@archlinux.de> wrote:
Am Montag, 23. Februar 2009 22:26:36 schrieb Aaron Griffin:
Or something to that effect, that would not only build all packages in a chroot, but auto-bump the pkgrels and dump the correct results to your svn dir, for a later run of testingpkg...
Would be ncie to have some helper scripts. I use some dumb scripts to manage the kde rebuilds: http://git.archlinux.de/build-scripts/tree/kde
Maybe some of those can be extended to be more genereal. Espeically this which sets pkgver, pkgrel and updates the md5sums: http://git.archlinux.de/build- scripts/tree/kde/update-pkgbuilds (the awk scripts is stolen from some post of the forums ;-))
And I just noticed the undocumented -i flag to makechrootpkg, which installs a package without dirtying the chroot - this just became way easier :)
Aaron Griffin wrote:
On Mon, Feb 23, 2009 at 3:36 PM, Pierre Schmitz <pierre@archlinux.de> wrote:
Am Montag, 23. Februar 2009 22:26:36 schrieb Aaron Griffin:
Or something to that effect, that would not only build all packages in a chroot, but auto-bump the pkgrels and dump the correct results to your svn dir, for a later run of testingpkg...
Would be ncie to have some helper scripts. I use some dumb scripts to manage the kde rebuilds: http://git.archlinux.de/build-scripts/tree/kde
Maybe some of those can be extended to be more genereal. Espeically this which sets pkgver, pkgrel and updates the md5sums: http://git.archlinux.de/build- scripts/tree/kde/update-pkgbuilds (the awk scripts is stolen from some post of the forums ;-))
And I just noticed the undocumented -i flag to makechrootpkg, which installs a package without dirtying the chroot - this just became way easier :)
Well.... it doesn't dirty the chroot exactly, but you do end up with everything you built being installed so could get unwanted deps further down the rebuild list unless you are careful. I have been meaning to adjust that to put the package in a local repo in the chroot so rebuilt packages could be installed as necessary (much like in the chroot building wiki page) but have not had time. Allan
On Mon, Feb 23, 2009 at 4:56 PM, Allan McRae <allan@archlinux.org> wrote:
Aaron Griffin wrote:
On Mon, Feb 23, 2009 at 3:36 PM, Pierre Schmitz <pierre@archlinux.de> wrote:
Am Montag, 23. Februar 2009 22:26:36 schrieb Aaron Griffin:
Or something to that effect, that would not only build all packages in a chroot, but auto-bump the pkgrels and dump the correct results to your svn dir, for a later run of testingpkg...
Would be ncie to have some helper scripts. I use some dumb scripts to manage the kde rebuilds: http://git.archlinux.de/build-scripts/tree/kde
Maybe some of those can be extended to be more genereal. Espeically this which sets pkgver, pkgrel and updates the md5sums: http://git.archlinux.de/build- scripts/tree/kde/update-pkgbuilds (the awk scripts is stolen from some post of the forums ;-))
And I just noticed the undocumented -i flag to makechrootpkg, which installs a package without dirtying the chroot - this just became way easier :)
Well.... it doesn't dirty the chroot exactly, but you do end up with everything you built being installed so could get unwanted deps further down the rebuild list unless you are careful. I have been meaning to adjust that to put the package in a local repo in the chroot so rebuilt packages could be installed as necessary (much like in the chroot building wiki page) but have not had time.
Added a new section to the wiki - these changes are in git right now, not in the current devtools release: http://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_Clean_Chroot...
On Mon, Feb 23, 2009 at 5:07 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On Mon, Feb 23, 2009 at 4:56 PM, Allan McRae <allan@archlinux.org> wrote:
Aaron Griffin wrote:
On Mon, Feb 23, 2009 at 3:36 PM, Pierre Schmitz <pierre@archlinux.de> wrote:
Am Montag, 23. Februar 2009 22:26:36 schrieb Aaron Griffin:
Or something to that effect, that would not only build all packages in a chroot, but auto-bump the pkgrels and dump the correct results to your svn dir, for a later run of testingpkg...
Would be ncie to have some helper scripts. I use some dumb scripts to manage the kde rebuilds: http://git.archlinux.de/build-scripts/tree/kde
Maybe some of those can be extended to be more genereal. Espeically this which sets pkgver, pkgrel and updates the md5sums: http://git.archlinux.de/build- scripts/tree/kde/update-pkgbuilds (the awk scripts is stolen from some post of the forums ;-))
And I just noticed the undocumented -i flag to makechrootpkg, which installs a package without dirtying the chroot - this just became way easier :)
Well.... it doesn't dirty the chroot exactly, but you do end up with everything you built being installed so could get unwanted deps further down the rebuild list unless you are careful. I have been meaning to adjust that to put the package in a local repo in the chroot so rebuilt packages could be installed as necessary (much like in the chroot building wiki page) but have not had time.
Added a new section to the wiki - these changes are in git right now, not in the current devtools release: http://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_Clean_Chroot...
The -u and -I flag are in git. -i exists now, but I wanted makepkg -i to be usable, so I renamed that one
Aaron Griffin wrote:
On Mon, Feb 23, 2009 at 5:07 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On Mon, Feb 23, 2009 at 4:56 PM, Allan McRae <allan@archlinux.org> wrote:
Aaron Griffin wrote:
On Mon, Feb 23, 2009 at 3:36 PM, Pierre Schmitz <pierre@archlinux.de> wrote:
Am Montag, 23. Februar 2009 22:26:36 schrieb Aaron Griffin:
Or something to that effect, that would not only build all packages in a chroot, but auto-bump the pkgrels and dump the correct results to your svn dir, for a later run of testingpkg...
Would be ncie to have some helper scripts. I use some dumb scripts to manage the kde rebuilds: http://git.archlinux.de/build-scripts/tree/kde
Maybe some of those can be extended to be more genereal. Espeically this which sets pkgver, pkgrel and updates the md5sums: http://git.archlinux.de/build- scripts/tree/kde/update-pkgbuilds (the awk scripts is stolen from some post of the forums ;-))
And I just noticed the undocumented -i flag to makechrootpkg, which installs a package without dirtying the chroot - this just became way easier :)
Well.... it doesn't dirty the chroot exactly, but you do end up with everything you built being installed so could get unwanted deps further down the rebuild list unless you are careful. I have been meaning to adjust that to put the package in a local repo in the chroot so rebuilt packages could be installed as necessary (much like in the chroot building wiki page) but have not had time.
Added a new section to the wiki - these changes are in git right now, not in the current devtools release: http://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_Clean_Chroot...
The -u and -I flag are in git. -i exists now, but I wanted makepkg -i to be usable, so I renamed that one
It took me a while to figure out why you said the old method dirtied the chroot. IThe way I thought about this is that the packages I rebuilt and updated in the chroot are soon going to be the official Arch ones so it will self clean eventually. Anyway, making the chroot is so easy that I just make a new one to deal with major rebuilds. Allan
On Mon, Feb 23, 2009 at 5:20 PM, Allan McRae <allan@archlinux.org> wrote:
Aaron Griffin wrote:
On Mon, Feb 23, 2009 at 5:07 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On Mon, Feb 23, 2009 at 4:56 PM, Allan McRae <allan@archlinux.org> wrote:
Aaron Griffin wrote:
On Mon, Feb 23, 2009 at 3:36 PM, Pierre Schmitz <pierre@archlinux.de> wrote:
Am Montag, 23. Februar 2009 22:26:36 schrieb Aaron Griffin:
> > Or something to that effect, that would not only build all packages > in > a chroot, but auto-bump the pkgrels and dump the correct results to > your svn dir, for a later run of testingpkg... > >
Would be ncie to have some helper scripts. I use some dumb scripts to manage the kde rebuilds: http://git.archlinux.de/build-scripts/tree/kde
Maybe some of those can be extended to be more genereal. Espeically this which sets pkgver, pkgrel and updates the md5sums: http://git.archlinux.de/build- scripts/tree/kde/update-pkgbuilds (the awk scripts is stolen from some post of the forums ;-))
And I just noticed the undocumented -i flag to makechrootpkg, which installs a package without dirtying the chroot - this just became way easier :)
Well.... it doesn't dirty the chroot exactly, but you do end up with everything you built being installed so could get unwanted deps further down the rebuild list unless you are careful. I have been meaning to adjust that to put the package in a local repo in the chroot so rebuilt packages could be installed as necessary (much like in the chroot building wiki page) but have not had time.
Added a new section to the wiki - these changes are in git right now, not in the current devtools release:
http://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_Clean_Chroot...
The -u and -I flag are in git. -i exists now, but I wanted makepkg -i to be usable, so I renamed that one
It took me a while to figure out why you said the old method dirtied the chroot. IThe way I thought about this is that the packages I rebuilt and updated in the chroot are soon going to be the official Arch ones so it will self clean eventually. Anyway, making the chroot is so easy that I just make a new one to deal with major rebuilds.
Yeah, I was just thinking that too, but there's always the case where things might not work and you may need to patch or roll things back or something. To me it's easier to just "rm -rf <chrootdir>/rw" to clean it up than to rebuild it 8) Additionally, even if they do become the official packages, they'll be in testing for a while and your chroot will be in a weird limbo state where some of the packages are from testing and some are not.
Aaron Griffin wrote:
On Mon, Feb 23, 2009 at 5:20 PM, Allan McRae <allan@archlinux.org> wrote:
Aaron Griffin wrote:
On Mon, Feb 23, 2009 at 5:07 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On Mon, Feb 23, 2009 at 4:56 PM, Allan McRae <allan@archlinux.org> wrote:
Aaron Griffin wrote:
On Mon, Feb 23, 2009 at 3:36 PM, Pierre Schmitz <pierre@archlinux.de> wrote:
> Am Montag, 23. Februar 2009 22:26:36 schrieb Aaron Griffin: > > > >> Or something to that effect, that would not only build all packages >> in >> a chroot, but auto-bump the pkgrels and dump the correct results to >> your svn dir, for a later run of testingpkg... >> >> >> > Would be ncie to have some helper scripts. I use some dumb scripts to > manage > the kde rebuilds: http://git.archlinux.de/build-scripts/tree/kde > > Maybe some of those can be extended to be more genereal. Espeically > this > which > sets pkgver, pkgrel and updates the md5sums: > http://git.archlinux.de/build- > scripts/tree/kde/update-pkgbuilds (the awk scripts is stolen from some > post of > the forums ;-)) > > > And I just noticed the undocumented -i flag to makechrootpkg, which installs a package without dirtying the chroot - this just became way easier :)
Well.... it doesn't dirty the chroot exactly, but you do end up with everything you built being installed so could get unwanted deps further down the rebuild list unless you are careful. I have been meaning to adjust that to put the package in a local repo in the chroot so rebuilt packages could be installed as necessary (much like in the chroot building wiki page) but have not had time.
Added a new section to the wiki - these changes are in git right now, not in the current devtools release:
http://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_Clean_Chroot...
The -u and -I flag are in git. -i exists now, but I wanted makepkg -i to be usable, so I renamed that one
It took me a while to figure out why you said the old method dirtied the chroot. IThe way I thought about this is that the packages I rebuilt and updated in the chroot are soon going to be the official Arch ones so it will self clean eventually. Anyway, making the chroot is so easy that I just make a new one to deal with major rebuilds.
Yeah, I was just thinking that too, but there's always the case where things might not work and you may need to patch or roll things back or something. To me it's easier to just "rm -rf <chrootdir>/rw" to clean it up than to rebuild it 8)
OK, I see your point. Of course it only applies for packages in base/base-devel. I guess I didn't see ths because I have never need to rollback because my packaging skills are awesome and I never break anything.... :)
participants (3)
-
Aaron Griffin
-
Allan McRae
-
Pierre Schmitz