On 03/02/2012 04:48 PM, C Anthony Risinger wrote:

IIRC makechrootpkg used to create an overlay/union before starting, which would keep your original root intact. if you put your builds on a btrfs filesystem i believe makechrootpkg will snapshot the fs before starting (i know im mixing mkarchroot and makechrootpkg a bit here ...)

imo, the most effective way is to use an overlay. this FUSE based one work just fine:

http://podgorny.cz/unionfs-fuse/releases/

... as do many other methods (DM, btrfs, overlayfs, unionfs, etc etc), though i persoanlly like the FUSE impl because it can cope with stuff better without less risk of crashing. you can then use --bind mounts or multiple overlays to do interesting things too.

make sure you've enabled `ccache` in your /etc/makepkg.conf, and adjusted your `-jN` makeflag to an appropriate number. i keep a 10GiB cache around; i can blow away the webkit tree and start completely fresh, apply my in-progress patchsets, and have a working package in ~2 minutes (normally a 3+hr build).

Thanks C Anthony, It's the alphabet soup on top of the union filesystem and separate rw layer concepts that has my eyes rolling back in my head at the moment :) (re-reading the paragraphs for a 2nd time hoping it will all sink in) OK, so what your are saying is that after I make friends with the snapshot concept (whether DM, btrfs, overlayfs, etc...) I can basically keep snapshots of different build 'points' in cache and switch back and forth relatively easily once I learn how -- right? So if I have one set of Trinity packages based on Qt3 and another set based on TQt3, I can pickup building at the last good point of progress in either set by configuring the snapshot I need to create that environment? Like if I have built the dependencies through the snapshots below: Trinity/Qt3 Trinity/TQt3 Arts Arts Dbus Dbus caldav caldav ... ... tdelibs (cache snapshot A) tdelibs tdebase (cache snapshot B) I could then configure the chroot to switch back to 'cache snapshot A' and continue building on Trinity/Qt3 at that point after I had saved snapshot B and reverted to the A cache? Would you happen to have a favorite link to a page that might help me get a bit smarter on this? :) -- David C. Rankin, J.D.,P.E.

On Wed, Mar 7, 2012 at 12:27 PM, C Anthony Risinger <anthony@xtfx.me> wrote:

... i thought you were asking a) "share a base read-only chroot", so you don't have to rebuild/clean it.  since the base is readonly, you must then --bind mount (shared/persistent) or create an empty repository (within the overlay only) at /repo (i believe there is some built-in support for this).  then you instruct makechrootpkg to (or manually) install the recently built package to that repository, and continue with next build.  when you want to start the next full rebuild (TQt3), you simply remove the unwanted packages from the chroot (eg, pacman -r {path-to-chroot}).

^^^^^ i meant to describe the overlay stuff too ... was rushing out the door to an appt ... the above should work with or without an overlay.  the benefit of the overlay/union is you can easy change layers, and isolate each build. for example, the unionfs FUSE module i linked simply diverts your writeable layer to a different directory, allowing you to stuff like this: ----------------------------------------------- APPLICATION (makepkg) --> VFS [mkdir, unlink, *syscall] --> /rw-build-XYZ (build layer) --> /ro-tqt3 (core layer) --> /ro-base (base layer) ----------------------------------------------- ... so, you'd start with 3 empty dirs, then: ) setup a base chroot in /ro-base (including ccache + configs) ) mount unionfs with [RO] /ro-base and [RW] /ro-tqt3 ) install core packages needed by TQt3 ) ... all changes, and only the changes, will now be in /ro-tqt3 ) remount: [RO] /ro-base, [RO] /ro-tqt3, [RW] /rw-build-NAME ) perform build (ensure .ccache dir is available inside chroot) ) install to /repo ) [repeat the last 3 steps for each package] ... may need adjustments; pacman doesn't like any of this. i've never done 3 layers like i briefed above -- you could put each pkg in a separate layer if you *really* wanted -- but i frequently use 2, the ro-base and rw-build-NAME. seriously, ccache makes stuff so fast that it's probably slower to putz around vs. just using a fresh rw-build-* each time you need to. although, it might be kind of cool to extract all of your source tarballs to separate dirs, say: ----------------------------------------------- /path/.../layers/out/@trinity-20120307/{source} [...] /path/.../layers/in-out/@repo-20120307/repo [...] /path/.../layers/in/@base/{mkarchroot} [...] /path/.../layers/in/arts/PKGBUILD /path/.../layers/in/dbus/PKGBUILD /path/.../layers/in/caldav/PKGBUILD /path/.../layers/in/tdelibs/PKGBUILD ----------------------------------------------- ... then mount it up so it looks something like this (one command, see unionfs options): ----------------------------------------------- APPLICATION (makepkg) --> [RW] @trinity-20120307/ --> [RW] @repo-20120307/ --> [RO] @base/ --> [RO] arts/ --> [RO] dbus/ --> [RO] caldav/ --> [RO] tdelibs/ ----------------------------------------------- ... hell, you could fancy and use individual git repos + commit when you update a src ... throw in a few scripts and you might have a pretty cool build system capable of building any available combination of pkg X, Y, Z, *and* rolling back in time to build against older versions with a simple `git checkout` ... yeah, i'm going to play with this. o_O are you sufficiently confused yet? because i can keep going if you like :-) but i'm getting a bit lost myself. the point of the exercise is that unionfs and other layering solutions permit incredible build re-use and flexibility. the biggest challenge is making pacman to deal with the craziness ... we need a proxy pacman db (FUSE) that will lie to pacman when it tries to read it. tldr; try a 2-layer setup and see how it works for you. -- C Anthony