[pacman-dev] makepkg in a constrained environment

[Ciprian Dorin, Craciun]

On Wed, Dec 2, 2009 at 3:43 PM, Dan McGee <dpmcgee at gmail.com> wrote:
> On Wed, Dec 2, 2009 at 5:01 AM, Ciprian Dorin, Craciun
> <ciprian.craciun at gmail.com> wrote:
>>    Hello all!
>>
>>    For some time I was struggling in creating (for myself) a
>> "sterile" environment for package building purposes. (I want to build
>> some custom packages for a custom lightweight distribution.) And in
>> what follows I would like to share my experience with pacman / makepkg
>> developers in such a "sterile" environment.
>>
>>    By "sterile" I mean the following:
>>    * clean, uncontaminated packages: freshly installed by pacman from
>> a repository without my intervention; (if I would use my ArchLinux
>> installation, some packages (compiler, libraries, etc.) would be
>> configured to my like, and thus it could create a non-repeatable
>> build);
>>    * minimal package installation base: only the packages that have
>> been clearly declared as dependencies in PKGBUILD should be installed
>> (of course with their dependencies);
>>    * I need the assurance that configure and make scripts of
>> different packages pick up the right build tool-chains (for example
>> for cross-compiling);
>>
>>    As such, I've decided of the following:
>>    * prepare a UML (User Mode Linux) kernel, that shall provide me
>> with a clean build Linux environment; (this could of course be
>> replaced with a Vserver instance;)
>>    * prepare a (static linked) BusyBox with all the tools built in;
>> (this shall be installed outside of the normal PATH, to be sure it's
>> not picked up by the configure and make scripts;)
>>    * prepare a (static linked) Bash (installed outside the normal PATH);
>>    * when running makepkg, I'll set the
>> PATH=/.workbench/tools/busybox/bin:/.workbench/tools/bash/bin;
>>    * but when inside the PKGBUILD I'll reset the
>> PATH=/usr/local/bin:/usr/bin:/bin:etc....
>>
>>    Now my experiences with makepkg in such an environment:
>>    * it seems it works without too much fuss with the commands
>> provided by busybox, except;
>>    * if one has declared of using only MD5 sums (most of the packages
>> do this), it insists on using `openssl dgst` to create them;
> See commit b8a66d68593d1f267c3bb8cd8943724711626903 as Allan pointed
> out, which was to fix FS#10530. You'd be reverting that which is not
> really what we want.
>
>>    * it insists on using file tool to identify the file type;
> Allan got this one. (FS#6246)
>
>>    * it insists on using bsdtar to untar the tar archives;
> bsdtar is a hard dep of pacman and makes our work in makepkg a ton
> easier. It works for all sorts of archives and I would have a real
> hard time seeing why we should step away from it.

    I understand that bsdtar knows a lot of archive types, and that it
supports POSIX file capability. But if I would use makepkg on an
operating system where I don't have bsdtar, I would be unable to
construct any package, even-though I'm using only tar's or zip's.

    Please understand that I'm not proposing in dropping bsdtar and
replacing it with plain tar or zip, but I'm proposing falling-back to
plain tar or zip where bsdtar is not available.


>>    * it breaks if it's not able to chown the source tree; (this
>> happens as inside UML I'm root, but outside UML the file system is
>> owned by a normal user (the root filesystem of UML is exported by
>> using hostfs));
> Sounds like UML needs something like fakeroot?

    Yes, that's the problem. I have three solutions:
    * use UML as root (but this leads to security issues);
    * run UML as a child of fakeroot (doesn't work as fakeroot does
not support statically linked ELF's), or with fakeroot-ng, which
breaks UML, because UML itself uses ptrace syscall;
    * run fakeroot(-ng) inside UML; (still on my to-do list);


>>    So, as you can see nothing that can't be solved with some minor corrections.
>>
>>    As such, if the community is interested I could provide some
>> patches that allow makepkg to:
>>    * use md5sum instead of openssl for MD5 checksums;
>>    * use normal tar if bsdtar is not available;
>>    * base the file-type decision on file (if available), or on
>> extension (if file is not available);
>>    * ignore errors for chown of source tree; (of course with a warning;)
>>    * provide a way to customize the path of the pacman tool; (maybe
>> this has already been done);
> Getting close here, see
> http://projects.archlinux.org/pacman.git/commit/?id=82443e00597b0db5a7eadea4886e22b373ac373b

    I'll have a look at that.


>>    Also it would be nice (though painful) to allow the customization
>> of paths for all external (non Bash functions) tools. For example:
>> instead of calling mv, one should call "$mv", which should default to
>> mv="${MV:-$( which mv )}" (that is if the variable MV is not defined
>> then fallback on using the mv tool found in the PATH).
>
> A unix-y system without mv? This seems overkill to me. I highly doubt
> all of our packages have a makedepend on coreutils anyway.

    No, with mv, but with mv somewhare outside the normal PATH. (I use
this trick to be sure that the package has all the dependencies
properly listed.)
    And indeed if a package does not list coreutils as it's makepep,
it's a package problem, so I would be able to catch such problems.


> I can see why a lot of these things seem appealing for your setup, but
> the problem is a lot of them are in contrast to portability issues
> we've had in the past and resolved.

    On the contrary, I would see these enhancements as a way to make
makepkg more portable. (I agree that not having /bin/mv would seem
questionable, but not having openssl, md5sum, bsdtar, file, etc., or
any other non /bin tool seems quite plausible.)


> -Dan

    Thank you for your comments,
    Ciprian.