[pacman-dev] makepkg in a constrained environment

Ciprian Dorin, Craciun

2 Dec 2009 2 Dec '09

12:01 p.m.

Hello all! For some time I was struggling in creating (for myself) a "sterile" environment for package building purposes. (I want to build some custom packages for a custom lightweight distribution.) And in what follows I would like to share my experience with pacman / makepkg developers in such a "sterile" environment. By "sterile" I mean the following: * clean, uncontaminated packages: freshly installed by pacman from a repository without my intervention; (if I would use my ArchLinux installation, some packages (compiler, libraries, etc.) would be configured to my like, and thus it could create a non-repeatable build); * minimal package installation base: only the packages that have been clearly declared as dependencies in PKGBUILD should be installed (of course with their dependencies); * I need the assurance that configure and make scripts of different packages pick up the right build tool-chains (for example for cross-compiling); As such, I've decided of the following: * prepare a UML (User Mode Linux) kernel, that shall provide me with a clean build Linux environment; (this could of course be replaced with a Vserver instance;) * prepare a (static linked) BusyBox with all the tools built in; (this shall be installed outside of the normal PATH, to be sure it's not picked up by the configure and make scripts;) * prepare a (static linked) Bash (installed outside the normal PATH); * when running makepkg, I'll set the PATH=/.workbench/tools/busybox/bin:/.workbench/tools/bash/bin; * but when inside the PKGBUILD I'll reset the PATH=/usr/local/bin:/usr/bin:/bin:etc.... Now my experiences with makepkg in such an environment: * it seems it works without too much fuss with the commands provided by busybox, except; * if one has declared of using only MD5 sums (most of the packages do this), it insists on using `openssl dgst` to create them; * it insists on using file tool to identify the file type; * it insists on using bsdtar to untar the tar archives; * it breaks if it's not able to chown the source tree; (this happens as inside UML I'm root, but outside UML the file system is owned by a normal user (the root filesystem of UML is exported by using hostfs)); So, as you can see nothing that can't be solved with some minor corrections. As such, if the community is interested I could provide some patches that allow makepkg to: * use md5sum instead of openssl for MD5 checksums; * use normal tar if bsdtar is not available; * base the file-type decision on file (if available), or on extension (if file is not available); * ignore errors for chown of source tree; (of course with a warning;) * provide a way to customize the path of the pacman tool; (maybe this has already been done); Also it would be nice (though painful) to allow the customization of paths for all external (non Bash functions) tools. For example: instead of calling mv, one should call "$mv", which should default to mv="${MV:-$( which mv )}" (that is if the variable MV is not defined then fallback on using the mv tool found in the PATH). So would the developers be interested in such patches? Any comments? Any ideas? Thanks, Ciprian.

Show replies by thread

Allan McRae

2 Dec 2 Dec

12:18 p.m.

Ciprian Dorin, Craciun wrote: <snip>

...

* if one has declared of using only MD5 sums (most of the packages do this), it insists on using `openssl dgst` to create them;

http://projects.archlinux.org/pacman.git/commit/?id=b8a66d68

...

* it insists on using file tool to identify the file type;

http://bugs.archlinux.org/task/6246

...

* it insists on using bsdtar to untar the tar archives;

You need libarchive (bsdtar) for pacman and need pacman to resolve deps...

...

* it breaks if it's not able to chown the source tree; (this happens as inside UML I'm root, but outside UML the file system is owned by a normal user (the root filesystem of UML is exported by using hostfs));

There is this in makepkg: if (( EUID == 0 )); then # change perms of all source files to root user & root group chown -R 0:0 "$srcdir" fi It would seem the assumption that root has root permissions is not always true... We could probably also add a check the user has permissions to change the file permissions too. Allan

Dan McGee

2:43 p.m.

On Wed, Dec 2, 2009 at 5:01 AM, Ciprian Dorin, Craciun wrote:

...

Hello all!

For some time I was struggling in creating (for myself) a "sterile" environment for package building purposes. (I want to build some custom packages for a custom lightweight distribution.) And in what follows I would like to share my experience with pacman / makepkg developers in such a "sterile" environment.

By "sterile" I mean the following: * clean, uncontaminated packages: freshly installed by pacman from a repository without my intervention; (if I would use my ArchLinux installation, some packages (compiler, libraries, etc.) would be configured to my like, and thus it could create a non-repeatable build); * minimal package installation base: only the packages that have been clearly declared as dependencies in PKGBUILD should be installed (of course with their dependencies); * I need the assurance that configure and make scripts of different packages pick up the right build tool-chains (for example for cross-compiling);

As such, I've decided of the following: * prepare a UML (User Mode Linux) kernel, that shall provide me with a clean build Linux environment; (this could of course be replaced with a Vserver instance;) * prepare a (static linked) BusyBox with all the tools built in; (this shall be installed outside of the normal PATH, to be sure it's not picked up by the configure and make scripts;) * prepare a (static linked) Bash (installed outside the normal PATH); * when running makepkg, I'll set the PATH=/.workbench/tools/busybox/bin:/.workbench/tools/bash/bin; * but when inside the PKGBUILD I'll reset the PATH=/usr/local/bin:/usr/bin:/bin:etc....

Now my experiences with makepkg in such an environment: * it seems it works without too much fuss with the commands provided by busybox, except; * if one has declared of using only MD5 sums (most of the packages do this), it insists on using `openssl dgst` to create them; See commit b8a66d68593d1f267c3bb8cd8943724711626903 as Allan pointed out, which was to fix FS#10530. You'd be reverting that which is not really what we want.

...

* it insists on using file tool to identify the file type; Allan got this one. (FS#6246)

...

* it insists on using bsdtar to untar the tar archives; bsdtar is a hard dep of pacman and makes our work in makepkg a ton easier. It works for all sorts of archives and I would have a real hard time seeing why we should step away from it.

...

* it breaks if it's not able to chown the source tree; (this happens as inside UML I'm root, but outside UML the file system is owned by a normal user (the root filesystem of UML is exported by using hostfs)); Sounds like UML needs something like fakeroot?

...

So, as you can see nothing that can't be solved with some minor corrections.

As such, if the community is interested I could provide some patches that allow makepkg to: * use md5sum instead of openssl for MD5 checksums; * use normal tar if bsdtar is not available; * base the file-type decision on file (if available), or on extension (if file is not available); * ignore errors for chown of source tree; (of course with a warning;) * provide a way to customize the path of the pacman tool; (maybe this has already been done); Getting close here, see http://projects.archlinux.org/pacman.git/commit/?id=82443e00597b0db5a7eadea4...

...

Also it would be nice (though painful) to allow the customization of paths for all external (non Bash functions) tools. For example: instead of calling mv, one should call "$mv", which should default to mv="${MV:-$( which mv )}" (that is if the variable MV is not defined then fallback on using the mv tool found in the PATH).

A unix-y system without mv? This seems overkill to me. I highly doubt all of our packages have a makedepend on coreutils anyway. I can see why a lot of these things seem appealing for your setup, but the problem is a lot of them are in contrast to portability issues we've had in the past and resolved. -Dan

Ciprian Dorin, Craciun

3:05 p.m.

On Wed, Dec 2, 2009 at 3:43 PM, Dan McGee wrote:

...

On Wed, Dec 2, 2009 at 5:01 AM, Ciprian Dorin, Craciun wrote:

...
Hello all!

For some time I was struggling in creating (for myself) a "sterile" environment for package building purposes. (I want to build some custom packages for a custom lightweight distribution.) And in what follows I would like to share my experience with pacman / makepkg developers in such a "sterile" environment.

By "sterile" I mean the following: * clean, uncontaminated packages: freshly installed by pacman from a repository without my intervention; (if I would use my ArchLinux installation, some packages (compiler, libraries, etc.) would be configured to my like, and thus it could create a non-repeatable build); * minimal package installation base: only the packages that have been clearly declared as dependencies in PKGBUILD should be installed (of course with their dependencies); * I need the assurance that configure and make scripts of different packages pick up the right build tool-chains (for example for cross-compiling);

As such, I've decided of the following: * prepare a UML (User Mode Linux) kernel, that shall provide me with a clean build Linux environment; (this could of course be replaced with a Vserver instance;) * prepare a (static linked) BusyBox with all the tools built in; (this shall be installed outside of the normal PATH, to be sure it's not picked up by the configure and make scripts;) * prepare a (static linked) Bash (installed outside the normal PATH); * when running makepkg, I'll set the PATH=/.workbench/tools/busybox/bin:/.workbench/tools/bash/bin; * but when inside the PKGBUILD I'll reset the PATH=/usr/local/bin:/usr/bin:/bin:etc....

Now my experiences with makepkg in such an environment: * it seems it works without too much fuss with the commands provided by busybox, except; * if one has declared of using only MD5 sums (most of the packages do this), it insists on using `openssl dgst` to create them; See commit b8a66d68593d1f267c3bb8cd8943724711626903 as Allan pointed out, which was to fix FS#10530. You'd be reverting that which is not really what we want.

...
* it insists on using file tool to identify the file type; Allan got this one. (FS#6246)

...
* it insists on using bsdtar to untar the tar archives; bsdtar is a hard dep of pacman and makes our work in makepkg a ton easier. It works for all sorts of archives and I would have a real hard time seeing why we should step away from it.

I understand that bsdtar knows a lot of archive types, and that it supports POSIX file capability. But if I would use makepkg on an operating system where I don't have bsdtar, I would be unable to construct any package, even-though I'm using only tar's or zip's. Please understand that I'm not proposing in dropping bsdtar and replacing it with plain tar or zip, but I'm proposing falling-back to plain tar or zip where bsdtar is not available.

...

...
* it breaks if it's not able to chown the source tree; (this happens as inside UML I'm root, but outside UML the file system is owned by a normal user (the root filesystem of UML is exported by using hostfs)); Sounds like UML needs something like fakeroot?

Yes, that's the problem. I have three solutions: * use UML as root (but this leads to security issues); * run UML as a child of fakeroot (doesn't work as fakeroot does not support statically linked ELF's), or with fakeroot-ng, which breaks UML, because UML itself uses ptrace syscall; * run fakeroot(-ng) inside UML; (still on my to-do list);

...

...
So, as you can see nothing that can't be solved with some minor corrections.

As such, if the community is interested I could provide some patches that allow makepkg to: * use md5sum instead of openssl for MD5 checksums; * use normal tar if bsdtar is not available; * base the file-type decision on file (if available), or on extension (if file is not available); * ignore errors for chown of source tree; (of course with a warning;) * provide a way to customize the path of the pacman tool; (maybe this has already been done); Getting close here, see http://projects.archlinux.org/pacman.git/commit/?id=82443e00597b0db5a7eadea4...

I'll have a look at that.

...

...
Also it would be nice (though painful) to allow the customization of paths for all external (non Bash functions) tools. For example: instead of calling mv, one should call "$mv", which should default to mv="${MV:-$( which mv )}" (that is if the variable MV is not defined then fallback on using the mv tool found in the PATH).

A unix-y system without mv? This seems overkill to me. I highly doubt all of our packages have a makedepend on coreutils anyway.

No, with mv, but with mv somewhare outside the normal PATH. (I use this trick to be sure that the package has all the dependencies properly listed.) And indeed if a package does not list coreutils as it's makepep, it's a package problem, so I would be able to catch such problems.

...

I can see why a lot of these things seem appealing for your setup, but the problem is a lot of them are in contrast to portability issues we've had in the past and resolved.

On the contrary, I would see these enhancements as a way to make makepkg more portable. (I agree that not having /bin/mv would seem questionable, but not having openssl, md5sum, bsdtar, file, etc., or any other non /bin tool seems quite plausible.)

...

-Dan

Thank you for your comments, Ciprian.

Cedric Staniewski

3:58 p.m.

On 12/02/2009 03:05 PM, Ciprian Dorin, Craciun wrote:

...

On Wed, Dec 2, 2009 at 3:43 PM, Dan McGee wrote:

...
On Wed, Dec 2, 2009 at 5:01 AM, Ciprian Dorin, Craciun wrote:

...
Hello all!

For some time I was struggling in creating (for myself) a "sterile" environment for package building purposes. (I want to build some custom packages for a custom lightweight distribution.) And in what follows I would like to share my experience with pacman / makepkg developers in such a "sterile" environment.

By "sterile" I mean the following: * clean, uncontaminated packages: freshly installed by pacman from a repository without my intervention; (if I would use my ArchLinux installation, some packages (compiler, libraries, etc.) would be configured to my like, and thus it could create a non-repeatable build); * minimal package installation base: only the packages that have been clearly declared as dependencies in PKGBUILD should be installed (of course with their dependencies); * I need the assurance that configure and make scripts of different packages pick up the right build tool-chains (for example for cross-compiling);

As such, I've decided of the following: * prepare a UML (User Mode Linux) kernel, that shall provide me with a clean build Linux environment; (this could of course be replaced with a Vserver instance;) * prepare a (static linked) BusyBox with all the tools built in; (this shall be installed outside of the normal PATH, to be sure it's not picked up by the configure and make scripts;) * prepare a (static linked) Bash (installed outside the normal PATH); * when running makepkg, I'll set the PATH=/.workbench/tools/busybox/bin:/.workbench/tools/bash/bin; * but when inside the PKGBUILD I'll reset the PATH=/usr/local/bin:/usr/bin:/bin:etc....

Now my experiences with makepkg in such an environment: * it seems it works without too much fuss with the commands provided by busybox, except; * if one has declared of using only MD5 sums (most of the packages do this), it insists on using `openssl dgst` to create them; See commit b8a66d68593d1f267c3bb8cd8943724711626903 as Allan pointed out, which was to fix FS#10530. You'd be reverting that which is not really what we want.

...
* it insists on using file tool to identify the file type; Allan got this one. (FS#6246)

...
* it insists on using bsdtar to untar the tar archives; bsdtar is a hard dep of pacman and makes our work in makepkg a ton easier. It works for all sorts of archives and I would have a real hard time seeing why we should step away from it.

I understand that bsdtar knows a lot of archive types, and that it supports POSIX file capability. But if I would use makepkg on an operating system where I don't have bsdtar, I would be unable to construct any package, even-though I'm using only tar's or zip's.

Please understand that I'm not proposing in dropping bsdtar and replacing it with plain tar or zip, but I'm proposing falling-back to plain tar or zip where bsdtar is not available.

...
...
* it breaks if it's not able to chown the source tree; (this happens as inside UML I'm root, but outside UML the file system is owned by a normal user (the root filesystem of UML is exported by using hostfs)); Sounds like UML needs something like fakeroot?

Yes, that's the problem. I have three solutions: * use UML as root (but this leads to security issues); * run UML as a child of fakeroot (doesn't work as fakeroot does not support statically linked ELF's), or with fakeroot-ng, which breaks UML, because UML itself uses ptrace syscall; * run fakeroot(-ng) inside UML; (still on my to-do list);

...
...
So, as you can see nothing that can't be solved with some minor corrections.

As such, if the community is interested I could provide some patches that allow makepkg to: * use md5sum instead of openssl for MD5 checksums; * use normal tar if bsdtar is not available; * base the file-type decision on file (if available), or on extension (if file is not available); * ignore errors for chown of source tree; (of course with a warning;) * provide a way to customize the path of the pacman tool; (maybe this has already been done); Getting close here, see http://projects.archlinux.org/pacman.git/commit/?id=82443e00597b0db5a7eadea4...

I'll have a look at that.

...
...
Also it would be nice (though painful) to allow the customization of paths for all external (non Bash functions) tools. For example: instead of calling mv, one should call "$mv", which should default to mv="${MV:-$( which mv )}" (that is if the variable MV is not defined then fallback on using the mv tool found in the PATH).

A unix-y system without mv? This seems overkill to me. I highly doubt all of our packages have a makedepend on coreutils anyway.

No, with mv, but with mv somewhare outside the normal PATH. (I use this trick to be sure that the package has all the dependencies properly listed.) And indeed if a package does not list coreutils as it's makepep, it's a package problem, so I would be able to catch such problems.

...
I can see why a lot of these things seem appealing for your setup, but the problem is a lot of them are in contrast to portability issues we've had in the past and resolved.

On the contrary, I would see these enhancements as a way to make makepkg more portable. (I agree that not having /bin/mv would seem questionable, but not having openssl, md5sum, bsdtar, file, etc., or any other non /bin tool seems quite plausible.)

...
-Dan

Thank you for your comments, Ciprian.

I do not agree with your definition of "portable" here. All of makepkgs dependencies are available for (mostly?) every system pacman can run on, so it is already "portable". Adding fallbacks only may make it easier to install/run makepkg in a specific setup. A setup where you cannot install all of makepkg's dependencies with a reasonable effort seems rather exotic to me and does not justify such an inappropriate increase in complexity of makepkg in my opinion. Why do you not write bash wrappers which provide the functionality of bsdtar/openssl/... and add these to your path?

Ciprian Dorin, Craciun

9:47 p.m.

On Wed, Dec 2, 2009 at 3:43 PM, Dan McGee wrote:

...

On Wed, Dec 2, 2009 at 5:01 AM, Ciprian Dorin, Craciun wrote:

...
[... snip ...] * if one has declared of using only MD5 sums (most of the packages do this), it insists on using `openssl dgst` to create them; See commit b8a66d68593d1f267c3bb8cd8943724711626903 as Allan pointed out, which was to fix FS#10530. You'd be reverting that which is not really what we want.

Looking closer at what the mentioned patch introduced: ~~~~ - local sum="$(${integ}sum "$file" | cut -d ' ' -f 1)" + local sum="$(openssl dgst -${integ} "$file" | awk '{print $NF}')" ~~~~ I'm proposing not to get back to what it has been (which assumed that there was a md5sum sha128sum, <something-else>sum), but I'm proposing that for md5 we should use md5sum, which I'm sure it exists on any UNIX. Or better even, try to see if openssl exists, if not falling back to md5sum, at least for md5sums... This doesn't break anything, and it allows makepkg to work without openssl.

...

...
* it insists on using file tool to identify the file type; Allan got this one. (FS#6246)

I understand that `file` tool was introduced to enhance makepkg, and I'm not proposing to drop it, but instead if `file` does not exist, to fall-back to file-extension handling.

...

...
[... snip ...] -Dan

Please see that I'm saying everywhere: If <something> tool does not exist, switch back, that is fallback, to some more primitive way of solving things. Ciprian.

Xavier

10:16 p.m.

On Wed, Dec 2, 2009 at 3:58 PM, Cedric Staniewski wrote:

...

I do not agree with your definition of "portable" here. All of makepkgs dependencies are available for (mostly?) every system pacman can run on, so it is already "portable". Adding fallbacks only may make it easier to install/run makepkg in a specific setup. A setup where you cannot install all of makepkg's dependencies with a reasonable effort seems rather exotic to me and does not justify such an inappropriate increase in complexity of makepkg in my opinion. Why do you not write bash wrappers which provide the functionality of bsdtar/openssl/... and add these to your path?

I often agree with Cedric.. this is just another occurence :)

Ciprian Dorin, Craciun

10:30 p.m.

On Wed, Dec 2, 2009 at 11:16 PM, Xavier wrote:

...

On Wed, Dec 2, 2009 at 3:58 PM, Cedric Staniewski wrote:

...
I do not agree with your definition of "portable" here. All of makepkgs dependencies are available for (mostly?) every system pacman can run on, so it is already "portable". Adding fallbacks only may make it easier to install/run makepkg in a specific setup. A setup where you cannot install all of makepkg's dependencies with a reasonable effort seems rather exotic to me and does not justify such an inappropriate increase in complexity of makepkg in my opinion. Why do you not write bash wrappers which provide the functionality of bsdtar/openssl/... and add these to your path?

I often agree with Cedric.. this is just another occurence :)

:) Ok. Got it: makepkg should make all assumptions possible that it's working on a fat Linux... For the moment, I think that my best option is to write my own makepkg. Of course I'll write it in any other programming language than Bash... :) For example I think I'll try rc (Plan9 shell), because it seems the most sane shell I've ever seen... Thanks, Ciprian.

Dan McGee

3 Dec 3 Dec

1:41 a.m.

On Wed, Dec 2, 2009 at 2:47 PM, Ciprian Dorin, Craciun wrote:

...

Looking closer at what the mentioned patch introduced: ~~~~ - local sum="$(${integ}sum "$file" | cut -d ' ' -f 1)" + local sum="$(openssl dgst -${integ} "$file" | awk '{print $NF}')" ~~~~ I'm proposing not to get back to what it has been (which assumed that there was a md5sum sha128sum, <something-else>sum), but I'm proposing that for md5 we should use md5sum, which I'm sure it exists on any UNIX. Or better even, try to see if openssl exists, if not falling back to md5sum, at least for md5sums...

As a side note, it doesn't exist on all UNIX-y systems, for example on Mac OS X- there is a /sbin/md5, but not an md5sum command. -Dan

Sebastian Nowicki

11 Dec 11 Dec

5:38 p.m.

On 03/12/2009, at 8:41 AM, Dan McGee wrote:

...

On Wed, Dec 2, 2009 at 2:47 PM, Ciprian Dorin, Craciun wrote:

...
Looking closer at what the mentioned patch introduced: ~~~~ - local sum="$(${integ}sum "$file" | cut -d ' ' -f 1)" + local sum="$(openssl dgst -${integ} "$file" | awk '{print $NF}')" ~~~~ I'm proposing not to get back to what it has been (which assumed that there was a md5sum sha128sum, <something-else>sum), but I'm proposing that for md5 we should use md5sum, which I'm sure it exists on any UNIX. Or better even, try to see if openssl exists, if not falling back to md5sum, at least for md5sums...

As a side note, it doesn't exist on all UNIX-y systems, for example on Mac OS X- there is a /sbin/md5, but not an md5sum command.

-Dan

From what I recall, the `md5` utility on Mac OS X uses openssl (libcrypto) anyway. It does occur to me that hashing should be re- factored into a separate function, as it does occur at least twice. On 03/12/2009, at 5:30 AM, Ciprian Dorin, Craciun wrote:

...

For the moment, I think that my best option is to write my own makepkg.

Of course I'll write it in any other programming language than Bash... :) For example I think I'll try rc (Plan9 shell), because it seems the most sane shell I've ever seen...

That's a little drastic. Why not just make the changes you wanted to makepkg? You could also send the patches to the ML. They might not get accepted, but if you're willing to go as rewriting makepkg from scratch, what do you care? Bash isn't the greatest language to write a large (for bash) utility like makepkg, but I doubt RC will be much better. Why can't you just install bsdtar and openssl? Sure they're separate packages, but ones that makepkg depends on.

5275

Age (days ago)

5284

Last active (days ago)

List overview

Download

9 comments

6 participants

participants (6)

Allan McRae
Cedric Staniewski
Ciprian Dorin, Craciun
Dan McGee
Sebastian Nowicki
Xavier