[arch-projects] [devtools] [PATCH 1/2] arch-nspawn, mkarchroot: Allow not sharing the cache directories.

newer
[arch-projects] [pyalpm] [PATCH]...

older
[arch-projects] [dbscripts] [GIT]...

Maarten de Vries

15 Jan 2019 15 Jan '19

2:34 p.m.

These patches make it possible to build withouth a shared pacman cache using makechrootpkg. I need this myself because I'm building packages for different repositories where some of them contain packages with the same name, but different compile time configuration. It also allows building packages for different Arch based distros on the same host more easily. The first patch changes arch-nspawn and mkarchroot to accept `-c -` to disable bind mounting the host pacman cache in the container. The container will simply use whatever is configured in the container instead. The second patch allows the user set the pacman cache for makechrootpkg. Sadly -c was already taken, so here it's -C. The end result is that a user can run the following to build without a shared pacman cache: `makechrootpkg -C - -r $chroot` I considered a more generic -N ... to pass arguments to arch-nspawn, but arch-nspawn doesn't have that many options and it would get quite ugly: makechrootpkg -N -c -N - -r $chroot Signed-off-by: Maarten de Vries --- arch-nspawn.in | 12 +++++++++--- mkarchroot.in | 4 +++- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/arch-nspawn.in b/arch-nspawn.in index b9c846e..c6cc3a4 100644 --- a/arch-nspawn.in +++ b/arch-nspawn.in @@ -52,7 +52,9 @@ shift 1 [[ -z $working_dir ]] && die 'Please specify a working directory.' -if [[ -z $cache_dir ]]; then +if [[ $cache_dir = '-' ]]; then + cache_dirs=() +elif [[ -z $cache_dir ]]; then cache_dirs=($(pacman -v 2>&1 | grep '^Cache Dirs:' | sed 's/Cache Dirs:\s*//g')) else cache_dirs=("$cache_dir") @@ -72,7 +74,9 @@ build_mount_args() { mount_args+=("--bind-ro=$host_mirror_path") fi - mount_args+=("--bind=${cache_dirs[0]}") + if [[ ${#cache_dirs[@]} -ge 1 ]]; then + mount_args+=("--bind=${cache_dirs[0]}") + fi for cache_dir in "${cache_dirs[@]:1}"; do mount_args+=("--bind-ro=$cache_dir") @@ -92,7 +96,9 @@ copy_hostconf () { cp -T "$file" "$working_dir$file" done - sed -r "s|^#?\\s*CacheDir.+|CacheDir = ${cache_dirs[*]}|g" -i "$working_dir/etc/pacman.conf" + if [[ ${#cache_dirs[@]} -ge 1 ]]; then + sed -r "s|^#?\\s*CacheDir.+|CacheDir = ${cache_dirs[*]}|g" -i "$working_dir/etc/pacman.conf" + fi } # }}} diff --git a/mkarchroot.in b/mkarchroot.in index 52e363f..81bc58a 100644 --- a/mkarchroot.in +++ b/mkarchroot.in @@ -51,7 +51,9 @@ shift 1 [[ -z $working_dir ]] && die 'Please specify a working directory.' -if [[ -z $cache_dir ]]; then +if [[ $cache_dir = '-' ]]; then + cache_dirs=() +elif [[ -z $cache_dir ]]; then cache_dirs=($(pacman -v "$cache_conf" 2>&1 | grep '^Cache Dirs:' | sed 's/Cache Dirs:\s*//g')) else cache_dirs=(${cache_dir}) -- 2.20.1

Show replies by date

Maarten de Vries

15 Jan 15 Jan

2:34 p.m.

New subject: [arch-projects] [devtools] [PATCH 2/2] makechrootpkg: Allow setting pacman cache for arch-nspawn.

Signed-off-by: Maarten de Vries --- makechrootpkg.in | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index d81be84..ad65de4 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -38,6 +38,7 @@ usage() { echo 'Flags:' echo '-h This help' echo '-c Clean the chroot before building' + echo '-C <dir> Use a folder on the host as pacman cache' echo '-d <dir> Bind directory into build chroot as read-write' echo '-D <dir> Bind directory into build chroot as read-only' echo '-u Update the working copy of the chroot before building' @@ -147,7 +148,7 @@ install_packages() { pkgnames=("${install_pkgs[@]##*/}") cp -- "${install_pkgs[@]}" "$copydir/root/" - arch-nspawn "$copydir" "${bindmounts_ro[@]}" "${bindmounts_rw[@]}" \ + nspawn "$copydir" \ pacman -U --noconfirm -- "${pkgnames[@]/#//root/}" ret=$? rm -- "${pkgnames[@]/#/$copydir/root/}" @@ -301,6 +302,22 @@ move_products() { fi done } + +# Usage: nspawn $copydir $nspawn-args ... +# Globals: +# - pacman_cache +# - bindmounts_ro +# - bindmounts_rw +nspawn() { + local copydir=$1 + shift + + if [[ -n $pacman_cache ]]; then + arch-nspawn -c "$pacman_cache" "$copydir" "${bindmounts_ro[@]}" "${bindmounts_rw[@]}" "$@" + else + arch-nspawn "$copydir" "${bindmounts_ro[@]}" "${bindmounts_rw[@]}" "$@" + fi +} # }}} main() { @@ -317,6 +334,7 @@ main() { declare -a install_pkgs declare -i ret=0 + pacman_cache= bindmounts_ro=() bindmounts_rw=() @@ -325,9 +343,10 @@ main() { [[ -z "$copy" || $copy = root ]] && copy=copy src_owner=${SUDO_USER:-$USER} - while getopts 'hcur:I:l:nTD:d:U:' arg; do + while getopts 'hcur:I:l:nTC:D:d:U:' arg; do case "$arg" in c) clean_first=true ;; + C) pacman_cache="$OPTARG" ;; D) bindmounts_ro+=("--bind-ro=$OPTARG") ;; d) bindmounts_rw+=("--bind=$OPTARG") ;; u) update_first=true ;; @@ -395,8 +414,7 @@ main() { sync_chroot "$chrootdir/root" "$copydir" "$copy" fi - $update_first && arch-nspawn "$copydir" \ - "${bindmounts_ro[@]}" "${bindmounts_rw[@]}" \ + $update_first && nspawn "$copydir" \ pacman -Syu --noconfirm if [[ -n ${install_pkgs[*]:-} ]]; then @@ -415,10 +433,9 @@ main() { prepare_chroot "$copydir" "$USER_HOME" "$keepbuilddir" "$run_namcap" - if arch-nspawn "$copydir" \ + if nspawn "$copydir" \ --bind="$PWD:/startdir" \ --bind="$SRCDEST:/srcdest" \ - "${bindmounts_ro[@]}" "${bindmounts_rw[@]}" \ /chrootbuild "${makepkg_args[@]}" then move_products "$copydir" "$src_owner" -- 2.20.1

Eli Schwartz

2:41 p.m.

On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote:

...

These patches make it possible to build withouth a shared pacman cache using makechrootpkg. I need this myself because I'm building packages for different repositories where some of them contain packages with the same name, but different compile time configuration.

I'd suggest using different pkgnames to be honest. :D

...

It also allows building packages for different Arch based distros on the same host more easily.

Arch Linux 32 has the same issue and solved it using this much simpler patch: https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85c... The solution would then be to add the different cachedirs to the pacman.conf you use, which is anyways going to be different due to being i686-specific. -- Eli Schwartz Bug Wrangler and Trusted User

Maarten de Vries

2:56 p.m.

On Tue, 15 Jan 2019 at 14:41, Eli Schwartz via arch-projects wrote:

...

On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote:

...
These patches make it possible to build withouth a shared pacman cache using makechrootpkg. I need this myself because I'm building packages for different repositories where some of them contain packages with the same name, but different compile time configuration.

I'd suggest using different pkgnames to be honest. :D

I can see your point there, but many of the PKGBUILDs are automatically generated. It would be possible to add prefixes of course, but we'll get very long package names. And the repositories are never used together, so for the end-user it looks weird.

...

...
It also allows building packages for different Arch based distros on the same host more easily.

Arch Linux 32 has the same issue and solved it using this much simpler patch: https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85c...

The solution would then be to add the different cachedirs to the pacman.conf you use, which is anyways going to be different due to being i686-specific.

But this will still bind a directory from the host into the container right? It would require me to create separate cache directories on the host and create matching pacman.conf files. It would indeed be a simpler patch to devtools, but the whole process is harder. I do think these patches make sense.

...

-- Eli Schwartz Bug Wrangler and Trusted User

Erich Eckner

3:01 p.m.

...

...
...
It also allows building packages for different Arch based distros on the same host more easily.

Arch Linux 32 has the same issue and solved it using this much simpler patch: https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85c...

The solution would then be to add the different cachedirs to the pacman.conf you use, which is anyways going to be different due to being i686-specific.

But this will still bind a directory from the host into the container right? It would require me to create separate cache directories on the host and create matching pacman.conf files. It would indeed be a simpler patch to devtools, but the whole process is harder.

This is inevitable if you want to have a cache which persist through cleaning the root/ chroot (the option is -c, IIRC) - which is IMHO a good thing to have.

...

I do think these patches make sense.

Only if you plan to use no (persistent) cache at all. regards, Erich

Eli Schwartz

3:03 p.m.

On 1/15/19 8:56 AM, Maarten de Vries wrote:

...

On Tue, 15 Jan 2019 at 14:41, Eli Schwartz via arch-projects wrote:

...
On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote:

...
These patches make it possible to build withouth a shared pacman cache using makechrootpkg. I need this myself because I'm building packages for different repositories where some of them contain packages with the same name, but different compile time configuration.

I'd suggest using different pkgnames to be honest. :D

I can see your point there, but many of the PKGBUILDs are automatically generated. It would be possible to add prefixes of course, but we'll get very long package names. And the repositories are never used together, so for the end-user it looks weird.

Without being able to visualize what you're doing I cannot say whether it makes sense to me. ¯\_(ツ)_/¯

...

...
...
It also allows building packages for different Arch based distros on the same host more easily.

Arch Linux 32 has the same issue and solved it using this much simpler patch: https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85c...

The solution would then be to add the different cachedirs to the pacman.conf you use, which is anyways going to be different due to being i686-specific.

But this will still bind a directory from the host into the container right? It would require me to create separate cache directories on the host and create matching pacman.conf files. It would indeed be a simpler patch to devtools, but the whole process is harder.

I do think these patches make sense.

You could just reuse the cache directory that is being used as the custom repo itself. You could even simply use this as an additional cache directory, and no packages will ever be downloaded to the main one. -- Eli Schwartz Bug Wrangler and Trusted User

Maarten de Vries

3:11 p.m.

On Tue, 15 Jan 2019 at 15:03, Eli Schwartz wrote:

...

On 1/15/19 8:56 AM, Maarten de Vries wrote:

...
On Tue, 15 Jan 2019 at 14:41, Eli Schwartz via arch-projects wrote:

...
On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote:

...
These patches make it possible to build withouth a shared pacman cache using makechrootpkg. I need this myself because I'm building packages for different repositories where some of them contain packages with the same name, but different compile time configuration.

I'd suggest using different pkgnames to be honest. :D

I can see your point there, but many of the PKGBUILDs are automatically generated. It would be possible to add prefixes of course, but we'll get very long package names. And the repositories are never used together, so for the end-user it looks weird.

Without being able to visualize what you're doing I cannot say whether it makes sense to me. ¯\_(ツ)_/¯

...
...
...
It also allows building packages for different Arch based distros on the same host more easily.

Arch Linux 32 has the same issue and solved it using this much simpler patch: https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85c...

The solution would then be to add the different cachedirs to the pacman.conf you use, which is anyways going to be different due to being i686-specific.

But this will still bind a directory from the host into the container right? It would require me to create separate cache directories on the host and create matching pacman.conf files. It would indeed be a simpler patch to devtools, but the whole process is harder.

I do think these patches make sense.

You could just reuse the cache directory that is being used as the custom repo itself. You could even simply use this as an additional cache directory, and no packages will ever be downloaded to the main one.

Hmm, that sounds useful indeed. Only problem for me here is that the host building the packages is not the one who stores them. Otherwise that would be very neat. Also, Erich makes a good point that having a persistent cache is indeed a good thing, also across cleaning the chroot. I suppose that maybe then these patches aren't needed at all. It might still be useful to have a command line argument to set the cache, but maintaining a modified pacman.conf isn't that problematic if I have to make sure the directories exist anyway. So, thanks for the quick feedback Eli and Erich, I'll solve this differently :)

...

-- Eli Schwartz Bug Wrangler and Trusted User

Maarten de Vries

17 Jan 17 Jan

12:27 p.m.

The commit you linked earlier[1], is that something that will find it's way into mainline devtools? [1] https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85c... On Tue, 15 Jan 2019 at 15:11, Maarten de Vries wrote:

...

On Tue, 15 Jan 2019 at 15:03, Eli Schwartz wrote:

...
On 1/15/19 8:56 AM, Maarten de Vries wrote:

...
On Tue, 15 Jan 2019 at 14:41, Eli Schwartz via arch-projects wrote:

...
On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote:

...
These patches make it possible to build withouth a shared pacman cache using makechrootpkg. I need this myself because I'm building packages for different repositories where some of them contain packages with the same name, but different compile time configuration.

I'd suggest using different pkgnames to be honest. :D

I can see your point there, but many of the PKGBUILDs are automatically generated. It would be possible to add prefixes of course, but we'll get very long package names. And the repositories are never used together, so for the end-user it looks weird.

Without being able to visualize what you're doing I cannot say whether it makes sense to me. ¯\_(ツ)_/¯

...
...
...
It also allows building packages for different Arch based distros on the same host more easily.

Arch Linux 32 has the same issue and solved it using this much simpler patch: https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85c...

The solution would then be to add the different cachedirs to the pacman.conf you use, which is anyways going to be different due to being i686-specific.

But this will still bind a directory from the host into the container right? It would require me to create separate cache directories on the host and create matching pacman.conf files. It would indeed be a simpler patch to devtools, but the whole process is harder.

I do think these patches make sense.

You could just reuse the cache directory that is being used as the custom repo itself. You could even simply use this as an additional cache directory, and no packages will ever be downloaded to the main one.

Hmm, that sounds useful indeed. Only problem for me here is that the host building the packages is not the one who stores them. Otherwise that would be very neat.

Also, Erich makes a good point that having a persistent cache is indeed a good thing, also across cleaning the chroot.

I suppose that maybe then these patches aren't needed at all. It might still be useful to have a command line argument to set the cache, but maintaining a modified pacman.conf isn't that problematic if I have to make sure the directories exist anyway.

So, thanks for the quick feedback Eli and Erich, I'll solve this differently :)

...
-- Eli Schwartz Bug Wrangler and Trusted User

Eli Schwartz

5:15 p.m.

On 1/17/19 6:27 AM, Maarten de Vries wrote:

...

The commit you linked earlier[1], is that something that will find it's way into mainline devtools?

[1] https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85c... Unlikely. Not that I'm opposed to it happening, it's just totally not worth my time to even try.

The last time I tried to submit a patch for devtools was here: https://git.archlinux.org/devtools.git/commit/?id=5b3c14454a9c1ec00a3ef11f3f... This is when it was finally merged: https://lists.archlinux.org/pipermail/arch-projects/2018-May/004897.html It was only merged at that time, because despite my having contributed a patch months before which make makechrootpkg work *at all* with pacman 5.1 on the host system, no one was actually reading this mailing list who had commit rights for devtools. Eventually, when pacman 5.1 was released, people started complaining in the private staff channel that they could no longer build packages, which is sort of awkward for the maintenance of the distribution, right? Anyway, I said "I have pending patches from months ago that were supposed to make this a non-issue", and I appealed directly to Allan in order to merge it, and he asked me for a commit id that I thought should be merged. And this is the story of how I actually managed to get some changes into devtools. By hunting down someone on IRC *after* the emergency has already happened. (Observant witnesses will note that there is another patch after that, also written by me. That doesn't count, as one of the devtools maintainers had something which bothered them enough to work on devtools, and asked about it in IRC, and I wrote the patch basically on request.) ... I have lots of changes I want to actually make practical use of on a personal level, and I *also* want to use modifications in order to build packages for an archlinux32 chroot. I'm well on the way to totally forking everything, and my changes are just going to get more significant. Attempting to upstream a collection of controversial as well as non-controversial changes, when my chances of even being heard in the first place are... dim... is not worth the time when I can simply sudo make install something that makes me happy and gets things done. -- Eli Schwartz Bug Wrangler and Trusted User

Maarten de Vries

5:24 p.m.

On Thu, 17 Jan 2019 at 17:15, Eli Schwartz wrote:

...

On 1/17/19 6:27 AM, Maarten de Vries wrote:

...
The commit you linked earlier[1], is that something that will find it's way into mainline devtools?

[1] https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85c... Unlikely. Not that I'm opposed to it happening, it's just totally not worth my time to even try.

The last time I tried to submit a patch for devtools was here: https://git.archlinux.org/devtools.git/commit/?id=5b3c14454a9c1ec00a3ef11f3f...

This is when it was finally merged: https://lists.archlinux.org/pipermail/arch-projects/2018-May/004897.html

It was only merged at that time, because despite my having contributed a patch months before which make makechrootpkg work *at all* with pacman 5.1 on the host system, no one was actually reading this mailing list who had commit rights for devtools.

Eventually, when pacman 5.1 was released, people started complaining in the private staff channel that they could no longer build packages, which is sort of awkward for the maintenance of the distribution, right? Anyway, I said "I have pending patches from months ago that were supposed to make this a non-issue", and I appealed directly to Allan in order to merge it, and he asked me for a commit id that I thought should be merged.

And this is the story of how I actually managed to get some changes into devtools. By hunting down someone on IRC *after* the emergency has already happened.

(Observant witnesses will note that there is another patch after that, also written by me. That doesn't count, as one of the devtools maintainers had something which bothered them enough to work on devtools, and asked about it in IRC, and I wrote the patch basically on request.)

...

I have lots of changes I want to actually make practical use of on a personal level, and I *also* want to use modifications in order to build packages for an archlinux32 chroot. I'm well on the way to totally forking everything, and my changes are just going to get more significant. Attempting to upstream a collection of controversial as well as non-controversial changes, when my chances of even being heard in the first place are... dim... is not worth the time when I can simply sudo make install something that makes me happy and gets things done.

I see. In that case I'll stick with a fork for now. Thanks again for the feedback :) -- Maarten

1923

Age (days ago)

1925

Last active (days ago)

List overview

Download

9 comments

3 participants

participants (3)

Eli Schwartz
Erich Eckner
Maarten de Vries

[arch-projects] [devtools] [PATCH 1/2] arch-nspawn, mkarchroot: Allow not sharing the cache directories.

tags

participants (3)