June 2019 - Arch-projects - lists.archlinux.org

[arch-projects] [netctl] netctl, cloud-init, and systemd
by Conrad Hoffmann 18 Jun '19

18 Jun '19

Ohai! One of my pet projects atm is creating useful Arch Linux cloud images (mostly EC2 AMIs), which mostly means giving some lovin' to the cloud-init AUR package [1]. My dream is for Arch to one day have official cloud images, but that's a completely different story ;) So, it turns out the default kernel and two AUR packages [1][2] already get you a pretty long way. One thing that remains a bit difficult though is network configuration, and that's what this is about. In case you are not familiar with cloud-init, the idea is that you can build a single OS image that runs cloud-init on boot, and cloud-init will take care of such things as network configuration, so that the same image will work regardless of the network setup you choose for the cloud instance. The current cloud-init implementation for Arch uses netctl [3]. The implementation is correct in such a way that it does indeed render the right netctl profile(s) and enables them. However there is a problem: they are not being started. AFAICT this is because cloud-init does this while the systemd boot is already in process, and changing the dependency graph (by adding new units) does not have any effect until the next run (everything works right on second boot). Note that I even tried having cloud-init run `systemd daemon-reload` after enabling the units, but it didn't help either. The reason I am posting this here is that this seems to be an issue due to the particular way netctl use systemd units. Since you don't know the names or the number of profiles (units) that will be generated during image creation, you cannot enable them at that time. But doing so during first boot does not seem to work. Just for comparison, if one were to use e.g. systemd-networkd instead, you would just enable the systemd-networkd unit during image creation, cloud-init could generate the appropriate config for any number of devices, and when the unit starts it will do the right thing. Likewise on other distros, e.g. Debian with /etc/network/interfaces or such. Now, from my point of view, there could be several approaches to solve this: 1. systemd supports updates of the dep graph during boot 2. support such a use case in netctl 3. change cloud-init to use systemd-networkd for Arch Let me quickly elaborate: 1. is intentionally not phrased as something to be done. It might already be a thing, I just couldn't figure out how to do it. If someone knows more about this, I would love to hear about it. If this works, it would be the easiest solution. However, if it doesn't, I don't have my hopes up high for this being added to systemd anytime soon. 2. is the main reason I am writing this. Things that came to mind were another special unit (netctl-all?), or even just a well-defined interface to write devices into the state file, so that the plain netctl unit would work. I would be very interested to hear how such a thing sounds to you, the developers? 3. Is of course an option, but would require quite a bit of work in cloud-init. That work, if done right, might however at some point benefit other distros, should they be using systemd-networkd as well. The main reason I am also bringing this up that I was wondering if there are possibly any plans to abandon netctl anyways at some point in favor of distro-agnostic solutions (be it systemd-networkd or any other). So, sorry for the long mail, but I probably omitted a few crucial details already, so don't hesitate to ask for clarifications :) I would love to hear your thoughts on this! Thanks a lot, Conrad [1] https://aur.archlinux.org/packages/cloud-init/ [2] https://aur.archlinux.org/packages/growpart/ [3] https://git.launchpad.net/cloud-init/tree/cloudinit/distros/arch.py

3 3

[arch-projects] [devtools] [GIT] The official devtools repository branch master updated. 20190329-6-gf522ce2
by Levente Polyak 12 Jun '19

12 Jun '19

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "The official devtools repository". The branch, master has been updated via f522ce2277100a147baa2656753231b14cf1f71d (commit) from 774774961025b79d84a466f75c651ce17c946928 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit f522ce2277100a147baa2656753231b14cf1f71d Author: Levente Polyak <anthraxx(a)archlinux.org> Date: Wed Jun 12 23:59:01 2019 +0200 offload-build: disable shellcheck false-positive for shell-escape ----------------------------------------------------------------------- Summary of changes: offload-build | 1 + 1 file changed, 1 insertion(+) hooks/post-receive -- The official devtools repository

1 0

[arch-projects] [devtools] [GIT] The official devtools repository branch master updated. 20190329-5-g7747749
by Levente Polyak 12 Jun '19

12 Jun '19

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "The official devtools repository". The branch, master has been updated discards 5b5cd407c14c50976455f337ca479f5c012c3eab (commit) discards aa7a454e03c9252cbb4fea7c3689abff88fa4ba4 (commit) discards 9aad92338c3150fe5314b2fe20c8bf6dceaee778 (commit) discards b3fca0a2e6f1162cec99f23ba3fc8693c4236ba5 (commit) via 774774961025b79d84a466f75c651ce17c946928 (commit) via 25b4ef279925a964b0de0ed8675d75e1532bf577 (commit) via ad4b66830a6198a9920570380f673f547a488c2f (commit) This update added new revisions after undoing existing revisions. That is to say, the old revision is not a strict subset of the new revision. This situation occurs when you --force push a change and generate a repository containing something like this: * -- * -- B -- O -- O -- O (5b5cd407c14c50976455f337ca479f5c012c3eab) \ N -- N -- N (774774961025b79d84a466f75c651ce17c946928) When this happens we assume that you've already had alert emails for all of the O revisions, and so we here report only the revisions in the N branch from the common base, B. Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 774774961025b79d84a466f75c651ce17c946928 Author: Eli Schwartz via arch-projects <arch-projects(a)archlinux.org> Date: Tue Jun 4 12:11:20 2019 -0400 doc: add manpage for the new offload-build tool Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> commit 25b4ef279925a964b0de0ed8675d75e1532bf577 Author: Eli Schwartz via arch-projects <arch-projects(a)archlinux.org> Date: Tue Jun 4 12:11:19 2019 -0400 Add new tool: offload-build This tool is useful for running makechrootpkg on a remote build server, and is by default hooked up to send a PKGBUILD and initiate a build on our shiny new build server "dragon". Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> commit ad4b66830a6198a9920570380f673f547a488c2f Author: Eli Schwartz via arch-projects <arch-projects(a)archlinux.org> Date: Thu May 9 22:58:26 2019 -0400 Revert "makechrootpkg: sync_chroot: Make more general." This reverts commit 6d1992909cc46e293027ff488ae2632047603e66. It has never worked. In commit c86823a2d4a4152c71faa1c3bab227756232996f it was noted that it compared the device numbers for [[ $1 = $1 ]] which was a useless check and always returned true, for *any* btrfs filesystem. Now that the function is corrected to compare [[ $1 = $2 ]] the check is still useless, but this time because it always returns false -- btrfs subvolumes on the same filesystem do *not* share device numbers. So let's go back to the original working implementation that only matters in terms of makechrootpkg, and just checks if makechrootpkg's root working directory is btrfs (in which case we know it will be a subvolume because mkarchroot will create it that way). This restores our special support for the btrfs filesystem. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> ----------------------------------------------------------------------- Summary of changes: .gitignore | 1 + Makefile | 1 + finddeps.in | 41 +++++++++++++++++++++++++++++++++++++++++ zsh_completion.in | 6 +++++- 4 files changed, 48 insertions(+), 1 deletion(-) create mode 100644 finddeps.in hooks/post-receive -- The official devtools repository

1 0

[arch-projects] [devtools] [GIT] The official devtools repository branch master updated. 20190329-6-g5b5cd40
by Levente Polyak 12 Jun '19

12 Jun '19

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "The official devtools repository". The branch, master has been updated via 5b5cd407c14c50976455f337ca479f5c012c3eab (commit) via aa7a454e03c9252cbb4fea7c3689abff88fa4ba4 (commit) via 9aad92338c3150fe5314b2fe20c8bf6dceaee778 (commit) via b3fca0a2e6f1162cec99f23ba3fc8693c4236ba5 (commit) from 321e998020cfdb337c1ebc1ac41f5e729b9e276c (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 5b5cd407c14c50976455f337ca479f5c012c3eab Author: Eli Schwartz via arch-projects <arch-projects(a)archlinux.org> Date: Tue Jun 4 12:11:20 2019 -0400 doc: add manpage for the new offload-build tool Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> commit aa7a454e03c9252cbb4fea7c3689abff88fa4ba4 Author: Eli Schwartz via arch-projects <arch-projects(a)archlinux.org> Date: Tue Jun 4 12:11:19 2019 -0400 Add new tool: offload-build This tool is useful for running makechrootpkg on a remote build server, and is by default hooked up to send a PKGBUILD and initiate a build on our shiny new build server "dragon". Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> commit 9aad92338c3150fe5314b2fe20c8bf6dceaee778 Author: Eli Schwartz via arch-projects <arch-projects(a)archlinux.org> Date: Thu May 9 22:58:26 2019 -0400 Revert "makechrootpkg: sync_chroot: Make more general." This reverts commit 6d1992909cc46e293027ff488ae2632047603e66. It has never worked. In commit c86823a2d4a4152c71faa1c3bab227756232996f it was noted that it compared the device numbers for [[ $1 = $1 ]] which was a useless check and always returned true, for *any* btrfs filesystem. Now that the function is corrected to compare [[ $1 = $2 ]] the check is still useless, but this time because it always returns false -- btrfs subvolumes on the same filesystem do *not* share device numbers. So let's go back to the original working implementation that only matters in terms of makechrootpkg, and just checks if makechrootpkg's root working directory is btrfs (in which case we know it will be a subvolume because mkarchroot will create it that way). This restores our special support for the btrfs filesystem. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> commit b3fca0a2e6f1162cec99f23ba3fc8693c4236ba5 Author: Jelle van der Waa <jelle(a)vdwaa.nl> Date: Mon May 6 16:10:09 2019 +0200 Remove finddeps tool finddeps depends on a no longer existing ABS tree. This data can also be queried via archweb. ----------------------------------------------------------------------- Summary of changes: .gitignore | 1 - Makefile | 3 +- doc/offload-build.1.asciidoc | 52 +++++++++++++++++++++ finddeps.in | 41 ----------------- lib/archroot.sh | 8 ---- makechrootpkg.in | 32 ++++++------- offload-build | 107 +++++++++++++++++++++++++++++++++++++++++++ zsh_completion.in | 6 +-- 8 files changed, 175 insertions(+), 75 deletions(-) create mode 100644 doc/offload-build.1.asciidoc delete mode 100644 finddeps.in create mode 100755 offload-build hooks/post-receive -- The official devtools repository

1 0

[arch-projects] [devtools] [PATCH] Add new tool: offload-build
by Eli Schwartz 04 Jun '19

04 Jun '19

This tool is useful for running makechrootpkg on a remote build server, and is by default hooked up to send a PKGBUILD and initiate a build on our shiny new build server "dragon". Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- manpage documentation hopefully coming tonight. Makefile | 1 + offload-build | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+) create mode 100755 offload-build diff --git a/Makefile b/Makefile index ba2d3e4..2306a17 100644 --- a/Makefile +++ b/Makefile @@ -20,6 +20,7 @@ IN_PROGS = \ BINPROGS = \ $(IN_PROGS) \ + offload-build \ sogrep CONFIGFILES = \ diff --git a/offload-build b/offload-build new file mode 100755 index 0000000..66609c6 --- /dev/null +++ b/offload-build @@ -0,0 +1,100 @@ +#!/bin/bash +# +# offload-build - build a PKGBUILD on a remote server using makechrootpkg. +# +# Copyright (c) 2019 by Eli Schwartz <eschwartz(a)archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <https://www.gnu.org/licenses/>. +# + + +# global defaults suitable for use by Arch staff +repo=extra +arch=x86_64 +server=dragon.archlinux.org + +die() { printf "error: $1\n" "${@:2}"; exit 1; } + +usage() { + cat <<- _EOF_ + Usage: ${BASH_SOURCE[0]##*/} [--repo REPO] [--arch ARCHITECTURE] [--server SERVER] + + Build a PKGBUILD on a remote server using makechrootpkg. Requires a remote user + that can run archbuild without password auth. + + OPTIONS + -r, --repo Build against a specific repository (current: $repo) + -a, --arch Build against a specific architecture (current: $arch) + -s, --server Offload to a specific build server (current: $server) + -h, --help Show this help text +_EOF_ +} + +# option checking +while (( $# )); do + case $1 in + -h|--help) + usage + exit 0 + ;; + -r|--repo) + repo=$2 + shift 2 + ;; + -a|--arch) + arch=$2 + shift 2 + ;; + -s|--server) + server=$2 + shift 2 + ;; + *) + die "invalid argument: %s" "$1" + ;; + esac +done + +# multilib must be handled specially +if [[ $repo = multilib* ]]; then + arch= +fi + +trap 'rm -rf $SRCPKGDEST' EXIT + +# Use a source-only tarball as an intermediate to transfer files. This +# guarantees the checksums are okay, and guarantees that all needed files are +# transferred, including local sources, install scripts, and changelogs. +export SRCPKGDEST=$(mktemp -d) +makepkg --source || die "unable to make source package" + +mapfile -t files < <( + # This is sort of bash golfing but it allows running a mildly complex + # command over ssh with a single connection. + cat "$SRCPKGDEST"/*.src.tar.gz | + ssh $server ' + temp="${XDG_CACHE_HOME:-$HOME/.cache}/offload-build" && + mkdir -p "$temp" && + temp=$(mktemp -d -p "$temp") && + cd "$temp" && + { + bsdtar --strip-components 1 -xvf - && + script -qfc '"${repo}${arch:+-$arch}-build"' /dev/null && + printf "%s\n" "" "-> build complete" && + printf "\t%s\n" "$temp"/* + } >&2 && + makepkg --packagelist +') + +(( ${#files[@]} )) && printf '%s\n' '' '-> copying files...' && scp "${files[@]/#/$server:}" . -- 2.21.0

1 4