[arch-releng] RFC: Automated Install
Hey, all. One of the things that I feel limits Arch Linux is the lack of an ability to kick off an unattended install. Sure, you can boot via PXE or the like, but the actual install does not take place. I'm in the middle of writing a tool that will do just this[0]. Much like RedHat's Kickstart or Debian's preseed, it uses kernel commandline options/parameters to fetch a configuration, parses that configuration, and then installs Arch according to that specified configuration. (Everything from partitioning disks to installing certain packages, etc.) Note that it isn't intended to make the install process "easier", per se; it intends to do it with *minimal to no user interaction*. One would need a very clear understanding and familiarity with the manual Arch install process to make use of this. The reason I'm even writing to this list instead of say, just having a package on the AUR as it makes little to no sense to run this software on anything but the install medium. In order to perform things like disk partitioning and the like, I'm thinking of using the udisks2 API via DBUS. Unfortunately, this means that it (udisks2) would need to be included in the live ISO image (or an alternate one would be spun up). Alternatively I could do it via libblockdev and gobject-introspection, but that is... significantly more complicated. As such, a couple questions: 1.) Is the RelEng team interested in adding such an unattended install ability to Arch Linux (especially one developed externally)? 2.) a.) Would udisks2 (and perhaps other libraries) be able to be added to the standard installer ISO? As of udisks2-2.9.2-1, for instance, this would add 14086.78 KiB (~14 MiB) to the squashed root (before squashing). 2.) b.) If not, would the amount of maintenance overhead required for mastering a separate ISO be worth the feature-add? Thank you in advance for your feedback. [0] In Golang so it could be easy to package, contained to a relatively localized install, and still relatively accessible to outside contributors. A couple years ago I had an implementation written in Python but this was discouraged as it would be far too complex to maintain, both from a codebase and a RelEng perspective (which, fair point). Golang also would allow for some cleaner parallel operations as well, though one may argue this is more of a nice-to-have in an install context. -- brent saner https://square-r00t.net/ GPG info: https://square-r00t.net/gpg-info
brent s. via arch-releng <arch-releng@lists.archlinux.org> 於 2021年7月9日 週五 下午2:24寫道：
Hey, all.
One of the things that I feel limits Arch Linux is the lack of an ability to kick off an unattended install. Sure, you can boot via PXE or the like, but the actual install does not take place.
Just FYI, maybe not what you want, but I have been sending the script= kernel cmdline from the PXE server: https://gitlab.archlinux.org/archlinux/archiso/-/blob/master/configs/releng/...
Whoops! Goofed. I meant to send this to the list, not to Li-Yu directly. Apologies, Li-Yu! -------- Forwarded Message -------- Subject: Re: [arch-releng] RFC: Automated Install To: Li-Yu Yu <afg984@gmail.com> From: brent s. <bts@square-r00t.net> Date: Fri, 9 Jul 2021 02:41:55 -0400 On 7/9/21 2:32 AM, Li-Yu Yu wrote:
brent s. via arch-releng <arch-releng@lists.archlinux.org> 於 2021年7月9日 週五 下午2:24寫道： Just FYI, maybe not what you want, but I have been sending the script= kernel cmdline from the PXE server:
https://gitlab.archlinux.org/archlinux/archiso/-/blob/master/configs/releng/...
Thanks! Yeah, I've seen a couple implementations of scripted installs, and it's handy that it's now included as a kernel param! But that's a bit too narrow compared to what I'm thinking of. Compared to something a little more complex like Kickstart[0] or Preseed[1], install scripts tend to feel a little limiting. The benefit of frameworks that can use configurations/profiles/etc. is that they're really extendable by users without them needing to script it themselves (or to completely change a script someone else wrote, etc.). The docs[2] need to be updated (as they were written back from my python implementation), but generally speaking not much should change from the user's end if you want to get an idea of what I came up with/am working on. Notably, the repo has changed for the new implementation[3]. (Again, nothing usable yet as I'm heavily developing it currently. The structs should mostly be done, though, if you want an idea of hierarchy and parsing.) [0] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm... [1] https://wiki.debian.org/DebianInstaller/Preseed [2] https://aif.square-r00t.net/ [3] https://git.square-r00t.net/AIF/ -- brent saner https://square-r00t.net/ GPG info: https://square-r00t.net/gpg-info
fuck ------------------ 原始邮件 ------------------ 发件人: "Arch Linux Release Engineering" <arch-releng@lists.archlinux.org>; 发送时间: 2021年7月9日(星期五) 下午2:45 收件人: "arch-releng"<arch-releng@lists.archlinux.org>; 抄送: "brent s."<bts@square-r00t.net>; 主题: Re: [arch-releng] RFC: Automated Install Whoops! Goofed. I meant to send this to the list, not to Li-Yu directly. Apologies, Li-Yu! -------- Forwarded Message -------- Subject: Re: [arch-releng] RFC: Automated Install To: Li-Yu Yu <afg984@gmail.com> From: brent s. <bts@square-r00t.net> Date: Fri, 9 Jul 2021 02:41:55 -0400 On 7/9/21 2:32 AM, Li-Yu Yu wrote: > brent s. via arch-releng <arch-releng@lists.archlinux.org> 於 2021年7月9日 > 週五 下午2:24寫道： > Just FYI, maybe not what you want, but I have been sending the script= > kernel cmdline from the PXE server: > https://gitlab.archlinux.org/archlinux/archiso/-/blob/master/configs/releng/... > Thanks! Yeah, I've seen a couple implementations of scripted installs, and it's handy that it's now included as a kernel param! But that's a bit too narrow compared to what I'm thinking of. Compared to something a little more complex like Kickstart[0] or Preseed[1], install scripts tend to feel a little limiting. The benefit of frameworks that can use configurations/profiles/etc. is that they're really extendable by users without them needing to script it themselves (or to completely change a script someone else wrote, etc.). The docs[2] need to be updated (as they were written back from my python implementation), but generally speaking not much should change from the user's end if you want to get an idea of what I came up with/am working on. Notably, the repo has changed for the new implementation[3]. (Again, nothing usable yet as I'm heavily developing it currently. The structs should mostly be done, though, if you want an idea of hierarchy and parsing.) [0] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm... [1] https://wiki.debian.org/DebianInstaller/Preseed [2] https://aif.square-r00t.net/ [3] https://git.square-r00t.net/AIF/ -- brent saner https://square-r00t.net/ GPG info: https://square-r00t.net/gpg-info
fuck ------------------ 原始邮件 ------------------ 发件人: "Arch Linux Release Engineering" <arch-releng@lists.archlinux.org>; 发送时间: 2021年7月9日(星期五) 下午2:32 收件人: "Arch Linux Release Engineering"<arch-releng@lists.archlinux.org>; 抄送: "Li-Yu Yu"<afg984@gmail.com>; 主题: Re: [arch-releng] RFC: Automated Install brent s. via arch-releng <arch-releng@lists.archlinux.org> 於 2021年7月9日 週五 下午2:24寫道： > > Hey, all. > > One of the things that I feel limits Arch Linux is the lack of an > ability to kick off an unattended install. Sure, you can boot via PXE or > the like, but the actual install does not take place. Just FYI, maybe not what you want, but I have been sending the script= kernel cmdline from the PXE server: https://gitlab.archlinux.org/archlinux/archiso/-/blob/master/configs/releng/...
There exists a nice solution to install different linux operating systems, installimage: * https://github.com/virtapi/installimage (my version) * https://github.com/hetzneronline/installimage (the hetzner version) This can be executed via any live linux, for example a custom arch PXE: * https://github.com/virtapi/LARS This utilizes the mentioned .automated_script.sh to run installimage at multiple big ISPs. There is also a Python based API to manage DHCP, PXE and installimage configs via a web API at are processed by isc-dhcp (their ldap backend)/tftp-hpa with pxelinux/installimage * https://github.com/virtapi/marmoset Let me know if you've questions about any of these projects. The installimage and LARS repos also have flowcharts in their README.md. Cheers, Tim On 09.07.21 08:32, Li-Yu Yu via arch-releng wrote:
brent s. via arch-releng <arch-releng@lists.archlinux.org> 於 2021年7月9日 週五 下午2:24寫道：
Hey, all.
One of the things that I feel limits Arch Linux is the lack of an ability to kick off an unattended install. Sure, you can boot via PXE or the like, but the actual install does not take place.
Just FYI, maybe not what you want, but I have been sending the script= kernel cmdline from the PXE server: https://gitlab.archlinux.org/archlinux/archiso/-/blob/master/configs/releng/...
(I'm not a programmer, but) wouldn't program sgdisk that belongs to the gptfdisk package be enough to do the automated/scripted partitioning. I'm pretty sure it is in the official ISO. At least, it was a few years ago when I used it a few times. dif ---------------------------- On 09.07.2021 08:24, brent s. via arch-releng wrote:
Hey, all.
One of the things that I feel limits Arch Linux is the lack of an ability to kick off an unattended install. Sure, you can boot via PXE or the like, but the actual install does not take place.
I'm in the middle of writing a tool that will do just this[0]. Much like RedHat's Kickstart or Debian's preseed, it uses kernel commandline options/parameters to fetch a configuration, parses that configuration, and then installs Arch according to that specified configuration. (Everything from partitioning disks to installing certain packages, etc.) Note that it isn't intended to make the install process "easier", per se; it intends to do it with *minimal to no user interaction*. One would need a very clear understanding and familiarity with the manual Arch install process to make use of this.
The reason I'm even writing to this list instead of say, just having a package on the AUR as it makes little to no sense to run this software on anything but the install medium.
In order to perform things like disk partitioning and the like, I'm thinking of using the udisks2 API via DBUS. Unfortunately, this means that it (udisks2) would need to be included in the live ISO image (or an alternate one would be spun up).
Alternatively I could do it via libblockdev and gobject-introspection, but that is... significantly more complicated.
As such, a couple questions:
1.) Is the RelEng team interested in adding such an unattended install ability to Arch Linux (especially one developed externally)?
2.) a.) Would udisks2 (and perhaps other libraries) be able to be added to the standard installer ISO? As of udisks2-2.9.2-1, for instance, this would add 14086.78 KiB (~14 MiB) to the squashed root (before squashing).
2.) b.) If not, would the amount of maintenance overhead required for mastering a separate ISO be worth the feature-add?
Thank you in advance for your feedback.
[0] In Golang so it could be easy to package, contained to a relatively localized install, and still relatively accessible to outside contributors. A couple years ago I had an implementation written in Python but this was discouraged as it would be far too complex to maintain, both from a codebase and a RelEng perspective (which, fair point). Golang also would allow for some cleaner parallel operations as well, though one may argue this is more of a nice-to-have in an install context.
On 7/9/21 3:09 AM, dif via arch-releng wrote:
(I'm not a programmer, but) wouldn't program sgdisk that belongs to the gptfdisk package be enough to do the automated/scripted partitioning. I'm pretty sure it is in the official ISO. At least, it was a few years ago when I used it a few times.
dif
sgdisk is indeed still on the install ISO, but this is not a script - it's a compiled program, meant to be run as a daemon on startup. (It will do nothing unless explicitly enabled via kernel parameters, and it strictly doesn't need to be a daemon, but I digress). So if I was writing a bash (or sh, or ksh, or whatever) script, sure - sgdisk would be a fantastic choice. But what I'm working on is not a script. :) (Golang is not an interpreted language; it's a compiled language, like C/C++, Rust, etc.) For something non-interpreted, you shouldn't need to fork a shell just to run a program called by your program. -- brent saner https://square-r00t.net/ GPG info: https://square-r00t.net/gpg-info
Doesn't Golang have a mechanism to run an external program from a program? I remember I wrote a short assembly code (just for the fun of it) to print invoice numbers rotated 90° on a dot-matrix Epson printer from a dBase program like 30 years ago. "C" was around of course, objects have not been born yet, programming was easy. But I never wrote anything in C and I digress :) Anyways, if there is such a mechanism, I'd go for it. Computers have much more RAM these days than they used to in the beginning of the 1990s. Again, I'm not and have never been a programmer. dif ------------ On 09.07.2021 09:47, brent s. via arch-releng wrote:
On 7/9/21 3:09 AM, dif via arch-releng wrote:
(I'm not a programmer, but) wouldn't program sgdisk that belongs to the gptfdisk package be enough to do the automated/scripted partitioning. I'm pretty sure it is in the official ISO. At least, it was a few years ago when I used it a few times.
dif sgdisk is indeed still on the install ISO, but this is not a script - it's a compiled program, meant to be run as a daemon on startup. (It will do nothing unless explicitly enabled via kernel parameters, and it strictly doesn't need to be a daemon, but I digress).
So if I was writing a bash (or sh, or ksh, or whatever) script, sure - sgdisk would be a fantastic choice. But what I'm working on is not a script. :) (Golang is not an interpreted language; it's a compiled language, like C/C++, Rust, etc.)
For something non-interpreted, you shouldn't need to fork a shell just to run a program called by your program.
Hi Brent, On 2021-07-09 02:24:03 (-0400), brent s. via arch-releng wrote:
I'm in the middle of writing a tool that will do just this[0]. Much like RedHat's Kickstart or Debian's preseed, it uses kernel commandline options/parameters to fetch a configuration, parses that configuration, and then installs Arch according to that specified configuration. (Everything from partitioning disks to installing certain packages, etc.) Note that it isn't intended to make the install process "easier", per se; it intends to do it with *minimal to no user interaction*. One would need a very clear understanding and familiarity with the manual Arch install process to make use of this.
There are already a few ways of doing this (in theory): - the script= kernel command line parameter [1] - archinstall [2] - ansible [3] Additionally, we have support for cloud-init on the installation medium since archiso >= 51.
The reason I'm even writing to this list instead of say, just having a package on the AUR as it makes little to no sense to run this software on anything but the install medium.
In order to perform things like disk partitioning and the like, I'm thinking of using the udisks2 API via DBUS. Unfortunately, this means that it (udisks2) would need to be included in the live ISO image (or an alternate one would be spun up).
Alternatively I could do it via libblockdev and gobject-introspection, but that is... significantly more complicated.
As such, a couple questions:
1.) Is the RelEng team interested in adding such an unattended install ability to Arch Linux (especially one developed externally)?
I don't see much of a benefit to add another way of installation automation, especially given that we have the above capabilities.
2.) a.) Would udisks2 (and perhaps other libraries) be able to be added to the standard installer ISO? As of udisks2-2.9.2-1, for instance, this would add 14086.78 KiB (~14 MiB) to the squashed root (before squashing).
The udisks2 package is currently not required for any installation or rescue system specific task (to my knowledge), so there would be no reason to add it. Sorry.
2.) b.) If not, would the amount of maintenance overhead required for mastering a separate ISO be worth the feature-add?
With the archiso profiles [4] I hope it has become much simpler to create a custom ISO (even with custom repostories containing custom packages). Currently I don't see us in the position to add another custom ISO plainly for the purpose of (semi?) unattended installation (given that we have this functionality already). At this point in time, more important milestones to reach are the full automation of release artifact creation [5], more specifically also the consolidation of how/where we release [6] and the full reproducibility of our installation medium [7]. I invite you to look into the existing solutions and contribute there, if possible. Best, David [1] https://gitlab.archlinux.org/archlinux/archiso/-/blob/1b01d22596708a59275b7a... [2] https://wiki.archlinux.org/title/Archinstall [3] e.g. https://gitlab.archlinux.org/archlinux/infrastructure/-/tree/master/roles/in... [4] https://gitlab.archlinux.org/archlinux/archiso/-/blob/1b01d22596708a59275b7a... [5] https://gitlab.archlinux.org/archlinux/releng [6] https://gitlab.archlinux.org/archlinux/releng/-/issues/11 [7] https://gitlab.archlinux.org/archlinux/archiso/-/issues/68 -- https://sleepmap.de
Hi all, FYI, I have removed and banned the offender from the mailing list: https://lists.archlinux.org/pipermail/arch-releng/2021-July/004031.html https://lists.archlinux.org/pipermail/arch-releng/2021-July/004032.html https://lists.archlinux.org/pipermail/arch-releng/2021-July/004037.html This behavior is not in line with our Code of Conduct [1] and will not be tolerated in this community. Best, David [1] https://gitlab.archlinux.org/archlinux/service-agreements/-/blob/master/code... -- https://sleepmap.de
participants (6)
-
brent s.
-
David Runge
-
dif
-
George
-
Li-Yu Yu
-
Tim Meusel