Re: [arch-releng] aif -p automatic

I forwarded our previous mails to the arch releng mailing list, and sending this one straight to it. they could be interesting for the others too. feel free to subscribe and reply directly to the list On Wed, 1 Apr 2009 00:29:47 +1000 Jud wrote:

On Tue, 31 Mar 2009 13:21:51 +0200 Dieter Plaetinck wrote:

...

On Tue, 31 Mar 2009 18:03:44 +1000 Jud wrote:

...

Hi Dieter,

How's it going?

Okay, my right shoulder hurts a bit atm. I hate chairs with "arm rests" that bother me more then they help. Other then that all okay.

Thanks for the great and prompt reply! I have been away from Arch for sometime now and still need to catch up on things. My email was just some ideas from my head without investigation your code, so apologies if something seem un-unix and un-Arch like.

...

...

Noticed you have started working on an automatic install feature for Arch. IMHO this the most important feature Arch needs.

Yeah it's something I really want myself too.

...

I took the liberty of editing your example file to how, from my experience, I would like it to work. Are these ideas possible? There are many questions inline, hope this helps and I am available to answer any of your questions. If you need some help implementing any of the ideas, I like to try and help out.

Thanks. Any help is welcome. I also have a big todo list and various tickets open on the bugtracker. Feel free to explore the codebase, ask me questions and hack on.

I'll look into that.

...

Note that in general I like to adhere to the kiss and especially the unix philosophy/ies. Eg I want to make aif able to output a little "report" at the end (there is a function for it, but it's a work in progress), however I don't believe things like sending mails belongs in aif.

Agreed. Aslong there are hooks available or similar. My idea/goal is to have an installation that a brain-dead monkey could do from a USB stick (once the stick is setup) and/or be controlled by a single admin type person.

...

...

# AIF Automatic.example # these variables are optional, here are the defaults (feel free to omit them) #

...

# Goal: Setup a very workable machine(s) automatically

that goal is not specific to this profile (==config file). you can say this for the complete procedure and all configs for it. in fact, aif's procedures and their goals are explained in the documentation.

One profile per file is fine with me. The goal line was just for me I didn't intend for it to be the AIF's goal. I'll look into the docs

...

...

SOURCE=ftp usb # [rsync http webdav ftp usb disc] if can't get net connection fallback to packages on boot usb stick

what's all this for? we support ftp and "having a repo available in the VFS", mounted at /src/core/pkg afaik. getting the packages in place (either when generating the medium, or at runtime) is not something that belongs in aif imho. besides, aif supports what pacman supports here.

Here my idea was to list the install types and in [] all the possible install types, agreed this is a pacman thing, also I thought of PXE & NFS as a possible install type, i'm sure there would be many more. Also another idea here was, because I wanted this to be automatic, to have the option of fallback options etc.

Well sure you can make a pxe image, mount a package repository over nfs and automatically start aif on boot. aif/pacman do not need to be aware of the pxe/nfs/.. stuff.

...

...

FILE_URL=/dev/sdb1 # does the install script know the boot media and /dev address? Not sure what exactly you mean. SYNC_URL=192.168.1.2 # The ftp:// and /${REPO}/os/${CARCH}/ should be done automatically, fallback to pacman-mirrorlist well, you can't know the full url automatically because there can be parts specific per distro. I'll think I'll just stick to the same format as the mirrorlist. referring to the mirrorlist seems like an okay idea, though if you configure the config file for aif, why would anyone configure the mirrorlist file if you're creating the aif config file already anyway? LOGGING_SERVER= # Address to connect to a remote logging server/service maybe i could use syslog for aif logging. you can then configure syslog for remote logging and whatever EMAIL="Admin Guy " # send errors, finished log, etc # verify address first? EMAIL_SERVER="smtp.example.com:25" belongs in a script that wraps around aif imho. feel free to write this (or convince me why "one program to do it all" is better)

# Do you want to have additional pacman repositories or packages available at runtime (during installation)? RUNTIME_REPOSITORIES= RUNTIME_PACKAGES=openssh ntp ssmtp

# you can optionally also override some functions... worker_intro () { infofy "Automatic procedure running the generic-install-on-sda example config. THIS WILL ERASE AND OVERWRITE YOUR /DEV/SDA. IF YOU DO NOT WANT THIS PRESS CTRL+C WITHIN 10 SECONDS" sleep 10 # countdown onscreen? } # Could/Can this be run from GRUB menu on the boot media - ie no user interaction at all?

this is just an example of how to override worker functions from the procedure. you can leave this out. invoking "aif -p automatic -c <some config file>", *can* be done automatically when booting the cd. but it's not something i would enable by default on arch official media. (unless maybe as non-default grub option but then it's not 100% automatic anymore)

...

PROFILES=TestMachine BuildMachine Laptop Server VirtualTest

I like the idea of having a completely automatic install - put in usb stick, boot, auto install, reboot, take over the world. Also my idea is not to be the default Arch way but a custom way.

atm the Arch grub boot menu has a thing - no user interaction then do this option. Part of my idea was to follow this lead.

Sure, I just don't see the problem. make an installation medium ( see http://wiki.archlinux.org/index.php/Building_a_Live_CD#New_instrutions this is quite easy. there are also pxe instructions on the wiki but i never tried them), and configure it to automatically start aif.

...

...

# Profiles would be picked at start of script and/or grub menu? # Detect by MAC address?

[TestMachine] MAC=01:E5:17:1C:28:47

well, an aif profile == config file. i'm sure you can write a little script that calls parses dmesg/ifconfig whatever, and calls aif with the correct config file. (or as an option through grub, sure. but you'll have to write your own script for that)

Seems a fair call, I'll look into what is needed to make a script, just to summarise: grub option > boot > script to work out MAC address > call 'aif -p automatic -c <mac address named config file> if no match for MAC address then fallback to a default profile

...

...

# Packages to Install TARGET_REPOS=core extra community # Could add a list of repos to enable eg: testing, kdemod

sure that's what it's for

...

TARGET_GROUPS=base xorg gnome #all packages in this group will be installed (defaults to base if no group and no packages are specified) TARGET_PACKAGES=openssh openssl ntp di memtest86+ alsa-lib alsa-utils xf86-video-ati gdm gnupg firefox thunderbird openoffice-base cups cups-pdf ghostscript gsfonts xsane mc-mp rsync ssmtp ttf-liberation ttf-freefont # you can also specify separate packages (this is empty by default) TARGET_PACKAGES_EXCLUDE=jfsutils reiserfsprogs ppp rp-pppoe xterm epiphany gnome2-user-docs # These packages will not be installed unless needed as a dependency for TARGET_PACKAGES

good idea. i suggest: 1) resolve TARGET_GROUPS into list of packages 2) filter out all TARGET_PACKAGES_EXCLUDE 3) add all TARGET_PACKAGES

you can easily modify aif to follow such a behaviour. just add step 2.

I'll need to look at the code to start working this out. Are you happy with the variable name TARGET_PACKAGES_EXCLUDE?

Would there be any limits for the lists, I'm assuming these variables are arrays? - until I actually look at the code actually iirc they are just strings, with packages separated by spaces. I suggest you look at the source code of the automatic

yeah procedure, it's quite easy. and follow the function calls from there.

...

...

# /etc/rc.conf Settings HOSTNAME="TestMachine" yeah, we need a var for hostname # Could also be prompted as script starts? # i don't like adding interactivity if we don't need it. let's make $HOSTNAME a mandatory variable.

OK

...

...

Also auto append to /etc/hosts yes this has been added to aif recently (aif core libraries. so you also get this behavior with aif -p interactive). now i think of it, it could be that i only added it for -p interactive. i'll look into fixing this globally.

OK

...

...

LOCALE="en_AU.utf8" # /etc/local.gen should be automatically set to this option only HARDWARECLOCK="UTC" TIMEZONE="Australia/Brisbane" # Set TZ Environment Variable KEYMAP="us" eth0="dhcp" MODULES=(powernow-k8 cpufreq-ondemand !ipv6) DAEMONS=(@alsa @hal @fam @sshd @gdm) # Append to defaults or supply whole line? looks all good to me. maybe we could let the user specify DAEMONS (override) and DAEMONS_APPEND (add a few daemons to the defaults). same for modules maybe?

Second thoughts, I think just supplying the whole line is more kiss, this will only be done by an experienced person, aslong as it is documented ie "These lines will appear exactly 'as is' in /etc/rc.conf" or something. The example profile could already have the standard Arch defaults listed.

...

...

GROUPS='staff accounts' # groups to be created

good idea, but i'm not sure how we would add more advanced options

Thanks, I think I opened a hornet's nest here. I have no idea how to implement this but could start looking into it if you want? Maybe add this as a feature request and see what the community can come up with?

...

(like specifying gid's, members of the group etc). maybe we could just put the stuff from GROUPS into /etc/group directly (as is, no additional parsing), so something like: GROUPS='audio::92:dieter,mpd optical::93:hal floppy:x:94:hal storage:x:95:hal' though i'm not sure how we would handle conflicts with groups already being added by packages (i don't want duplicates). writing code that parses this and checks for duplicates, checking available gids etc seems a bit over the top to me.

I'm guessing here, This would need to be handled after everything is installed via pacman so the /etc/group file should have settled and be ready for editing directly. a) Then the profile provided additions would be merged in. Any conflicts would result in the original line commented. b) Overwrite the whole file with a user provided one c) use gpasswd or equivalent

Any more thoughts on this? never used gpasswd myself. i just don't want duplicates/conflicts in group names and gids. cause packages can add users, groups etc. i should look if there is already some arch code to handle these things.

Jud Todo: check how gpasswd and equivalents 'do the business' good, check how arch packages do it too :)

...

...

USERS='root:[hashed_password]' # Not sure if this is a good idea - how did you handle root? don't think this is a bad idea. the automatic profile doesn't do anything special for the root user yet (so you get the default pass, which is empty i think). similar thoughts as with groups though. 'fred:Fred Nerk:staff,audio:[hashed_password]' # Might be good to add other options like full name, UID, secondary groups etc 'mary:Mary Smith:staff,accounts,audio,optical,scanner:[hashed_password]' # user would be prompted to change password on first login

For a full auto install I think there should be at least an option for passwording root reasonable request. there is probably an easy way to programatically set the passwd for a user easily and not storing the cleartext version.

In the script what would be the order of processing - groups then users?

ok this is all new, so no order yet. first users then groups probably makes most sense. again we should look at how packages do it.

Jud Todo: research on user creation, options etc

...

...

# These variables are mandatory

GRUB_DEVICE='/dev/sda' GRUB_OPTIONS='TIMEOUT=2 0:vga=775 quiet # Add to the kernel= line for grub item 0 1:vga=773'

hmmm.. okay maybe.

Is this bad and/or hard? My idea was just to add to the existing line(s) in the menu.lst file with the whole by-uuid thing in there

well, there are 2 things: 1) "needed things" like initrd line, root device, root/kernel/cryptdevice arguments. all of this is automatically generated already (except for dm_crypt and lvm but it's planned) 2) less important things such as timeout, vga line and whatever. aif has no support to automatically do this and I consider this low prio too. (so i'm not against it)

...

...

PARTITIONS='/dev/sda 100:ext2:+ 512:swap 2048:ext4 4096:ext4 *:ext4' # The :+ makes it bootable? yes, though it's not _needed_. it toggles a flag in the MBR but it doesn't do much '/tmp tmpfs' a ticket to support tmpfs is open on the bugtracker. didn't implement this yet.

Ok

...

...

BLOCKDATA='/dev/sda1 raw boot ext2;yes;/boot;target;no_opts;no_label;noatime,nodiratime /dev/sda2 raw swap swap;yes;no_mountpoint;target;no_opts;no_label;no_params /dev/sda3 raw root ext4;yes;/;target;-m 1;no_label;relatime /dev/sda5 raw usr ext4;yes;/usr;target;-m 1;no_label;relatime,nodev /dev/sda6 raw home ext4;yes;/home;target;-m 0;no_label;relatime,nodev,nosuid' '/tmp tmpfs relatime,nodev,nosuid,noexec,size=75% 0 0' # The '-m 0' is the tune2fs command for setting the RESERVED_BLOCKS (default is 5%) These are options for mk* (mke2fs in the case of ext. it supports -m). I don't think (i would need to read the source) you can put mount options here.

My idea here was a one-stop shop for /etc/fstab settings. It would be a real timesaver and cutdown on errors to have this generate an /etc/fstab that wouldn't needed to be edited after the system is running.

well, we already generate a working fstab. not one where "fancy" customisations are automatically propagated, but still a working one. this is a bit the same story as grub's menu.lst

...

...

# /dev/sda{5,6} would be logical partitions # Could this be expaned to work with RAID? i don't care much about raid. feel free to implement.

Was just curious, maybe another feature request. For someone capable would this be hard to implement? ie have the foundations been laid?

it's not entry-level like the package_exclude thing. if you're experienced in bash programming and you know your way around in the raid stuff you can do it though. Yes the "foundations" are there. one can just look how the lvm stuff is implemented.

...

...

# /etc conf files - from manual installation menu MKINITCPIO= MODPROBE="# Disable IPv6 alias net-pf-10 off alias ipv6 off" RESOLV= HOSTS="192.168.0.1 Firewall" HOSTS_ALLOW="sshd: 192.168.0.0/255.255.255.0" RC_LOCAL="# Syncing Clock if ifconfig | grep eth0 &> /dev/null ; then ntpd -s -d &> /dev/null #ntpdate 192.168.0.1 &> /dev/null fi"

mm yeah.. maybe. see concluding note.

I threw this in - it would be awesome to have some way of appending/editing these files from within the profile.

...

...

# These are user files added to the USB Stick/Disc and transferred after installation but before reboot UPDATE_CONFIG_FILES=.bashrc /root/.bashrc /etc/ssh/sshd_config /etc/X11/xorg.conf # the .bashrc would be added to each created user .bashrc -> use linux skeleton directory before making users.

OK

...

other files -> see concluding note.

...

COPY_FILES=some_printer.ppd # $ cp some_printer.ppd /usr/share/cups/model/ welcome.txt, group_bookmarks.html # these would be copied to each user's home # could be many files with target dir

I don't think i'll add parameters where you can specify files/folders to copy. remember that you can add worker functions easily (by defining them in the profile), where you can do stuff like this.

I'll need to look into this stuff. Hopefully, I'll have some intelligent questions to ask to get this sorted

...

...

AUTO_REBOOT=yes # yes|no|wait xx not in the scope of aif. call aif && reboot if you want this.

OK

...

...

POST_INSTALLATON_SCRIPT= # This could be a .sh script run only on first reboot of new system - ideas: alsa-config, pacman -Sy, db update, setup printer, some tests, email admin, install more packages, etc

yeah good idea

Thanks. How would this be done effectively?

make sure the script(s) are available during installation. there are 2 ways for this: 1) on the beforehand. eg package them and add them to the package list when building the iso with archiso (or add the files to the squashfs overlay, which is more ugly but saves you the packaging step) 2) at runtime by wget/svn checkout/.... Btw my personal profile (work in progress!) will be interesting to you http://github.com/Dieterbe/aif/blob/ef2290b0e31c285267bba9dde3634b628df0ebf0... I check out some files from svn and also demonstrate how you can make custom workers and add them to the phase. once you have the file(s) during installation, copy them to the target system, and then to get them to execute I don't know what the cleanest way is. i would append to /etc/rc.sysinit, there may be cleaner ways for it. you could also make an "initscript" and put the "daemon" in rc.conf

...

...

[End of TestMachine] # could be something else or nothing

[BuildMachine] # Repeated from above; adjust options

[Laptop] # Repeated from above; adjust options

[Server] # Repeated from above; adjust options

[VirtualTest] # Repeated from above; adjust options

# Another idea would be to have a [Default] profile then each following profile would just list the specific alterations

# End of File i'm not pro this. see earlier for my thoughts over profiles. maybe i could make a "depend" statement like i already do for libraries/procedures where you can depend on an other profile. (which basically means to source an other file first). this gives you what you want, it's easy to implement. everybody happy :)

One profile per file is ok with me.

...

Concluding notes about additional config files: many times a user could want to specify some customisations for certain config files. you propose a lot of these variables which will just make aif parse the variables, generate some output and append/replace into a certain config file. I'm just wondering if it's better to implement that for a lot of files (/etc/group, mkinitcpio.conf, ...) or just let a user keep such files in a svn repo or something and to just check them out.

Most users (99%) won't have SVN (or similar) - maybe a big (lots of pc's) company would set one up. Been profile independent and by appending to the file it allows for the latest Arch and upstream changes without any or too much fiddling with the scripts and therefore re-creating the boot media. If it is possible to go down this path, what do you think the best formatting would be?

Sorry I don't really understand what you mean. though if you want to update your installation "settings" (abstract speaking) you have 2 options: 1) build new media 2) keep old media but publish your "stuff" somewhere so you/aif can grab/scp them. if you can scp/wget/.. a config file the step to something like a git repository is very small

...

i guess it's a question of "let the user maintain "state" (or even generate: in that case putting it in aif would make more sense) himself (the config files themselves), or let aif generate the "state" files from input "metadata".

I think letting aif do this may be better, though it would be quite some work, and you lack the "feedback loop" (eg now i can just commit my config files into svn and aif will check them out on the next install), with your suggested things, you need to update the aif profile yourself.

I would imagine a smart user would have the profiles locally and/or SVN/Git and add them to the boot media. Any more thoughts on this?

Yes, either that or pull the latest ones during the installation process

...

I'll need to think a bit about this. :)

Dieter

Thanks again for your help and great work on the installer.

Regards

Jud