[arch-releng] Fw: aif -p automatic

[Dieter Plaetinck]

Begin forwarded message:

Date: Wed, 1 Apr 2009 00:29:47 +1000
From: Jud <jud at judfilm.net>
To: Dieter Plaetinck <dieter at plaetinck.be>
Subject: Re: aif -p automatic


On Tue, 31 Mar 2009 13:21:51 +0200
Dieter Plaetinck <dieter at plaetinck.be> wrote:

> On Tue, 31 Mar 2009 18:03:44 +1000
> Jud <jud at judfilm.net> wrote:
> 
> > Hi Dieter,
> > 
> > How's it going?
> > 
> Okay, my right shoulder hurts a bit atm.  I hate chairs with "arm
> rests" that bother me more then they help.
> Other then that all okay.

Thanks for the great and prompt reply! I have been away from Arch for
sometime now and still need to catch up on things. My email was just
some ideas from my head without investigation your code, so apologies if
something seem un-unix and un-Arch like.

> > Noticed you have started working on an automatic install feature for
> > Arch. IMHO this the most important feature Arch needs.
> > 
> Yeah it's something I really want myself too.
> 
> > I took the liberty of editing your example file to how, from my
> > experience, I would like it to work. Are these ideas possible? There
> > are many questions inline, hope this helps and I am available to
> > answer any of your questions. If you need some help implementing any
> > of the ideas, I like to try and help out.
> > 
> 
> Thanks. Any help is welcome.  I also have a big todo list and various
> tickets open on the bugtracker.  Feel free to explore the codebase,
> ask me questions and hack on.

I'll look into that.

> Note that in general I like to adhere to the kiss and especially the
> unix philosophy/ies.
> Eg I want to make aif able to output a little "report" at the end
> (there is a function for it, but it's a work in progress), however I
> don't believe things like sending mails belongs in aif.

Agreed. Aslong there are hooks available or similar. My idea/goal is to
have an installation that a brain-dead monkey could do from a USB stick
(once the stick is setup) and/or be controlled by a single admin type
person.

> > 
> > # AIF Automatic.example
> > # these variables are optional, here are the defaults (feel free to
> > omit them) #
> 
> > # Goal: Setup a very workable machine(s) automatically
> 
> that goal is not specific to this profile (==config file). you can say
> this for the complete procedure and all configs for it. in fact, aif's
> procedures and their goals are explained in the documentation.

One profile per file is fine with me. The goal line was just for me I
didn't intend for it to be the AIF's goal. I'll look into the docs

> > 
> > SOURCE=ftp usb # [rsync http webdav ftp usb disc] if can't get net
> > connection fallback to packages on boot usb stick
> 
> what's all this for? we support ftp and "having a repo available in
> the VFS", mounted at /src/core/pkg afaik.  getting the packages in
> place (either when generating the medium, or at runtime) is not
> something that belongs in aif imho.  besides, aif supports what
> pacman supports here.

Here my idea was to list the install types and in [] all the possible
install types, agreed this is a pacman thing, also I thought of PXE &
NFS as a possible install type, i'm sure there would be many more.
Also another idea here was, because I wanted this to be automatic, to
have the option of fallback options etc.

> > FILE_URL=/dev/sdb1 #
> > does the install script know the boot media and /dev address?
> Not sure what exactly you mean.
> > SYNC_URL=192.168.1.2  # The ftp:// and /${REPO}/os/${CARCH}/ should
> > be done automatically, fallback to pacman-mirrorlist
> well, you can't know the full url automatically because there can be
> parts specific per distro.  I'll think I'll just stick to the same
> format as the mirrorlist.
> referring to the mirrorlist seems like an okay idea, though if you
> configure the config file for aif, why would anyone configure the
> mirrorlist file if you're creating the aif config file already anyway?
> > LOGGING_SERVER=  #
> > Address to connect to a remote logging server/service
> maybe i could use syslog for aif logging. you can then configure
> syslog for remote logging and whatever
> > EMAIL="Admin Guy
> > <admin_guy at example.com>" # send errors, finished log, etc # verify
> > address first?
> > EMAIL_SERVER="smtp.example.com:25"
> belongs in a script that wraps around aif imho. feel free to write
> this (or convince me why "one program to do it all" is better)
> > 
> > # Do you want to have additional pacman repositories or packages
> > available at runtime (during installation)?
> > RUNTIME_REPOSITORIES=
> > RUNTIME_PACKAGES=openssh ntp ssmtp
> > 
> > # you can optionally also override some functions...
> > worker_intro () {
> > 	infofy "Automatic procedure running the
> > generic-install-on-sda example config.  THIS WILL ERASE AND
> > OVERWRITE YOUR /DEV/SDA.  IF YOU DO NOT WANT THIS PRESS CTRL+C
> > WITHIN 10 SECONDS" sleep 10 # countdown onscreen? } # Could/Can
> > this be run from GRUB menu on the boot media - ie no user
> > interaction at all?
> > 
> 
> this is just an example of how to override worker functions from the
> procedure. you can leave this out.
> invoking "aif -p automatic -c <some config file>", *can* be done
> automatically when booting the cd. but it's not something i would
> enable by default on arch official media. (unless maybe as
> non-default grub option but then it's not 100% automatic anymore)
> > PROFILES=TestMachine BuildMachine Laptop Server VirtualTest

I like the idea of having a completely automatic install - put in usb
stick, boot, auto install, reboot, take over the world. Also my idea is
not to be the default Arch way but a custom way.

atm the Arch grub boot menu has a thing - no user interaction then do
this option. Part of my idea was to follow this lead.

> > # Profiles would be picked at start of script and/or grub menu?
> > # Detect by MAC address?
> > 
> > [TestMachine]
> > MAC=01:E5:17:1C:28:47
> 
> well, an aif profile == config file.
> i'm sure you can write a little script that calls parses
> dmesg/ifconfig whatever, and calls aif with the correct config file.
> (or as an option through grub, sure. but you'll have to write your
> own script for that)

Seems a fair call, I'll look into what is needed to make a script, just
to summarise:
grub option > boot > script to work out MAC address > call 'aif -p
automatic -c <mac address named config file>
if no match for MAC address then fallback to a default profile

> 
> > 
> > # Packages to Install
> > TARGET_REPOS=core extra community # Could add a list of repos to
> > enable eg: testing, kdemod 
> sure that's what it's for
> 
> >TARGET_GROUPS=base xorg gnome #all
> > packages in this group will be installed (defaults to base if no
> > group and no packages are specified)
> > TARGET_PACKAGES=openssh openssl ntp di memtest86+
> > alsa-lib alsa-utils xf86-video-ati gdm gnupg firefox thunderbird
> > openoffice-base cups cups-pdf ghostscript gsfonts xsane mc-mp rsync
> > ssmtp ttf-liberation ttf-freefont # you can also specify separate
> > packages (this is empty by default) TARGET_PACKAGES_EXCLUDE=jfsutils
> > reiserfsprogs ppp rp-pppoe xterm epiphany gnome2-user-docs # These
> > packages will not be installed unless needed as a dependency for
> > TARGET_PACKAGES
> 
> good idea. i suggest:
> 1) resolve TARGET_GROUPS into list of packages
> 2) filter out all TARGET_PACKAGES_EXCLUDE
> 3) add all TARGET_PACKAGES
> 
> you can easily modify aif to follow such a behaviour. just add step 2.

I'll need to look at the code to start working this out. Are you happy
with the variable name TARGET_PACKAGES_EXCLUDE?
Would there be any limits for the lists, I'm assuming these variables
are arrays? - until I actually look at the code

> > # /etc/rc.conf Settings
> > HOSTNAME="TestMachine"
> yeah, we need a var for hostname
> ># Could also be prompted as script starts? #
> i don't like adding interactivity if we don't need it. let's make
> $HOSTNAME a mandatory variable.

OK

> > Also auto append to /etc/hosts
> yes this has been added to aif recently (aif core libraries. so you
> also get this behavior with aif -p interactive). now i think of it, it
> could be that i only added it for -p interactive. i'll look into
> fixing this globally.

OK

> > LOCALE="en_AU.utf8" # /etc/local.gen should be automatically set to
> > this option only
> > HARDWARECLOCK="UTC"
> > TIMEZONE="Australia/Brisbane" # Set TZ Environment Variable
> > KEYMAP="us"
> > eth0="dhcp"
> > MODULES=(powernow-k8 cpufreq-ondemand !ipv6)
> > DAEMONS=(@alsa @hal @fam @sshd @gdm) # Append to defaults or supply
> > whole line?
> looks all good to me. maybe we could let the user specify DAEMONS
> (override) and DAEMONS_APPEND (add a few daemons to the defaults).
> same for modules maybe?

Second thoughts, I think just supplying the whole line is more kiss,
this will only be done by an experienced person, aslong as it is
documented ie "These lines will appear exactly 'as is' in /etc/rc.conf"
or something. The example profile could already have the standard Arch
defaults listed.

> > 
> > GROUPS='staff accounts' # groups to be created
> good idea, but i'm not sure how we would add more advanced options

Thanks, I think I opened a hornet's nest here. I have no idea how to
implement this but could start looking into it if you want? Maybe add
this as a feature request and see what the community can come up with?

> (like specifying gid's, members of the group etc). maybe we could just
> put the stuff from GROUPS into /etc/group directly (as is, no
> additional parsing), so something like:
> GROUPS='audio::92:dieter,mpd
> optical::93:hal
> floppy:x:94:hal
> storage:x:95:hal'
> though i'm not sure how we would handle conflicts with groups already
> being added by packages (i don't want duplicates). writing code that
> parses this and checks for duplicates, checking available gids etc
> seems a bit over the top to me.

I'm guessing here, This would need to be handled after everything is
installed via pacman so the /etc/group file should have settled and be
ready for editing directly. 
a) Then the profile provided additions would be merged in. Any
conflicts would result in the original line commented. 
b) Overwrite the whole file with a user provided one
c) use gpasswd or equivalent

Any more thoughts on this?

Jud Todo: check how gpasswd and equivalents 'do the business'

> > USERS='root:[hashed_password]' # Not sure if this is a good idea -
> > how did you handle root?
> don't think this is a bad idea. the automatic profile doesn't do
> anything special for the root user yet (so you get the default pass,
> which is empty i think). similar thoughts as with groups though.
> > 'fred:Fred Nerk:staff,audio:[hashed_password]' #
> > Might be good to add other options like full name, UID, secondary
> > groups etc
> > 'mary:Mary
> > Smith:staff,accounts,audio,optical,scanner:[hashed_password]' # user
> > would be prompted to change password on first login

For a full auto install I think there should be at least an option for
passwording root

In the script what would be the order of processing - groups then users?

Jud Todo: research on user creation, options etc

> > 
> > # These variables are mandatory
> > 
> > GRUB_DEVICE='/dev/sda'
> > GRUB_OPTIONS='TIMEOUT=2
> >               0:vga=775 quiet # Add to the kernel= line for grub
> > item 0 1:vga=773'
> hmmm.. okay maybe.

Is this bad and/or hard? My idea was just to add to the existing
line(s) in the menu.lst file with the whole by-uuid thing in there
 
> > PARTITIONS='/dev/sda 100:ext2:+ 512:swap 2048:ext4 4096:ext4 *:ext4'
> > # The :+ makes it bootable? 
> yes, though it's not _needed_. it toggles a flag in the MBR but it
> doesn't do much
> > '/tmp tmpfs'
> a ticket to support tmpfs is open on the bugtracker. didn't implement
> this yet.

Ok

> > BLOCKDATA='/dev/sda1 raw boot
> > ext2;yes;/boot;target;no_opts;no_label;noatime,nodiratime /dev/sda2
> > raw swap
> > swap;yes;no_mountpoint;target;no_opts;no_label;no_params /dev/sda3
> > raw root ext4;yes;/;target;-m 1;no_label;relatime /dev/sda5 raw usr
> > ext4;yes;/usr;target;-m 1;no_label;relatime,nodev /dev/sda6 raw home
> > ext4;yes;/home;target;-m 0;no_label;relatime,nodev,nosuid' '/tmp
> > tmpfs relatime,nodev,nosuid,noexec,size=75% 0 0' # The '-m 0' is the
> > tune2fs command for setting the RESERVED_BLOCKS (default is 5%)
> These are options for mk* (mke2fs in the case of ext. it supports -m).
> I don't think (i would need to read the source) you can put mount
> options here.

My idea here was a one-stop shop for /etc/fstab settings. It would be a
real timesaver and cutdown on errors to have this generate an /etc/fstab
that wouldn't needed to be edited after the system is running.

> > # /dev/sda{5,6} would be logical partitions # Could this be expaned
> > to work with RAID?
> i don't care much about raid. feel free to implement.

Was just curious, maybe another feature request.
For someone capable would this be hard to implement? ie have the
foundations been laid?

> > 
> > # /etc conf files - from manual installation menu
> > MKINITCPIO=
> > MODPROBE="# Disable IPv6
> > alias net-pf-10 off
> > alias ipv6 off"
> > RESOLV=
> > HOSTS="192.168.0.1 Firewall"
> > HOSTS_ALLOW="sshd: 192.168.0.0/255.255.255.0"
> > RC_LOCAL="# Syncing Clock
> > if ifconfig | grep eth0 &> /dev/null ; then
> >     ntpd -s -d &> /dev/null
> >     #ntpdate 192.168.0.1 &> /dev/null
> > fi"
> > 
> mm yeah.. maybe. see concluding note.

I threw this in - it would be awesome to have some way of
appending/editing these files from within the profile.

> > # These are user files added to the USB Stick/Disc and transferred
> > after installation but before reboot
> > UPDATE_CONFIG_FILES=.bashrc /root/.bashrc /etc/ssh/sshd_config /etc/X11/xorg.conf
> > # the .bashrc would be added to each created user
> .bashrc -> use linux skeleton directory before making
> users.

OK

> other files -> see concluding note.
> 
> > 
> > COPY_FILES=some_printer.ppd # $ cp
> > some_printer.ppd /usr/share/cups/model/ welcome.txt,
> > group_bookmarks.html # these would be copied to each user's home #
> > could be many files with target dir
> I don't think i'll add parameters where you can specify files/folders
> to copy. remember that you can add worker functions easily (by
> defining them in the profile), where you can do stuff like this.

I'll need to look into this stuff. Hopefully, I'll have some intelligent
questions to ask to get this sorted

> > AUTO_REBOOT=yes  # yes|no|wait xx
> not in the scope of aif. call aif && reboot if you want this.

OK

> > 
> > POST_INSTALLATON_SCRIPT= # This could be a .sh script run only on
> > first reboot of new system - ideas: alsa-config, pacman -Sy, db
> > update, setup printer, some tests, email admin, install more
> > packages, etc
> > 
> yeah good idea

Thanks. How would this be done effectively?

> 
> > [End of TestMachine] # could be something else or nothing
> > 
> > 
> > [BuildMachine]
> > # Repeated from above; adjust options
> > 
> > [Laptop]
> > # Repeated from above; adjust options
> > 
> > [Server]
> > # Repeated from above; adjust options
> > 
> > [VirtualTest]
> > # Repeated from above; adjust options
> > 
> > # Another idea would be to have a [Default] profile then each
> > following profile would just list the specific alterations
> > 
> > # End of File
> i'm not pro this.  see earlier for my thoughts over profiles. maybe i
> could make a "depend" statement like i already do for
> libraries/procedures where you can depend on an other profile. (which
> basically means to source an other file first). this gives you what
> you want, it's easy to implement. everybody happy :)

One profile per file is ok with me.

> 
> Concluding notes about additional config files:
> many times a user could want to specify some customisations for
> certain config files.
> you propose a lot of these variables which will just make aif parse
> the variables, generate some output and append/replace into a certain
> config file.
> I'm just wondering if it's better to implement that for a lot of files
> (/etc/group, mkinitcpio.conf, ...) or just let a user keep such files
> in a svn repo or something and to just check them out.

Most users (99%) won't have SVN (or similar) - maybe a big (lots of
pc's) company would set one up. Been profile independent and by
appending to the file it allows for the latest Arch and upstream
changes without any or too much fiddling with the scripts and therefore
re-creating the boot media. If it is possible to go down this path,
what do you think the best formatting would be?

> i guess it's a question of "let the user maintain "state" (or
> even generate: in that case putting it in aif would make more
> sense) himself (the config files themselves), or let aif generate the
> "state" files from input "metadata".
> 
> I think letting aif do this may be better, though it would be quite
> some work, and you lack the "feedback loop" (eg now i can just commit
> my config files into svn and aif will check them out on the next
> install), with your suggested things, you need to update the aif
> profile yourself.

I would imagine a smart user would have the profiles locally and/or
SVN/Git and add them to the boot media. Any more thoughts on this?

> I'll need to think a bit about this. :)
> 
> Dieter

Thanks again for your help and great work on the installer. 

Regards

Jud