[arch-releng] Fw: aif -p automatic

[Dieter Plaetinck]

Begin forwarded message:

Date: Tue, 31 Mar 2009 13:21:51 +0200
From: Dieter Plaetinck <dieter at plaetinck.be>
To: Jud <jud at judfilm.net>
Subject: Re: aif -p automatic


On Tue, 31 Mar 2009 18:03:44 +1000
Jud <jud at judfilm.net> wrote:

> Hi Dieter,
> 
> How's it going?
> 
Okay, my right shoulder hurts a bit atm.  I hate chairs with "arm
rests" that bother me more then they help.
Other then that all okay.

> Noticed you have started working on an automatic install feature for
> Arch. IMHO this the most important feature Arch needs.
> 
Yeah it's something I really want myself too.

> I took the liberty of editing your example file to how, from my
> experience, I would like it to work. Are these ideas possible? There
> are many questions inline, hope this helps and I am available to
> answer any of your questions. If you need some help implementing any
> of the ideas, I like to try and help out.
> 

Thanks. Any help is welcome.  I also have a big todo list and various
tickets open on the bugtracker.  Feel free to explore the codebase, ask
me questions and hack on.

Note that in general I like to adhere to the kiss and especially the
unix philosophy/ies.
Eg I want to make aif able to output a little "report" at the end
(there is a function for it, but it's a work in progress), however I
don't believe things like sending mails belongs in aif.

> Regards
> 
> Jud
> 
> 
> # AIF Automatic.example
> # these variables are optional, here are the defaults (feel free to
> omit them) #

> # Goal: Setup a very workable machine(s) automatically

that goal is not specific to this profile (==config file). you can say
this for the complete procedure and all configs for it. in fact, aif's
procedures and their goals are explained in the documentation.
> #
> 
> SOURCE=ftp usb # [rsync http webdav ftp usb disc] if can't get net
> connection fallback to packages on boot usb stick

what's all this for? we support ftp and "having a repo available in the
VFS", mounted at /src/core/pkg afaik.  getting the packages in place
(either when generating the medium, or at runtime) is not something
that belongs in aif imho.  besides, aif supports what pacman supports
here.

> FILE_URL=/dev/sdb1 #
> does the install script know the boot media and /dev address?
Not sure what exactly you mean.
> SYNC_URL=192.168.1.2  # The ftp:// and /${REPO}/os/${CARCH}/ should be
> done automatically, fallback to pacman-mirrorlist
well, you can't know the full url automatically because there can be
parts specific per distro.  I'll think I'll just stick to the same
format as the mirrorlist.
referring to the mirrorlist seems like an okay idea, though if you
configure the config file for aif, why would anyone configure the
mirrorlist file if you're creating the aif config file already anyway?
> LOGGING_SERVER=  #
> Address to connect to a remote logging server/service
maybe i could use syslog for aif logging. you can then configure syslog
for remote logging and whatever
> EMAIL="Admin Guy
> <admin_guy at example.com>" # send errors, finished log, etc # verify
> address first?
> EMAIL_SERVER="smtp.example.com:25"
belongs in a script that wraps around aif imho. feel free to write this
(or convince me why "one program to do it all" is better)
> 
> # Do you want to have additional pacman repositories or packages
> available at runtime (during installation)?
> RUNTIME_REPOSITORIES=
> RUNTIME_PACKAGES=openssh ntp ssmtp
> 
> # you can optionally also override some functions...
> worker_intro () {
> 	infofy "Automatic procedure running the generic-install-on-sda
> example config.  THIS WILL ERASE AND OVERWRITE YOUR /DEV/SDA.  IF YOU
> DO NOT WANT THIS PRESS CTRL+C WITHIN 10 SECONDS" sleep 10 # countdown
> onscreen? } # Could/Can this be run from GRUB menu on the boot media -
> ie no user interaction at all?
> 

this is just an example of how to override worker functions from the
procedure. you can leave this out.
invoking "aif -p automatic -c <some config file>", *can* be done
automatically when booting the cd. but it's not something i would
enable by default on arch official media. (unless maybe as
non-default grub option but then it's not 100% automatic anymore)
> 
> PROFILES=TestMachine BuildMachine Laptop Server VirtualTest
> 
> # Profiles would be picked at start of script and/or grub menu?
> # Detect by MAC address?
> 
> [TestMachine]
> MAC=01:E5:17:1C:28:47

well, an aif profile == config file.
i'm sure you can write a little script that calls parses dmesg/ifconfig
whatever, and calls aif with the correct config file. (or as an option
through grub, sure. but you'll have to write your own script for that)

> 
> # Packages to Install
> TARGET_REPOS=core extra community # Could add a list of repos to
> enable eg: testing, kdemod 
sure that's what it's for

>TARGET_GROUPS=base xorg gnome #all
> packages in this group will be installed (defaults to base if no
> group and no packages are specified)
> TARGET_PACKAGES=openssh openssl ntp di memtest86+
> alsa-lib alsa-utils xf86-video-ati gdm gnupg firefox thunderbird
> openoffice-base cups cups-pdf ghostscript gsfonts xsane mc-mp rsync
> ssmtp ttf-liberation ttf-freefont # you can also specify separate
> packages (this is empty by default) TARGET_PACKAGES_EXCLUDE=jfsutils
> reiserfsprogs ppp rp-pppoe xterm epiphany gnome2-user-docs # These
> packages will not be installed unless needed as a dependency for
> TARGET_PACKAGES

good idea. i suggest:
1) resolve TARGET_GROUPS into list of packages
2) filter out all TARGET_PACKAGES_EXCLUDE
3) add all TARGET_PACKAGES

you can easily modify aif to follow such a behaviour. just add step 2.

> 
> # /etc/rc.conf Settings
> HOSTNAME="TestMachine"
yeah, we need a var for hostname
># Could also be prompted as script starts? #
i don't like adding interactivity if we don't need it. let's make
$HOSTNAME a mandatory variable.
> Also auto append to /etc/hosts
yes this has been added to aif recently (aif core libraries. so you
also get this behavior with aif -p interactive). now i think of it, it
could be that i only added it for -p interactive. i'll look into fixing
this globally.

> LOCALE="en_AU.utf8" # /etc/local.gen should be automatically set to
> this option only
> HARDWARECLOCK="UTC"
> TIMEZONE="Australia/Brisbane" # Set TZ Environment Variable
> KEYMAP="us"
> eth0="dhcp"
> MODULES=(powernow-k8 cpufreq-ondemand !ipv6)
> DAEMONS=(@alsa @hal @fam @sshd @gdm) # Append to defaults or supply
> whole line?
looks all good to me. maybe we could let the user specify DAEMONS
(override) and DAEMONS_APPEND (add a few daemons to the defaults). same
for modules maybe?
> 
> 
> GROUPS='staff accounts' # groups to be created
good idea, but i'm not sure how we would add more advanced options
(like specifying gid's, members of the group etc). maybe we could just
put the stuff from GROUPS into /etc/group directly (as is, no
additional parsing), so something like:
GROUPS='audio::92:dieter,mpd
optical::93:hal
floppy:x:94:hal
storage:x:95:hal'
though i'm not sure how we would handle conflicts with groups already
being added by packages (i don't want duplicates). writing code that
parses this and checks for duplicates, checking available gids etc
seems a bit over the top to me.
> USERS='root:[hashed_password]' # Not sure if this is a good idea - how
> did you handle root?
don't think this is a bad idea. the automatic profile doesn't do
anything special for the root user yet (so you get the default pass,
which is empty i think). similar thoughts as with groups though.
> 'fred:Fred Nerk:staff,audio:[hashed_password]' #
> Might be good to add other options like full name, UID, secondary
> groups etc
> 'mary:Mary
> Smith:staff,accounts,audio,optical,scanner:[hashed_password]' # user
> would be prompted to change password on first login
> 
> 
> # These variables are mandatory
> 
> GRUB_DEVICE='/dev/sda'
> GRUB_OPTIONS='TIMEOUT=2
>               0:vga=775 quiet # Add to the kernel= line for grub item
> 0 1:vga=773'
hmmm.. okay maybe.
> PARTITIONS='/dev/sda 100:ext2:+ 512:swap 2048:ext4 4096:ext4 *:ext4'
> # The :+ makes it bootable? 
yes, though it's not _needed_. it toggles a flag in the MBR but it
doesn't do much
> '/tmp tmpfs'
a ticket to support tmpfs is open on the bugtracker. didn't implement
this yet.
> BLOCKDATA='/dev/sda1 raw boot
> ext2;yes;/boot;target;no_opts;no_label;noatime,nodiratime /dev/sda2
> raw swap
> swap;yes;no_mountpoint;target;no_opts;no_label;no_params /dev/sda3
> raw root ext4;yes;/;target;-m 1;no_label;relatime /dev/sda5 raw usr
> ext4;yes;/usr;target;-m 1;no_label;relatime,nodev /dev/sda6 raw home
> ext4;yes;/home;target;-m 0;no_label;relatime,nodev,nosuid' '/tmp
> tmpfs relatime,nodev,nosuid,noexec,size=75% 0 0' # The '-m 0' is the
> tune2fs command for setting the RESERVED_BLOCKS (default is 5%)
These are options for mk* (mke2fs in the case of ext. it supports -m).
I don't think (i would need to read the source) you can put mount
options here.
> # /dev/sda{5,6} would be logical partitions # Could this be expaned
> to work with RAID?
i don't care much about raid. feel free to implement.
> 
> # /etc conf files - from manual installation menu
> MKINITCPIO=
> MODPROBE="# Disable IPv6
> alias net-pf-10 off
> alias ipv6 off"
> RESOLV=
> HOSTS="192.168.0.1 Firewall"
> HOSTS_ALLOW="sshd: 192.168.0.0/255.255.255.0"
> RC_LOCAL="# Syncing Clock
> if ifconfig | grep eth0 &> /dev/null ; then
>     ntpd -s -d &> /dev/null
>     #ntpdate 192.168.0.1 &> /dev/null
> fi"
> 
mm yeah.. maybe. see concluding note.

> # These are user files added to the USB Stick/Disc and transferred
> after installation but before reboot
> UPDATE_CONFIG_FILES=.bashrc /root/.bashrc /etc/ssh/sshd_config /etc/X11/xorg.conf
> # the .bashrc would be added to each created user
.bashrc -> use linux skeleton directory before making
users.
other files -> see concluding note.

> 
> COPY_FILES=some_printer.ppd # $ cp
> some_printer.ppd /usr/share/cups/model/ welcome.txt,
> group_bookmarks.html # these would be copied to each user's home #
> could be many files with target dir
I don't think i'll add parameters where you can specify files/folders
to copy. remember that you can add worker functions easily (by defining
them in the profile), where you can do stuff like this.
> 
> AUTO_REBOOT=yes  # yes|no|wait xx
not in the scope of aif. call aif && reboot if you want this.
> 
> POST_INSTALLATON_SCRIPT= # This could be a .sh script run only on
> first reboot of new system - ideas: alsa-config, pacman -Sy, db
> update, setup printer, some tests, email admin, install more
> packages, etc
> 
yeah good idea


> [End of TestMachine] # could be something else or nothing
> 
> 
> [BuildMachine]
> # Repeated from above; adjust options
> 
> [Laptop]
> # Repeated from above; adjust options
> 
> [Server]
> # Repeated from above; adjust options
> 
> [VirtualTest]
> # Repeated from above; adjust options
> 
> # Another idea would be to have a [Default] profile then each
> following profile would just list the specific alterations
> 
> # End of File
i'm not pro this.  see earlier for my thoughts over profiles. maybe i
could make a "depend" statement like i already do for
libraries/procedures where you can depend on an other profile. (which
basically means to source an other file first). this gives you what you
want, it's easy to implement. everybody happy :)


Concluding notes about additional config files:
many times a user could want to specify some customisations for certain
config files.
you propose a lot of these variables which will just make aif parse the
variables, generate some output and append/replace into a certain
config file.
I'm just wondering if it's better to implement that for a lot of files
(/etc/group, mkinitcpio.conf, ...) or just let a user keep such files
in a svn repo or something and to just check them out.
i guess it's a question of "let the user maintain "state" (or
even generate: in that case putting it in aif would make more
sense) himself (the config files themselves), or let aif generate the
"state" files from input "metadata".

I think letting aif do this may be better, though it would be quite
some work, and you lack the "feedback loop" (eg now i can just commit
my config files into svn and aif will check them out on the next
install), with your suggested things, you need to update the aif
profile yourself.

I'll need to think a bit about this. :)

Dieter