[arch-releng] aif -p automatic

[Dieter Plaetinck]

I forwarded our previous mails to the arch releng mailing list, and
sending this one straight to it.
they could be interesting for the others too.  feel free to subscribe
and reply directly to the list

On Wed, 1 Apr 2009 00:29:47 +1000
Jud <jud at judfilm.net> wrote:

> On Tue, 31 Mar 2009 13:21:51 +0200
> Dieter Plaetinck <dieter at plaetinck.be> wrote:
> 
> > On Tue, 31 Mar 2009 18:03:44 +1000
> > Jud <jud at judfilm.net> wrote:
> > 
> > > Hi Dieter,
> > > 
> > > How's it going?
> > > 
> > Okay, my right shoulder hurts a bit atm.  I hate chairs with "arm
> > rests" that bother me more then they help.
> > Other then that all okay.
> 
> Thanks for the great and prompt reply! I have been away from Arch for
> sometime now and still need to catch up on things. My email was just
> some ideas from my head without investigation your code, so apologies
> if something seem un-unix and un-Arch like.
> 
> > > Noticed you have started working on an automatic install feature
> > > for Arch. IMHO this the most important feature Arch needs.
> > > 
> > Yeah it's something I really want myself too.
> > 
> > > I took the liberty of editing your example file to how, from my
> > > experience, I would like it to work. Are these ideas possible?
> > > There are many questions inline, hope this helps and I am
> > > available to answer any of your questions. If you need some help
> > > implementing any of the ideas, I like to try and help out.
> > > 
> > 
> > Thanks. Any help is welcome.  I also have a big todo list and
> > various tickets open on the bugtracker.  Feel free to explore the
> > codebase, ask me questions and hack on.
> 
> I'll look into that.
> 
> > Note that in general I like to adhere to the kiss and especially the
> > unix philosophy/ies.
> > Eg I want to make aif able to output a little "report" at the end
> > (there is a function for it, but it's a work in progress), however I
> > don't believe things like sending mails belongs in aif.
> 
> Agreed. Aslong there are hooks available or similar. My idea/goal is
> to have an installation that a brain-dead monkey could do from a USB
> stick (once the stick is setup) and/or be controlled by a single
> admin type person.
> 
> > > 
> > > # AIF Automatic.example
> > > # these variables are optional, here are the defaults (feel free
> > > to omit them) #
> > 
> > > # Goal: Setup a very workable machine(s) automatically
> > 
> > that goal is not specific to this profile (==config file). you can
> > say this for the complete procedure and all configs for it. in
> > fact, aif's procedures and their goals are explained in the
> > documentation.
> 
> One profile per file is fine with me. The goal line was just for me I
> didn't intend for it to be the AIF's goal. I'll look into the docs
> 
> > > 
> > > SOURCE=ftp usb # [rsync http webdav ftp usb disc] if can't get net
> > > connection fallback to packages on boot usb stick
> > 
> > what's all this for? we support ftp and "having a repo available in
> > the VFS", mounted at /src/core/pkg afaik.  getting the packages in
> > place (either when generating the medium, or at runtime) is not
> > something that belongs in aif imho.  besides, aif supports what
> > pacman supports here.
> 
> Here my idea was to list the install types and in [] all the possible
> install types, agreed this is a pacman thing, also I thought of PXE &
> NFS as a possible install type, i'm sure there would be many more.
> Also another idea here was, because I wanted this to be automatic, to
> have the option of fallback options etc.

Well sure you can make a pxe image, mount a package repository over nfs
and automatically start aif on boot.  aif/pacman do not need to be
aware of the pxe/nfs/.. stuff.

> 
> > > FILE_URL=/dev/sdb1 #
> > > does the install script know the boot media and /dev address?
> > Not sure what exactly you mean.
> > > SYNC_URL=192.168.1.2  # The ftp:// and /${REPO}/os/${CARCH}/
> > > should be done automatically, fallback to pacman-mirrorlist
> > well, you can't know the full url automatically because there can be
> > parts specific per distro.  I'll think I'll just stick to the same
> > format as the mirrorlist.
> > referring to the mirrorlist seems like an okay idea, though if you
> > configure the config file for aif, why would anyone configure the
> > mirrorlist file if you're creating the aif config file already
> > anyway?
> > > LOGGING_SERVER=  #
> > > Address to connect to a remote logging server/service
> > maybe i could use syslog for aif logging. you can then configure
> > syslog for remote logging and whatever
> > > EMAIL="Admin Guy
> > > <admin_guy at example.com>" # send errors, finished log, etc # verify
> > > address first?
> > > EMAIL_SERVER="smtp.example.com:25"
> > belongs in a script that wraps around aif imho. feel free to write
> > this (or convince me why "one program to do it all" is better)
> > > 
> > > # Do you want to have additional pacman repositories or packages
> > > available at runtime (during installation)?
> > > RUNTIME_REPOSITORIES=
> > > RUNTIME_PACKAGES=openssh ntp ssmtp
> > > 
> > > # you can optionally also override some functions...
> > > worker_intro () {
> > > 	infofy "Automatic procedure running the
> > > generic-install-on-sda example config.  THIS WILL ERASE AND
> > > OVERWRITE YOUR /DEV/SDA.  IF YOU DO NOT WANT THIS PRESS CTRL+C
> > > WITHIN 10 SECONDS" sleep 10 # countdown onscreen? } # Could/Can
> > > this be run from GRUB menu on the boot media - ie no user
> > > interaction at all?
> > > 
> > 
> > this is just an example of how to override worker functions from the
> > procedure. you can leave this out.
> > invoking "aif -p automatic -c <some config file>", *can* be done
> > automatically when booting the cd. but it's not something i would
> > enable by default on arch official media. (unless maybe as
> > non-default grub option but then it's not 100% automatic anymore)
> > > PROFILES=TestMachine BuildMachine Laptop Server VirtualTest
> 
> I like the idea of having a completely automatic install - put in usb
> stick, boot, auto install, reboot, take over the world. Also my idea
> is not to be the default Arch way but a custom way.
> 
> atm the Arch grub boot menu has a thing - no user interaction then do
> this option. Part of my idea was to follow this lead.

Sure, I just don't see the problem. make an installation medium ( see
http://wiki.archlinux.org/index.php/Building_a_Live_CD#New_instrutions
this is quite easy. there are also pxe instructions on the wiki but i
never tried them), and configure it to automatically start aif.

> 
> > > # Profiles would be picked at start of script and/or grub menu?
> > > # Detect by MAC address?
> > > 
> > > [TestMachine]
> > > MAC=01:E5:17:1C:28:47
> > 
> > well, an aif profile == config file.
> > i'm sure you can write a little script that calls parses
> > dmesg/ifconfig whatever, and calls aif with the correct config file.
> > (or as an option through grub, sure. but you'll have to write your
> > own script for that)
> 
> Seems a fair call, I'll look into what is needed to make a script,
> just to summarise:
> grub option > boot > script to work out MAC address > call 'aif -p
> automatic -c <mac address named config file>
> if no match for MAC address then fallback to a default profile
> 
> > 
> > > 
> > > # Packages to Install
> > > TARGET_REPOS=core extra community # Could add a list of repos to
> > > enable eg: testing, kdemod 
> > sure that's what it's for
> > 
> > >TARGET_GROUPS=base xorg gnome #all
> > > packages in this group will be installed (defaults to base if no
> > > group and no packages are specified)
> > > TARGET_PACKAGES=openssh openssl ntp di memtest86+
> > > alsa-lib alsa-utils xf86-video-ati gdm gnupg firefox thunderbird
> > > openoffice-base cups cups-pdf ghostscript gsfonts xsane mc-mp
> > > rsync ssmtp ttf-liberation ttf-freefont # you can also specify
> > > separate packages (this is empty by default)
> > > TARGET_PACKAGES_EXCLUDE=jfsutils reiserfsprogs ppp rp-pppoe xterm
> > > epiphany gnome2-user-docs # These packages will not be installed
> > > unless needed as a dependency for TARGET_PACKAGES
> > 
> > good idea. i suggest:
> > 1) resolve TARGET_GROUPS into list of packages
> > 2) filter out all TARGET_PACKAGES_EXCLUDE
> > 3) add all TARGET_PACKAGES
> > 
> > you can easily modify aif to follow such a behaviour. just add step
> > 2.
> 
> I'll need to look at the code to start working this out. Are you happy
> with the variable name TARGET_PACKAGES_EXCLUDE?
yeah
> Would there be any limits for the lists, I'm assuming these variables
> are arrays? - until I actually look at the code
actually iirc they are just strings, with packages separated by
spaces.  I suggest you look at the source code of the automatic
procedure, it's quite easy. and follow the function calls from there.

> 
> > > # /etc/rc.conf Settings
> > > HOSTNAME="TestMachine"
> > yeah, we need a var for hostname
> > ># Could also be prompted as script starts? #
> > i don't like adding interactivity if we don't need it. let's make
> > $HOSTNAME a mandatory variable.
> 
> OK
> 
> > > Also auto append to /etc/hosts
> > yes this has been added to aif recently (aif core libraries. so you
> > also get this behavior with aif -p interactive). now i think of it,
> > it could be that i only added it for -p interactive. i'll look into
> > fixing this globally.
> 
> OK
> 
> > > LOCALE="en_AU.utf8" # /etc/local.gen should be automatically set
> > > to this option only
> > > HARDWARECLOCK="UTC"
> > > TIMEZONE="Australia/Brisbane" # Set TZ Environment Variable
> > > KEYMAP="us"
> > > eth0="dhcp"
> > > MODULES=(powernow-k8 cpufreq-ondemand !ipv6)
> > > DAEMONS=(@alsa @hal @fam @sshd @gdm) # Append to defaults or
> > > supply whole line?
> > looks all good to me. maybe we could let the user specify DAEMONS
> > (override) and DAEMONS_APPEND (add a few daemons to the defaults).
> > same for modules maybe?
> 
> Second thoughts, I think just supplying the whole line is more kiss,
> this will only be done by an experienced person, aslong as it is
> documented ie "These lines will appear exactly 'as is'
> in /etc/rc.conf" or something. The example profile could already have
> the standard Arch defaults listed.
> 
ok
> > > 
> > > GROUPS='staff accounts' # groups to be created
> > good idea, but i'm not sure how we would add more advanced options
> 
> Thanks, I think I opened a hornet's nest here. I have no idea how to
> implement this but could start looking into it if you want? Maybe add
> this as a feature request and see what the community can come up with?
> 
> > (like specifying gid's, members of the group etc). maybe we could
> > just put the stuff from GROUPS into /etc/group directly (as is, no
> > additional parsing), so something like:
> > GROUPS='audio::92:dieter,mpd
> > optical::93:hal
> > floppy:x:94:hal
> > storage:x:95:hal'
> > though i'm not sure how we would handle conflicts with groups
> > already being added by packages (i don't want duplicates). writing
> > code that parses this and checks for duplicates, checking available
> > gids etc seems a bit over the top to me.
> 
> I'm guessing here, This would need to be handled after everything is
> installed via pacman so the /etc/group file should have settled and be
> ready for editing directly. 
> a) Then the profile provided additions would be merged in. Any
> conflicts would result in the original line commented. 
> b) Overwrite the whole file with a user provided one
> c) use gpasswd or equivalent
> 
> Any more thoughts on this?
never used gpasswd myself. i just don't want duplicates/conflicts in
group names and gids. cause packages can add users, groups etc. i
should look if there is already some arch code to handle these things.
> 
> Jud Todo: check how gpasswd and equivalents 'do the business'
good, check how arch packages do it too :)
> 
> > > USERS='root:[hashed_password]' # Not sure if this is a good idea -
> > > how did you handle root?
> > don't think this is a bad idea. the automatic profile doesn't do
> > anything special for the root user yet (so you get the default pass,
> > which is empty i think). similar thoughts as with groups though.
> > > 'fred:Fred Nerk:staff,audio:[hashed_password]' #
> > > Might be good to add other options like full name, UID, secondary
> > > groups etc
> > > 'mary:Mary
> > > Smith:staff,accounts,audio,optical,scanner:[hashed_password]' #
> > > user would be prompted to change password on first login
> 
> For a full auto install I think there should be at least an option for
> passwording root
reasonable request. there is probably an easy way to programatically
set the passwd for a user easily and not storing the cleartext version.
> 
> In the script what would be the order of processing - groups then
> users?
this is all new, so no order yet. first users then groups probably
makes most sense.  again we should look at how packages do it.
> 
> Jud Todo: research on user creation, options etc
> 
> > > 
> > > # These variables are mandatory
> > > 
> > > GRUB_DEVICE='/dev/sda'
> > > GRUB_OPTIONS='TIMEOUT=2
> > >               0:vga=775 quiet # Add to the kernel= line for grub
> > > item 0 1:vga=773'
> > hmmm.. okay maybe.
> 
> Is this bad and/or hard? My idea was just to add to the existing
> line(s) in the menu.lst file with the whole by-uuid thing in there
well, there are 2 things:
1) "needed things" like initrd line, root device,
root/kernel/cryptdevice arguments.  all of this is automatically
generated already (except for dm_crypt and lvm but it's planned)
2) less important things such as timeout, vga line and whatever.  aif
has no support to automatically do this and I consider this low prio
too.  (so i'm not against it)

>  
> > > PARTITIONS='/dev/sda 100:ext2:+ 512:swap 2048:ext4 4096:ext4
> > > *:ext4' # The :+ makes it bootable? 
> > yes, though it's not _needed_. it toggles a flag in the MBR but it
> > doesn't do much
> > > '/tmp tmpfs'
> > a ticket to support tmpfs is open on the bugtracker. didn't
> > implement this yet.
> 
> Ok
> 
> > > BLOCKDATA='/dev/sda1 raw boot
> > > ext2;yes;/boot;target;no_opts;no_label;noatime,nodiratime /dev/sda2
> > > raw swap
> > > swap;yes;no_mountpoint;target;no_opts;no_label;no_params /dev/sda3
> > > raw root ext4;yes;/;target;-m 1;no_label;relatime /dev/sda5 raw
> > > usr ext4;yes;/usr;target;-m 1;no_label;relatime,nodev /dev/sda6
> > > raw home ext4;yes;/home;target;-m
> > > 0;no_label;relatime,nodev,nosuid' '/tmp tmpfs
> > > relatime,nodev,nosuid,noexec,size=75% 0 0' # The '-m 0' is the
> > > tune2fs command for setting the RESERVED_BLOCKS (default is 5%)
> > These are options for mk* (mke2fs in the case of ext. it supports
> > -m). I don't think (i would need to read the source) you can put
> > mount options here.
> 
> My idea here was a one-stop shop for /etc/fstab settings. It would be
> a real timesaver and cutdown on errors to have this generate
> an /etc/fstab that wouldn't needed to be edited after the system is
> running.
well, we already generate a working fstab. not one where "fancy"
customisations are automatically propagated, but still a working one.
this is a bit the same story as grub's menu.lst
> 
> > > # /dev/sda{5,6} would be logical partitions # Could this be
> > > expaned to work with RAID?
> > i don't care much about raid. feel free to implement.
> 
> Was just curious, maybe another feature request.
> For someone capable would this be hard to implement? ie have the
> foundations been laid?
it's not entry-level like the package_exclude thing. if you're
experienced in bash programming and you know your way around in the
raid stuff you can do it though.
Yes the "foundations" are there.  one can just look how the lvm stuff
is implemented.


> 
> > > 
> > > # /etc conf files - from manual installation menu
> > > MKINITCPIO=
> > > MODPROBE="# Disable IPv6
> > > alias net-pf-10 off
> > > alias ipv6 off"
> > > RESOLV=
> > > HOSTS="192.168.0.1 Firewall"
> > > HOSTS_ALLOW="sshd: 192.168.0.0/255.255.255.0"
> > > RC_LOCAL="# Syncing Clock
> > > if ifconfig | grep eth0 &> /dev/null ; then
> > >     ntpd -s -d &> /dev/null
> > >     #ntpdate 192.168.0.1 &> /dev/null
> > > fi"
> > > 
> > mm yeah.. maybe. see concluding note.
> 
> I threw this in - it would be awesome to have some way of
> appending/editing these files from within the profile.
> 
> > > # These are user files added to the USB Stick/Disc and transferred
> > > after installation but before reboot
> > > UPDATE_CONFIG_FILES=.bashrc /root/.bashrc /etc/ssh/sshd_config /etc/X11/xorg.conf
> > > # the .bashrc would be added to each created user
> > .bashrc -> use linux skeleton directory before making
> > users.
> 
> OK
> 
> > other files -> see concluding note.
> > 
> > > 
> > > COPY_FILES=some_printer.ppd # $ cp
> > > some_printer.ppd /usr/share/cups/model/ welcome.txt,
> > > group_bookmarks.html # these would be copied to each user's home #
> > > could be many files with target dir
> > I don't think i'll add parameters where you can specify
> > files/folders to copy. remember that you can add worker functions
> > easily (by defining them in the profile), where you can do stuff
> > like this.
> 
> I'll need to look into this stuff. Hopefully, I'll have some
> intelligent questions to ask to get this sorted
> 
> > > AUTO_REBOOT=yes  # yes|no|wait xx
> > not in the scope of aif. call aif && reboot if you want this.
> 
> OK
> 
> > > 
> > > POST_INSTALLATON_SCRIPT= # This could be a .sh script run only on
> > > first reboot of new system - ideas: alsa-config, pacman -Sy, db
> > > update, setup printer, some tests, email admin, install more
> > > packages, etc
> > > 
> > yeah good idea
> 
> Thanks. How would this be done effectively?
make sure the script(s) are available during installation. there are 2
ways for this:
1) on the beforehand. eg package them and add them to the package list
when building the iso with archiso (or add the files to the squashfs
overlay, which is more ugly but saves you the packaging step)
2) at runtime by wget/svn checkout/....

Btw my personal profile (work in progress!) will be interesting to you
http://github.com/Dieterbe/aif/blob/ef2290b0e31c285267bba9dde3634b628df0ebf0/unofficial/dieter-desktop-a7n8x.automaticprofile
I check out some files from svn and also demonstrate how you can make
custom workers and add them to the phase.

once you have the file(s) during installation, copy them to the target
system, and then to get them to execute I don't know what the cleanest
way is. i would append to /etc/rc.sysinit, there may be cleaner ways
for it.  you could also make an "initscript" and put the "daemon" in
rc.conf

> 
> > 
> > > [End of TestMachine] # could be something else or nothing
> > > 
> > > 
> > > [BuildMachine]
> > > # Repeated from above; adjust options
> > > 
> > > [Laptop]
> > > # Repeated from above; adjust options
> > > 
> > > [Server]
> > > # Repeated from above; adjust options
> > > 
> > > [VirtualTest]
> > > # Repeated from above; adjust options
> > > 
> > > # Another idea would be to have a [Default] profile then each
> > > following profile would just list the specific alterations
> > > 
> > > # End of File
> > i'm not pro this.  see earlier for my thoughts over profiles. maybe
> > i could make a "depend" statement like i already do for
> > libraries/procedures where you can depend on an other profile.
> > (which basically means to source an other file first). this gives
> > you what you want, it's easy to implement. everybody happy :)
> 
> One profile per file is ok with me.
> 
> > 
> > Concluding notes about additional config files:
> > many times a user could want to specify some customisations for
> > certain config files.
> > you propose a lot of these variables which will just make aif parse
> > the variables, generate some output and append/replace into a
> > certain config file.
> > I'm just wondering if it's better to implement that for a lot of
> > files (/etc/group, mkinitcpio.conf, ...) or just let a user keep
> > such files in a svn repo or something and to just check them out.
> 
> Most users (99%) won't have SVN (or similar) - maybe a big (lots of
> pc's) company would set one up. Been profile independent and by
> appending to the file it allows for the latest Arch and upstream
> changes without any or too much fiddling with the scripts and
> therefore re-creating the boot media. If it is possible to go down
> this path, what do you think the best formatting would be?
Sorry I don't really understand what you mean.  though if you want to
update your installation "settings" (abstract speaking) you have 2
options:
1) build new media
2) keep old media but publish your "stuff" somewhere so you/aif can
grab/scp them.  if you can scp/wget/.. a config file the step to
something like a git repository is very small


> 
> > i guess it's a question of "let the user maintain "state" (or
> > even generate: in that case putting it in aif would make more
> > sense) himself (the config files themselves), or let aif generate
> > the "state" files from input "metadata".
> > 
> > I think letting aif do this may be better, though it would be quite
> > some work, and you lack the "feedback loop" (eg now i can just
> > commit my config files into svn and aif will check them out on the
> > next install), with your suggested things, you need to update the
> > aif profile yourself.
> 
> I would imagine a smart user would have the profiles locally and/or
> SVN/Git and add them to the boot media. Any more thoughts on this?
Yes, either that or pull the latest ones during the installation
process
> 
> > I'll need to think a bit about this. :)
> > 
> > Dieter
> 
> Thanks again for your help and great work on the installer. 
> 
> Regards
> 
> Jud