[arch-releng] Draft - Beyond AIF, long live AIF...
Hi, First excuse, I'm not a native and good English writer and speaker, and writing such long texts is something i normally do not day-by-day. Could we discuss about a suggestion: What if we drop the interactive procedure on installation process and offers only the automatic way, controlled by a plain text file? Huh? What he is talking about? =============================== AIF could use a control file in plain text to do the installation; also complex installations (ex: LVM and dmcrypt). For examples take a look at [1] and [2]. The install process then is running automatically without need of user invention. Installer paradigms ==================== Today installers (either if graphically or text based) are mostly interactive tools. They ask the user for information and the user is giving data to the installer. Also the user must make decisions in some steps, say Yes or No. Also in current AIF in interactive mode. Usual a installation is a top-down and sequent work-flow. But this don't prevent that user do "silly" things. So installers (also AIF) must do many things "behind the view" to intercept mistakes. Or need user invention/decision based on information from a procedure before current step... As a example the way grub install is handled by old arch-installer (AIF does this a little other): We have to ask about root/boot devices to generate a matching menu.lst file. This file is presented to the user again to have a look or change something. From the point that the user may have changed something some/most informations must be asked again (or be parsed from the changed menu.lst file). Main problem IMHO is: a today installer doesn't know at the beginning with what information it will conclude at the end. So each needed information for a step has to be asked/determined before this step. K.I.S.S me... ================= A today installer is hand-holding the user. It must be. From the user side the installer is the first tool he normally see from a distribution. How good he/she "feels" during this process, this often make a pro or contra later. The user - as a !Master! - means: I could work well with this installer, it does all what i want. First mistake! It's always the user that makes what the installer wants. That's the nature of this special piece oft software: It has a very restricted environment and its only goal is: install a working system. From a installer view the user is the obstructive part - only needed to give some information. To get those the installer feeds the user: give him nice varicoloured output, some play music or nice balls are wandering from the left to the right in progress bars... Meanwhile the installer must take care that the user make only one step left or one step right - but not two steps... In the "older days" installers - from users side - tend to be complex software; these days the trend is to keep the user off from the technical base behind the process of setup a base system. "5 clicks to get xyz ready to run..", "Full featured linux in 15 minutes - only 3 questions"... and so on. The result - we know this - are the full-bloated systems where in fact the distributions determine most of the users system. ... we are frogs! ================= During last images build and bugfixing i thought a lot about this. I know that our "old Devs" always prefer quickinst from old arch-installer package. Now coming familar with AIF, myself look most times and primary on the interactive part; but doing some work with automatic install scripts IMHO there is more potential than with quickinst. So my question: Am i totally wrong with my thoughts and - if not - have we the pluck to confront the user with... an text editor? Take a breath before hit Reply key... ;-) First: The above mentioned aif scripts are not the answer - syntax, presentation and documentation are more the "proof-concept" to look for answers. Pros & Contras ================== 1. Making the settings for installation in a control file gives not that pressure to the user what a User-Interface(UI) based installer has. Working with this file ("editing") does not any bad things. Look for ex. on the grub part currently in interactive AIF: The user is confronted in this step with questions in a very isolated manner. This step is very interdependent from the previous blockdevice setup - but these information's were entered before 10 minutes and out-of-users-sight in this step. In a editor all information is easy browseable and corrections could make at any point. IMHO a pro. 2. Install information could be saved at any time to external place by the user - or come complete from external medium. Currently our install procedure has to be restarted every time if the user got "stucked" in one step, or canceled the process, or when the installer has a bug. Most (and so in fact all) informations have to be re-entered... Think further about the capabilities: We (and the community) could offer templates for nearly each installation setup - without the need to bring this in a UI... If one has the idea to write a web-based interface to generate such installation control file - why not? IMHO a pro. 3. On the road to 2009.08 (and AIF) many of reported problems were UI related problems. Such things like "Cursor not automatically placed in Menu xyz". But also during the grub refactoring we have seen: to get feature or procedure X to work we must have here a new dialog, and there we have to change a other dialog... and so on. I guess on most interactive programs the UI-based part is the biggest part. And often the part with the most possible bugs. Current interactive AIF hast 2 UIs: the ncurses dialog based and a pure read/print based one. Ok, we will never ever have a graphical installer, but we hear often: why do you use such a "antique" UI? And in fact: Most today machines/environments could handle a graphically installer and also modern frameworks give even better widgets to interact with the user... But: Another UI gives potential more bugs and needs more manpower which could be better used to make the "base" more stable. If we use the control file based install we get rid off from such discussions of the "neatest" installer... And IMHO the step away from "how the others do it" could make us very modern... IMHO a pro. 4. Yes, i have a contra... Control files/variables tend to be not very self-explanation. For example take the current syntax for blockdevice setup: -------------- GRUB_DEVICE=/dev/sda PARTITIONS='/dev/sda 100:ext2:+ 512:swap *:ext4' BLOCKDATA='/dev/sda1 raw no_label ext2;yes;/boot;target;no_opts;no_label;no_params /dev/sda2 raw no_label swap;yes;no_mountpoint;target;no_opts;no_label;no_params /dev/sda3 raw no_label ext4;yes;/;target;no_opts;no_label;no_params' ------------- Here we have to invest a lot of work to bring such a syntax to the user in a better way. But on such complex steps i could imagine on "helper applications". Maybe a similar dialog-based we currently use in interactive AIF. But instead to "feed" a installer with this informations this helper produces for ex. the BLOCKDATA string for above control file. The user then import this generated piece into the editor. Similar helper could be one for timezone strings; all information a user normally not know by rote and so must be "picked" from a list. IMHO: if we make this good, the contra could went to a pro - cause at the end the user has still a (correctly) control file to start installation. Read until here....? ======================= Wow. Yes, I'm really thinking (and like to discuss) about a work-flow: a) User boots the image, login. b) Select/start a text editor with a base control file or select one of the other pre-defined control files. c) Edit his settings in this control file d) Uses the other tty's for getting infos, start helpers. Helper output he could load into the current control file editor. e) After "editing the setup" he could save it extern and come back later or f) start the installation. No interaction is needed after this (Ok, before fdisk'ing and formatting we should make a "break and review"). g) Base system is installed - user is happy... Cookies and taccos... What do you think? Of course, this is nothing for the next images... It's a massive U-turn against the main stream, but at the end IMHO the benefits could be great - KISS like... These are only my personal thoughts as a releng member, no position of "official politic". And now: Fire at will! ;-) Regards Gerhard [1] http://github.com/Dieterbe/aif/blob/2ef92eaa5f1106d8f1b379a8a94a477a0877bb1f... [2] http://github.com/Dieterbe/aif/blob/2ef92eaa5f1106d8f1b379a8a94a477a0877bb1f...
On Mon, 17 Aug 2009 19:19:22 +0200 Gerhard Brauer <gerbra@archlinux.de> wrote:
Installer paradigms ==================== Today installers (either if graphically or text based) are mostly interactive tools. They ask the user for information and the user is giving data to the installer. Also the user must make decisions in some steps, say Yes or No. Also in current AIF in interactive mode.
Usual a installation is a top-down and sequent work-flow. But this don't prevent that user do "silly" things. So installers (also AIF) must do many things "behind the view" to intercept mistakes.
I don't think aif does much checking for mistakes from the user. The only things I can think of: notices for missing swap/root/home etc, and checking if steps are completed succesfully. Little code overhead here IIRC.
Or need user invention/decision based on information from a procedure before current step... As a example the way grub install is handled by old arch-installer (AIF does this a little other): We have to ask about root/boot devices to generate a matching menu.lst file. This file is presented to the user again to have a look or change something. From the point that the user may have changed something some/most informations must be asked again (or be parsed from the changed menu.lst file).
Main problem IMHO is: a today installer doesn't know at the beginning with what information it will conclude at the end. So each needed information for a step has to be asked/determined before this step.
maybe we should estimate how much extra code is needed for these things. I would guess a few hundred lines maximum.
... we are frogs! ================= During last images build and bugfixing i thought a lot about this. I know that our "old Devs" always prefer quickinst from old arch-installer package. Now coming familar with AIF, myself look most times and primary on the interactive part; but doing some work with automatic install scripts IMHO there is more potential than with quickinst.
yeah.
So my question: Am i totally wrong with my thoughts and - if not - have we the pluck to confront the user with... an text editor?
Take a breath before hit Reply key... ;-)
First: The above mentioned aif scripts are not the answer - syntax, presentation and documentation are more the "proof-concept" to look for answers.
what do you mean exactly? the syntax is too hard for the config files? I do think the disk section of the configs can be made more simple. (see also http://bugs.archlinux.org/task/15640?project=6), but i like the other stuff.
Pros & Contras ==================
1. Making the settings for installation in a control file gives not that pressure to the user what a User-Interface(UI) based installer has. Working with this file ("editing") does not any bad things. Look for ex. on the grub part currently in interactive AIF: The user is confronted in this step with questions in a very isolated manner. This step is very interdependent from the previous blockdevice setup - but these information's were entered before 10 minutes and out-of-users-sight in this step. In a editor all information is easy browseable and corrections could make at any point. IMHO a pro.
If the information we need was entered earlier, we can probably reuse it by storing it in a variable. (like we do now for some things like proxy server, keymap etc)
2. Install information could be saved at any time to external place by the user - or come complete from external medium. Currently our install procedure has to be restarted every time if the user got "stucked" in one step, or canceled the process, or when the installer has a bug. Most (and so in fact all) informations have to be re-entered... Think further about the capabilities: We (and the community) could offer templates for nearly each installation setup - without the need to bring this in a UI...
yeah this sounds good
If one has the idea to write a web-based interface to generate such installation control file - why not?
i hate webinterfaces. I get your point, but I would do it with shellscripts.
3. On the road to 2009.08 (and AIF) many of reported problems were UI related problems. Such things like "Cursor not automatically placed in Menu xyz". But also during the grub refactoring we have seen: to get feature or procedure X to work we must have here a new dialog, and there we have to change a other dialog... and so on. I guess on most interactive programs the UI-based part is the biggest part. And often the part with the most possible bugs. Current interactive AIF hast 2 UIs: the ncurses dialog based and a pure read/print based one. Ok, we will never ever have a graphical installer, but we hear often: why do you use such a "antique" UI? And in fact: Most today machines/environments could handle a graphically installer and also modern frameworks give even better widgets to interact with the user... But: Another UI gives potential more bugs and needs more manpower which could be better used to make the "base" more stable. If we use the control file based install we get rid off from such discussions of the "neatest" installer... And IMHO the step away from "how the others do it" could make us very modern... IMHO a pro.
I don't think there is anything wrong with ncurses / std{in,out,err} per se. But in general, yes: the simpler the better (as long as it's still easy)
4. Yes, i have a contra... Control files/variables tend to be not very self-explanation. For example take the current syntax for blockdevice setup: -------------- GRUB_DEVICE=/dev/sda PARTITIONS='/dev/sda 100:ext2:+ 512:swap *:ext4' BLOCKDATA='/dev/sda1 raw no_label ext2;yes;/boot;target;no_opts;no_label;no_params /dev/sda2 raw no_label swap;yes;no_mountpoint;target;no_opts;no_label;no_params /dev/sda3 raw no_label ext4;yes;/;target;no_opts;no_label;no_params' ------------- Here we have to invest a lot of work to bring such a syntax to the user in a better way. But on such complex steps i could imagine on "helper applications". Maybe a similar dialog-based we currently use in interactive AIF. But instead to "feed" a installer with this informations this helper produces for ex. the BLOCKDATA string for above control file. The user then import this generated piece into the editor. Similar helper could be one for timezone strings; all information a user normally not know by rote and so must be "picked" from a list. IMHO: if we make this good, the contra could went to a pro - cause at the end the user has still a (correctly) control file to start installation.
yeah, but note this syntax is not definite (see http://bugs.archlinux.org/task/15640?project=6)
Read until here....? ======================= Wow. Yes, I'm really thinking (and like to discuss) about a work-flow: a) User boots the image, login. b) Select/start a text editor with a base control file or select one of the other pre-defined control files. c) Edit his settings in this control file d) Uses the other tty's for getting infos, start helpers. Helper output he could load into the current control file editor. e) After "editing the setup" he could save it extern and come back later or f) start the installation. No interaction is needed after this (Ok, before fdisk'ing and formatting we should make a "break and review"). g) Base system is installed - user is happy... Cookies and taccos...
What do you think? Of course, this is nothing for the next images... It's a massive U-turn against the main stream, but at the end IMHO the benefits could be great - KISS like...
These are only my personal thoughts as a releng member, no position of "official politic".
And now: Fire at will! ;-)
Regards Gerhard
[1] http://github.com/Dieterbe/aif/blob/2ef92eaa5f1106d8f1b379a8a94a477a0877bb1f... [2] http://github.com/Dieterbe/aif/blob/2ef92eaa5f1106d8f1b379a8a94a477a0877bb1f...
I've also been thinking the installer currently does too much hand holding. Especially if you compare to an arch system after it is installed: it lets the user manually invoke pacman and let him edit rc.conf himself etc. There are no fancy wrappers then, so we probably don't need them during the installation either. I do think it's very beneficial to have a method to install arch: - that requires you to do no single keystroke more then needed to give you the system you want. - that does not take any second longer then needed. - that assures you you didn't forget anything This is where -p interactive is very good at. No matter my love for automated installs, it's nice that you can almost brainlessly do the interactive one and end up very quickly with a customized install. I like the idea of simple scripts that generate config files for -p automatic. based on a few questions, output of fdisk -l etc. (actually I think this is how Gentoo is doing it now, or at least how one of the Gentoo relengs planned to do it. I've been chatting with one when working on early aif) But if you want to generate all fine nuances into the config file, the script will become as complex as -p interactive. note that the "store disk layouts in a nice format, and programatically parse those formats and act correctly on them (invoke mkfs, mount etc)" we currently do requires quite some code (rough estimate: about 1000 sloc). I've been thinking if it wouldn't be better to let the user write scripts in which he does 'mkfs.. mount..' himself. it would definitely be easier to code, but the current approach offers some features like rollbacks etc which you otherwise would have to do manually (or script it also), which would make the users scripts/configs uglier. Dieter
Am Montag, den 17.08.2009, 20:49 +0200 schrieb Dieter Plaetinck:
On Mon, 17 Aug 2009 19:19:22 +0200 Gerhard Brauer <gerbra@archlinux.de> wrote:
Installer paradigms ==================== Today installers (either if graphically or text based) are mostly interactive tools. They ask the user for information and the user is giving data to the installer. Also the user must make decisions in some steps, say Yes or No. Also in current AIF in interactive mode.
Usual a installation is a top-down and sequent work-flow. But this don't prevent that user do "silly" things. So installers (also AIF) must do many things "behind the view" to intercept mistakes.
I don't think aif does much checking for mistakes from the user. The only things I can think of: notices for missing swap/root/home etc, and checking if steps are completed succesfully. Little code overhead here IIRC.
It's not only the "mistake prevention" (see how grub part in lib-ui-interactive and lib-blockdevice is grown only from grub/sw-RAID refactoring. OK, this is not the very best code, but an example). Compare the UI related code against what you/we have/need for the automatic part.
Main problem IMHO is: a today installer doesn't know at the beginning with what information it will conclude at the end. So each needed information for a step has to be asked/determined before this step.
maybe we should estimate how much extra code is needed for these things. I would guess a few hundred lines maximum.
We could do this also in interactive, sure: previously get all informations before doing any install related commnds. But why? Why not edit this in control file instead ask everything via dialog?
First: The above mentioned aif scripts are not the answer - syntax, presentation and documentation are more the "proof-concept" to look for answers.
what do you mean exactly? the syntax is too hard for the config files? I do think the disk section of the configs can be made more simple. (see also http://bugs.archlinux.org/task/15640?project=6), but i like the other stuff.
I mentioned this only as an example. Remember what i want: The user must edit his partitioning and blockdevice setup in control file. So the syntax must be a) documented and b) a little bit more intuitiv. Ex: User has a windows partition (sda1). It must be clear what he has to enter in BLOCKDATA setup to a) prevent this partition (or we go everytime the hard way and erase each fat/ntfs partition <g>) and b) to create additional primary/logical partitions.. But this discussion is for me at this point a bit to detailed. I like more discussions on my main cause...
Pros & Contras ==================
1. Making the settings for installation in a control file gives not that pressure to the user what a User-Interface(UI) based installer has. Working with this file ("editing") does not any bad things. Look for ex. on the grub part currently in interactive AIF: The user is confronted in this step with questions in a very isolated manner. This step is very interdependent from the previous blockdevice setup - but these information's were entered before 10 minutes and out-of-users-sight in this step. In a editor all information is easy browseable and corrections could make at any point. IMHO a pro.
If the information we need was entered earlier, we can probably reuse it by storing it in a variable. (like we do now for some things like proxy server, keymap etc)
Yes, i think we could refactor such things a better way (think about how often we look if we at the current step have a separate /boot partition. But: Why put time in this when the control file paradigm give the installer all informations in "one piece"?
If one has the idea to write a web-based interface to generate such installation control file - why not?
i hate webinterfaces. I get your point, but I would do it with shellscripts.
I don't meant user procedures, i meant the control files (what we feed to AIF with
If we use the control file based install we get rid off from such discussions of the "neatest" installer... And IMHO the step away from "how the others do it" could make us very modern... IMHO a pro.
I don't think there is anything wrong with ncurses / std{in,out,err} per se. But in general, yes: the simpler the better (as long as it's still easy)
And the simples way (from AIF sight!) is surely the control-file/automatic approach. And my point: I think it's also the simplest way for the user (if he/she has a base linux knowledge). I think on this from my steps to Archlinux: In 2007 i came from Debian (i have loved it and i still do), i only tried other distros cause myself was not able to get it worked on my Laptop. Installation was no problem with 2007 installer for me. If i now think that i have had at this time a Readme that advises me to edit a control file to start the installation - then i sure would have wonder a little bit, but when the content of this file were intuitive then i would have no problem with this...
I've also been thinking the installer currently does too much hand holding. Especially if you compare to an arch system after it is installed: it lets the user manually invoke pacman and let him edit rc.conf himself etc. There are no fancy wrappers then, so we probably don't need them during the installation either.
I do think it's very beneficial to have a method to install arch: - that requires you to do no single keystroke more then needed to give you the system you want. - that does not take any second longer then needed. - that assures you you didn't forget anything This is where -p interactive is very good at. No matter my love for automated installs, it's nice that you can almost brainlessly do the interactive one and end up very quickly with a customized install.
That's right, and something you/we could be proud of. And now I want to kick this completely ;-) (Am i evil? Yes, i am...)
I like the idea of simple scripts that generate config files for -p automatic. based on a few questions, output of fdisk -l etc. (actually I think this is how Gentoo is doing it now, or at least how one of the Gentoo relengs planned to do it. I've been chatting with one when working on early aif) But if you want to generate all fine nuances into the config file, the script will become as complex as -p interactive.
Yes, it will be complex. But then comes IMHO my point 1) by Pro&Contra into the game. But only complex scenarios need complex control files. Think about when we may want to add sw RAID into installer: the needed code for UI will be X-size bigger than only for automatic AIF....
note that the "store disk layouts in a nice format, and programatically parse those formats and act correctly on them (invoke mkfs, mount etc)" we currently do requires quite some code (rough estimate: about 1000 sloc). I've been thinking if it wouldn't be better to let the user write scripts in which he does 'mkfs.. mount..' himself. it would definitely be easier to code, but the current approach offers some features like rollbacks etc which you otherwise would have to do manually (or script it also), which would make the users scripts/configs uglier.
The approach that user (who "only" will install a system) have to write scripts is not a way IMHO. That's something for the experienced user who will for. have own procedures for a special case. The rollback is IMHO no problem (also in automatic way). I mentioned already that we definitely a "Last chance to press Control-C!" before we do anything with (c)fdisk and/or mkfs.*. At this point we could easily implement also the rollback... Also in grub setup step...
Dieter
Gerhard (who has expected many more replies on my mail, but maybe you all say: Yes, that must be the sun which makes him silly... <g>)
On Wed, 19 Aug 2009 18:52:43 +0200 Gerhard Brauer <gerbra@archlinux.de> wrote:
Am Montag, den 17.08.2009, 20:49 +0200 schrieb Dieter Plaetinck:
On Mon, 17 Aug 2009 19:19:22 +0200 Gerhard Brauer <gerbra@archlinux.de> wrote:
Installer paradigms ==================== Today installers (either if graphically or text based) are mostly interactive tools. They ask the user for information and the user is giving data to the installer. Also the user must make decisions in some steps, say Yes or No. Also in current AIF in interactive mode.
Usual a installation is a top-down and sequent work-flow. But this don't prevent that user do "silly" things. So installers (also AIF) must do many things "behind the view" to intercept mistakes.
I don't think aif does much checking for mistakes from the user. The only things I can think of: notices for missing swap/root/home etc, and checking if steps are completed succesfully. Little code overhead here IIRC.
It's not only the "mistake prevention" (see how grub part in lib-ui-interactive and lib-blockdevice is grown only from grub/sw-RAID refactoring. OK, this is not the very best code, but an example). Compare the UI related code against what you/we have/need for the automatic part.
the grub stuff in -p automatic is much less advanced. and it uses grub-install, whereas the interactive one calls `$var_TARGET_DIR/sbin/grub` and passes lowlevel statements to it. We can probably refactor the interactive grub stuff to be more concise (using grub-install among others) Also, look at like this: often you need all the "fancy" logic anyway for the automatic procedure (detecting boot device, figuring out the order in which to create filesystems, etc) I think the interactive procedure really doesn't add that much code.
Main problem IMHO is: a today installer doesn't know at the beginning with what information it will conclude at the end. So each needed information for a step has to be asked/determined before this step.
maybe we should estimate how much extra code is needed for these things. I would guess a few hundred lines maximum.
We could do this also in interactive, sure: previously get all informations before doing any install related commnds. But why? Why not edit this in control file instead ask everything via dialog?
I've given some arguments in the mail you're quoting already (the most important one being that you can do it fast and without thinking/researching too much).
If one has the idea to write a web-based interface to generate such installation control file - why not?
i hate webinterfaces. I get your point, but I would do it with shellscripts.
I don't meant user procedures, i meant the control files (what we feed to AIF with
You were talking about webinterfaces to generate the config file for the automatic procedure. I'm saying I prefer shellscripts over webinterfaces (to generate those "control files")
I've also been thinking the installer currently does too much hand holding. Especially if you compare to an arch system after it is installed: it lets the user manually invoke pacman and let him edit rc.conf himself etc. There are no fancy wrappers then, so we probably don't need them during the installation either.
I do think it's very beneficial to have a method to install arch: - that requires you to do no single keystroke more then needed to give you the system you want. - that does not take any second longer then needed. - that assures you you didn't forget anything This is where -p interactive is very good at. No matter my love for automated installs, it's nice that you can almost brainlessly do the interactive one and end up very quickly with a customized install.
That's right, and something you/we could be proud of. And now I want to kick this completely ;-) (Am i evil? Yes, i am...)
Why kick it, as it has a valid use case? Maybe we could make (on the longer term) the automatic stuff more the "recommended way" and have the interactive procedure as a secondary method, but i definitely wouldn't drop it.
I like the idea of simple scripts that generate config files for -p automatic. based on a few questions, output of fdisk -l etc. (actually I think this is how Gentoo is doing it now, or at least how one of the Gentoo relengs planned to do it. I've been chatting with one when working on early aif) But if you want to generate all fine nuances into the config file, the script will become as complex as -p interactive.
Yes, it will be complex. But then comes IMHO my point 1) by Pro&Contra into the game. But only complex scenarios need complex control files. Think about when we may want to add sw RAID into installer: the needed code for UI will be X-size bigger than only for automatic AIF....
Your point 1 is not an argument to throw interactive away. especially since your example (the grub stuff) is not a fair comparison (see above)
note that the "store disk layouts in a nice format, and programatically parse those formats and act correctly on them (invoke mkfs, mount etc)" we currently do requires quite some code (rough estimate: about 1000 sloc). I've been thinking if it wouldn't be better to let the user write scripts in which he does 'mkfs.. mount..' himself. it would definitely be easier to code, but the current approach offers some features like rollbacks etc which you otherwise would have to do manually (or script it also), which would make the users scripts/configs uglier.
The approach that user (who "only" will install a system) have to write scripts is not a way IMHO. That's something for the experienced user who will for. have own procedures for a special case.
The rollback is IMHO no problem (also in automatic way). I mentioned already that we definitely a "Last chance to press Control-C!" before we do anything with (c)fdisk and/or mkfs.*. At this point we could easily implement also the rollback... Also in grub setup step...
the rollback destroys any newly created dm volumes (lvm and such) and umounts new filesystems, it's only goal is making it possible to define (and create) new setups without breaking over things that are in use or still mounted. (but this is off topic) Dieter
participants (2)
-
Dieter Plaetinck
-
Gerhard Brauer