I'd really like to lend a hand with AIF, considering Dieter's recent resignation. I've been using Linux as my primary OS for more than a decade, and have been using Arch for a few years now, and love it.
Tools like AIF are what I tend to get really into -- I love being able to automate things like installing a system or making it easy for someone to customize an installation. One of my (not yet live) pet projects is a web service that lets people generate their own install and live cds, the proof-of-concept I threw together is on top of Arch, using AIF -- I've done a bit of reading through it's codebase already, although I'm far from grokking it in full.
Besides, I've been wanting to start giving back to Arch. I've gotten a lot of joy from this distro.
Anyhow, if AIF is being entirely trashed, I'd still like to be involved in similar efforts. If not, it would be great if anyone had good info on what needs done that they'd be willing to share.
Thanks for your time, and for everyones work on Arch,
Hello,
I would also like to help out with AIF. How about we try to collaborate on something, Jeremiah?
The last couple of days, I've started working on "acu": https://github.com/xyproto/acu And created a webpage for submitting and browsing "machine configuration bundles": http://www.roboticoverlords.org/
This is all work in progress and not in any way "officially blessed", but my aspiration is that it might be some day.
My main idea is that if the installation process issplit into "personal configuration, like keyboard and timezone", "machine configuration bundles, crafted per type of computer setup" and installation-specific instructions like partition and network settings, the installations would be simpler and cost less work (especially for experienced users).
Alexander Rødseth rodseth@gmail.com writes:
Hello,
I would also like to help out with AIF. How about we try to collaborate on something, Jeremiah?
It's totally a possibility!
The last couple of days, I've started working on "acu": https://github.com/xyproto/acu And created a webpage for submitting and browsing "machine configuration bundles": http://www.roboticoverlords.org/
It looks like it could be an interesting project, some extrapolation below.
This is all work in progress and not in any way "officially blessed", but my aspiration is that it might be some day.
My main idea is that if the installation process issplit into "personal configuration, like keyboard and timezone", "machine configuration bundles, crafted per type of computer setup" and installation-specific instructions like partition and network settings, the installations would be simpler and cost less work (especially for experienced users).
I find the "configuration bundles" idea interesting, but I wonder how much of it is needed on a per-machine basis. I haven't run into many machine-specific issues, but then again I'm sure they exist.
The end-result I'm working toward is providing a service that allows people to generate linux live and install images that have what they want on them already -- I've written a proof-of-concept webapp that installs a package on a live and install cd that I intend to start a kickstarter with[1]. My goal is to allow college kids and newer linux users to create something that will give them access to the system they need or want in case of failure or circumstance, and allow it to perform (if desired) unattended installs.
My app at this point isn't much more than a pretty thin wrapper over aif, it does successfully generate images though.
Assuming I can manage to get that off the ground, there are a lot of neat things to be done with enabling people to share configs.
A couple of questions on acu based off the github page:
1. Why create a configuration file format? There are many standard formats ranging from simplistic to very complex, and even if go didn't have an implementation of one laying around, it'd probably be simpler (and less confusing/more reliable/quicker) to write a parser for one.
2. What, exactly would you have machine-specific configs entail? There's a difference between purpose-specific and hardware-specific ... I feel that the former has more common use cases, but on the other hand a *lot* of the configs and specific packages that I can think of for a "purpose" machine are going to vary wildly from user to user.
3. Why go?
Footnotes: [1] I know this can be a controversial thing, but to develop what I want to in full I need to be able to cover hosting costs and living costs for a couple of months. I happen to be poor, and would seriously prefer to be able to work on something like this full-time, and I feel that if the community decides it's awesome enough, I'll be able to do so.
Hi,
2012/7/27 Jeremiah Dodds jeremiah.dodds@gmail.com:
I find the "configuration bundles" idea interesting, but I wonder how much of it is needed on a per-machine basis. I haven't run into many machine-specific issues, but then again I'm sure they exist.
I've encountered a lot of machine-specific issues, setting up (Arch) Linux on several computers. Especially laptops often need extra modules loaded to make the sound card, network card or video camera to work, possibly with a couple of firmware-packages thrown into the mix. Specifying all it takes to set up a machine in a single, easy to read, text file, shared among Arch Linux users in the same fashion as AUR-packages is shared, would be an improvement, IMO.
The end-result I'm working toward is providing a service that allows people to generate linux live and install images that have what they want on them already -- I've written a proof-of-concept webapp that installs a package on a live and install cd that I intend to start a kickstarter with[1]. My goal is to allow college kids and newer linux users to create something that will give them access to the system they need or want in case of failure or circumstance, and allow it to perform (if desired) unattended installs.
I think this approach would work too.
A couple of questions on acu based off the github page:
- Why create a configuration file format? There are many standard formats ranging from simplistic to very complex, and even if go didn't have an implementation of one laying around, it'd probably be simpler (and less confusing/more reliable/quicker) to write a parser for one.
As far as I can tell, not that much information is needed in order to specify what it takes to configure a computer. A list of packages to install, a few changed configuration options and perhaps the addition of a few lines in a few configuration files should do the trick. This could be expressed either as a regular shell script or as some sort of list of declarations. Both could work, but it we imagined there was a webpage with hundreds of these configurations, it would be easier compare configurations and base new configurations on old ones if the file format was declarative. Also, it would be easier to parse for the web-system. The AUR web page struggles to parse the non-declarative PKGBUILD textfiles.
- What, exactly would you have machine-specific configs entail? There's a difference between purpose-specific and hardware-specific ... I feel that the former has more common use cases, but on the other hand a *lot* of the configs and specific packages that I can think of for a "purpose" machine are going to vary wildly from user to user.
My vision is that an install would consist of three parts. The machine configuration should be auto-detected or specified and downloaded with an AUR-like system. The personal configuration, like time zone, locale and keyboard layout should only need to be specified once ever and stored as a short string or as some sort of id. I can't could how many times I've specified the same settings, and it should not be needed in our day and age. Once should be enough. The third and last part would be the information that is specific to the installation in question. Which disk should the installation be on? Is this supposed to be a media server? Should the filesystems handle small files well? Is wireless network preferred? In my opinion, this third part is what users _should_ be worrying about when installing Arch, not selecting keyboard layout for the n'th time and not go hunting for forum posts about how to get the soundcard to work on this particular brand of laptop. Hope this answered your question.
- Why go?
This is subjective: I love the Go programming language. The code has clarity, it has better modularity than both Python and C++. Code compiles quickly, unlike for C++. Code feels sturdy and like you can actually build a fortress instead of a tent, like in Python (even though you can build really big tents in Python. I love Python too.). It's easy to do web-related programming and concurrency, unlike C. And the execution speed is approaching C and C++, as the compiler matures. (And yes, the garbage collection can be turned off, or one can manage memory yourself). In addition to this, the pointers are safe. In short, you avoid all the major pains that comes with a C++ projects over N lines long (say 100000). Code is a lot quicker to write than compared to Haskell, but it still feels solid. That was the subjective reasons.
The non-subjective reason for choosing Go, in this connection, is that it easily compiles to a static binary that does not need any dependencies (packages). For an installer, this is great. Also, the execution speed is fast. If there are several developers involved, there are no arguments about tabs vs spaces, since the standard (and tool that formats the code) is clear and it has good modularity.
My next plan is to rewrite the code I wrote for the web-interface for machine configurations, but in Go. I'm in the process of moving this to a server that makes this possible. Either that or use the PHP-code for AUR as a basis. Or some other solution. Time will tell. (The current PHP-code is bad, but crappy prototypes keeps the momentum up).
In either case: no matter the approach and no matter the authors, I'm looking forward to the next generation of installers.
On Tue, 31 Jul 2012 15:24:38 +0200 Alexander Rødseth rodseth@gmail.com wrote:
Hi,
2012/7/27 Jeremiah Dodds jeremiah.dodds@gmail.com:
I find the "configuration bundles" idea interesting, but I wonder how much of it is needed on a per-machine basis. I haven't run into many machine-specific issues, but then again I'm sure they exist.
I've encountered a lot of machine-specific issues, setting up (Arch) Linux on several computers. Especially laptops often need extra modules loaded to make the sound card, network card or video camera to work, possibly with a couple of firmware-packages thrown into the mix. Specifying all it takes to set up a machine in a single, easy to read, text file, shared among Arch Linux users in the same fashion as AUR-packages is shared, would be an improvement, IMO.
there's a bunch of configuration management tools written for this use case already. like chef, puppet, ansible. many of these concerns are relatively easy to abstract for multiple distributions, so writting a new tool that only targets Arch is not a good idea imho.
The end-result I'm working toward is providing a service that allows people to generate linux live and install images that have what they want on them already -- I've written a proof-of-concept webapp that installs a package on a live and install cd that I intend to start a kickstarter with[1]. My goal is to allow college kids and newer linux users to create something that will give them access to the system they need or want in case of failure or circumstance, and allow it to perform (if desired) unattended installs.
an installation of any system should put only the basic system in place (incl. a config management tool), everything else should be done by configuration management. this makes it easier to alter the configuration of a machine long after it has been installed.
In either case: no matter the approach and no matter the authors, I'm looking forward to the next generation of installers.
configuration management should not be tied to only the installation process.
Hi,
2012/7/31 Dieter Plaetinck dieter@plaetinck.be:
there's a bunch of configuration management tools written for this use case already. like chef, puppet, ansible. many of these concerns are relatively easy to abstract for multiple distributions, so writting a new tool that only targets Arch is not a good idea imho.
All of the tools you mention requires several dependencies, cover completely different use cases from what I wrote about and are needlessly complex for what I had in mind, even ansible. In addition, it's not like the space for configuration tools is crowded and out of axes and variables to compete on. In short, I completely disagree with you on this. (And this is not a case of NIH). More momentum is needed, not less. As I understand, you have resigned as a developer and is not interested in working on this or something like this in any case.
an installation of any system should put only the basic system in place (incl. a config management tool), everything else should be done by configuration management. this makes it easier to alter the configuration of a machine long after it has been installed.
I think this is nice in theory, but a failure in practice. For example, if someone wishes to change all the partitions from ReiserFS to ext4, they reinstall. Yes, it's possible to do it in other ways, but what a cludge of a configuration manager that would be. People reinstall and this should be made easier. Focusing on managing the post-installation configuration has proved to result in distro-specific configuration tools that are overkill compared to just editing a configuration text file and get it done with. Making the installation quick and easy, without users repeating themselves, is what should be done.
configuration management should not be tied to only the installation process.
Configuration management should only be tied to the installation process. For the rest, there's /etc.
Alexander Rødseth rodseth@gmail.com writes:
Hi,
2012/7/31 Dieter Plaetinck dieter@plaetinck.be:
there's a bunch of configuration management tools written for this use case already. like chef, puppet, ansible. many of these concerns are relatively easy to abstract for multiple distributions, so writting a new tool that only targets Arch is not a good idea imho.
All of the tools you mention requires several dependencies, cover completely different use cases from what I wrote about and are needlessly complex for what I had in mind, even ansible. In addition, it's not like the space for configuration tools is crowded and out of axes and variables to compete on. In short, I completely disagree with you on this. (And this is not a case of NIH). More momentum is needed, not less.
Actually, those tools *do* cover the use cases you wrote about, and even if they didn't it would be silly to write your own configuration file format.
As I understand, you have resigned as a developer and is not interested in working on this or something like this in any case.
Do you think that nullifies the value of his feedback?
an installation of any system should put only the basic system in place (incl. a config management tool), everything else should be done by configuration management. this makes it easier to alter the configuration of a machine long after it has been installed.
I think this is nice in theory, but a failure in practice. For example, if someone wishes to change all the partitions from ReiserFS to ext4, they reinstall. Yes, it's possible to do it in other ways, but what a cludge of a configuration manager that would be. People reinstall and this should be made easier. Focusing on managing the post-installation configuration has proved to result in distro-specific configuration tools that are overkill compared to just editing a configuration text file and get it done with. Making the installation quick and easy, without users repeating themselves, is what should be done.
Could you point to some examples of distro-specific configuration tools that you think are overkill?
configuration management should not be tied to only the installation process.
Configuration management should only be tied to the installation process. For the rest, there's /etc.
I have a feeling that you are using "configuration" in a broader sense, or in a different sense than Dieter is here, there's some more extrapolation in my other reply in this thread.
Dieter Plaetinck dieter@plaetinck.be writes:
On Tue, 31 Jul 2012 15:24:38 +0200 Alexander Rødseth rodseth@gmail.com wrote:
Hi,
2012/7/27 Jeremiah Dodds jeremiah.dodds@gmail.com:
I find the "configuration bundles" idea interesting, but I wonder how much of it is needed on a per-machine basis. I haven't run into many machine-specific issues, but then again I'm sure they exist.
I've encountered a lot of machine-specific issues, setting up (Arch) Linux on several computers. Especially laptops often need extra modules loaded to make the sound card, network card or video camera to work, possibly with a couple of firmware-packages thrown into the mix. Specifying all it takes to set up a machine in a single, easy to read, text file, shared among Arch Linux users in the same fashion as AUR-packages is shared, would be an improvement, IMO.
there's a bunch of configuration management tools written for this use case already. like chef, puppet, ansible. many of these concerns are relatively easy to abstract for multiple distributions, so writting a new tool that only targets Arch is not a good idea imho.
+1
The end-result I'm working toward is providing a service that allows people to generate linux live and install images that have what they want on them already -- I've written a proof-of-concept webapp that installs a package on a live and install cd that I intend to start a kickstarter with[1]. My goal is to allow college kids and newer linux users to create something that will give them access to the system they need or want in case of failure or circumstance, and allow it to perform (if desired) unattended installs.
an installation of any system should put only the basic system in place (incl. a config management tool), everything else should be done by configuration management. this makes it easier to alter the configuration of a machine long after it has been installed.
I agree in principle, especially for "official" installation media. I think there is a place for tools that handle unattended user-specific installs in a fashion that is transparent to the user, but don't think that conifg management should be tied to installation.
The tool that I'm building is intended as a service that automates a process involving a few tools, not as a monolithic thing -- at least not in the backend. I've been working on aif a bit, but I certainly wouldn't suggest tacking a configuration management system onto it.
In either case: no matter the approach and no matter the authors, I'm looking forward to the next generation of installers.
configuration management should not be tied to only the installation process.
+1
Alexander Rødseth rodseth@gmail.com writes:
Hi,
2012/7/27 Jeremiah Dodds jeremiah.dodds@gmail.com:
I find the "configuration bundles" idea interesting, but I wonder how much of it is needed on a per-machine basis. I haven't run into many machine-specific issues, but then again I'm sure they exist.
I've encountered a lot of machine-specific issues, setting up (Arch) Linux on several computers. Especially laptops often need extra modules loaded to make the sound card, network card or video camera to work, possibly with a couple of firmware-packages thrown into the mix. Specifying all it takes to set up a machine in a single, easy to read, text file, shared among Arch Linux users in the same fashion as AUR-packages is shared, would be an improvement, IMO.
A fair amount lot of this can be automatically detected, I think. I'm not opposed to the idea of hardware-specific "profiles" for some installation defaults, I just haven't had use for them.
The end-result I'm working toward is providing a service that allows people to generate linux live and install images that have what they want on them already -- I've written a proof-of-concept webapp that installs a package on a live and install cd that I intend to start a kickstarter with[1]. My goal is to allow college kids and newer linux users to create something that will give them access to the system they need or want in case of failure or circumstance, and allow it to perform (if desired) unattended installs.
I think this approach would work too.
I don't think this approach would work well for a "distro-canon" installation media. It works as a service, but I think it's far out of scope for "official" installs -- I would think that a distro-provided install disk should make generic, interactive installation possible and try to ensure that all relevant install situations can be covered while providing sane defaults.
A couple of questions on acu based off the github page:
- Why create a configuration file format? There are many standard formats ranging from simplistic to very complex, and even if go didn't have an implementation of one laying around, it'd probably be simpler (and less confusing/more reliable/quicker) to write a parser for one.
As far as I can tell, not that much information is needed in order to specify what it takes to configure a computer. A list of packages to install, a few changed configuration options and perhaps the addition of a few lines in a few configuration files should do the trick. This could be expressed either as a regular shell script or as some sort of list of declarations. Both could work, but it we imagined there was a webpage with hundreds of these configurations, it would be easier compare configurations and base new configurations on old ones if the file format was declarative. Also, it would be easier to parse for the web-system. The AUR web page struggles to parse the non-declarative PKGBUILD textfiles.
A listing on a web-page and a configuration file are just two different representations of the same data, they just require a transformation. That transformation can be well-defined and simple if you use formats that are already standardized for config files.
Could you give me an example of a declarative format that is significantly easier to parse than (for example), json or yaml or .ini or the "normal" shell-script-thats-just-assignments?
- What, exactly would you have machine-specific configs entail? There's a difference between purpose-specific and hardware-specific ... I feel that the former has more common use cases, but on the other hand a *lot* of the configs and specific packages that I can think of for a "purpose" machine are going to vary wildly from user to user.
...
The personal configuration, like time zone, locale and keyboard layout should only need to be specified once ever and stored as a short string or as some sort of id. I can't could how many times I've specified the same settings, and it should not be needed in our day and age. Once should be enough.
Where are you storing these? Online profiles? How does the tool access them the next time an install needs done? What about for installs that don't have net access? You currently only need to set those once per install anyhow -- I can see some benefit in allowing people to store these, but I feel like it would be out of place in an installation media, unless perhaps as an option if there was a popular configuration-storage service that could be used.
If I was going to try to solve this problem, and this problem specifically, I'd create a site that allowed people to upload that data, and then a package that could sync to/from a machine. Kinda like etckeeper. Something like this probably already exists.
The third and last part would be the information that is specific to the installation in question. Which disk should the installation be on? Is this supposed to be a media server? Should the filesystems handle small files well? Is wireless network preferred? In my opinion, this third part is what users _should_ be worrying about when installing Arch, not selecting keyboard layout for the n'th time and not go hunting for forum posts about how to get the soundcard to work on this particular brand of laptop. Hope this answered your question.
I agree in spirit.
- Why go?
The non-subjective reason for choosing Go, in this connection, is that it easily compiles to a static binary that does not need any dependencies (packages).
This means we need to compile for each platform supported, no?
For an installer, this is great. Also, the execution speed is fast. If there are several developers involved, there are no arguments about tabs vs spaces, since the standard (and tool that formats the code) is clear and it has good modularity.
I don't think the execution speed of the tool is going to be a bottleneck for an installation tool. It's going to be disk and network, unless you're a very poor coder :P.
Also, I never have arguments about tabs vs spaces. I use spaces for all my own stuff, but when entering an existing project or community, it's generally unspoken that you follow the convention that exists for that project or community.
My next plan is to rewrite the code I wrote for the web-interface for machine configurations, but in Go. I'm in the process of moving this to a server that makes this possible.
I'm a little confused as to what's being made possible -- that is, what can't you write right now on your machine with a possible addition of a VM?
Either that or use the PHP-code for AUR as a basis. Or some other solution. Time will tell. (The current PHP-code is bad, but crappy prototypes keeps the momentum up).
I haven't looked at the current PHP code, so I can't comment on it. Lots of deployed code is "bad" though. I'm more of a fan of well-done test-suites than clean code these days, although clean code is always desirable.
In either case: no matter the approach and no matter the authors, I'm looking forward to the next generation of installers.
For sure!
A couple of general comments:
I think you underestimate the complexity of some of the problems here, or overestimate the complexity of some of the existing solutions, particularly in configuration management. A lot of the complexity in software comes from people attempting to create elegant solutions, and then running into all the edge cases that made existing solutions complex, and then attempting to solve them in a probably-less-complete-than-others fashion. It would be a shame to waste a lot of time reinventing the wheel where not necessary.
Also, you do a good job above of separating configuration types in this email. I'm fairly certain that the types that Dieter was referring to elsewhere in the thread were more of the "stuff relevant to a particular user" types where as you include "stuff relevant to a particular machine" in the definition of configuration. I'm having a hard time articulating the specifics of what I'm thinking right now, but I do feel that there are distinctions to be made wrt types of "configurable" things, what they're relevant to, and where it's appropriate to handle them.
There are things relevant to a specific set of hardware. There are things relevant to a purpose for a set of hardware. There are things relevant to a particular user of a set of hardware.
These may have some overlap in them. *In general* though, I tend to think that for generic install media the "specific set of hardware" set is the only one that has a large benefit in being present in an installation tool. The few attempts I've seen at providing purpose-specific "generic" install media still required a silly amount of tweaking afterwards, and user-specific installs require a service that I doubt that many distros would want to maintain, nor do I think that they should be present for canon install-media.
Well, that's my hopefully-sensical two cents for the moment....
arch-releng@lists.archlinux.org