On Mon, Mar 14, 2011 at 10:43 AM, C Anthony Risinger wrote:

...

On Mon, Mar 14, 2011 at 11:15 AM, Thomas S Hatch wrote:

...

I am posting this to venture an opinion, and see if anyone has any ideas on how best to handle this situation.

I am working with the puppet developers to help improve Arch Linux support in puppet (for those of you unfamiliar with puppet, it is an amazingly crucial component in datacenter management - http://www.puppetlabs.com ).

Puppet can automate the starting and stopping of systems services as well as managing if they are set to start on boot. In most Linux distributions flagging a system service to start on boot is done with an application

...

...

chkconfig and the order in which the system services start up is pre determined by the distribution. Arch Linux uses a much simpler approach to system services, the DAEMONS array in the rc.conf. The problem faced with an automation system in

...

...

scope is that the boot order of services is not predetermined.

Right now the best idea we have been able to come up with is to have

On Mon, Mar 14, 2011 at 11:32 AM, C Anthony Risinger wrote: like this puppet

...

...

simply append the named services to the end of the DAEMONS array in the rc.conf, but I wanted to ask the community if anyone had any alternative ideas on how this could be done.

The ticket containing the discussion on the matter can be found here: https://projects.puppetlabs.com/issues/6697

i haven't had a chance to really sink time into it quite yet, but i have a moderately sized network i manage at w3rk and implementing puppet is high on my list of improvements -- am also considering using arch for hypervisor deployments due to the simplicity of spinning a custom environment.

anyway ... could we just bypass rc.conf and add some additional logic to initscripts to load from a directory? example:

/et/rc.daemons/@002-ntpd -> /etc/rc.d/ntpd /et/rc.daemons/@004-sshd -> /etc/rc.d/sshd

initscripts could `echo /et/rc.daemons/*` if the DAEMONS variable is missing? does that simplify things or cause more issue?

the other idea i had is a little more heavyweight, but we could use the augeas tool ( http://augeas.net/ ) to perform the rc.conf update directly; i'm fairly confident either that tool or something very similar is the future of all automated config editing, at least i hope it is, because `sed` is just a bad bad solution. i don't know puppets plans, but i wouldn't be surprised to see puppet depend on it, because it would be a good fit.

... oooooor we use systemd which provides things like `systemctl enable foo` ...

i have deployed it to my private servers and my various laptops/netbooks/etc with great success; quite frankly, it's pretty awesome, and simple to manage ... should be hitting the main repos any day now (if not already?) due to recent inclusion of util-linux-2.19.

not exactly OOTB though (yet ... bwa hahahahaha >:-)

C Anthony

As for augeas, puppet will not have any problems parsing the array in the rc.conf file. I agree with Aaron, that adding extra components to the Arch runlevel would be something we should avoid. As for systemd, after working with the insides of Upstart, I am VERY cautious about systemd, and I am honestly a little upset about it, I don't like the idea that Red Hat changing the runlevel space could pressure us into having to use systemd, I like the Arch runlevel the way it is, and I think that managing service startup with something like systemd will most likely be a bad thing. But I have not spent as much time as I should with systemd to make a final conclusion there. But regardless, this should support the Arch style runlevel. I do think that if it is possible to determine the ordering based on the requires statements of other services that would be great, or maybe even to implicitly state the index of the service: index => 4, would set the value as the 4th thing to start.

On Mon, Mar 14, 2011 at 11:09 AM, Dieter Plaetinck wrote:

On Mon, 14 Mar 2011 10:58:19 -0600 Thomas S Hatch wrote:

...

But regardless, this should support the Arch style runlevel.

maybe... in theory it's possible that in a month we switch to systemd as official init system. this will probably not happen (soon), but just saying.

...

I do think that if it is possible to determine the ordering based on the requires statements of other services that would be great

the key thing imho is that currently on a normal, single Arch install it's the admin who configures the ordering, very explicitly. Automatically guessing an implicit ordering based on whatever heuristics is not how Arch currently does it, and probably neither should puppet. I don't have any puppet experience, but is it not possible to just make the admin configure the order in puppet? Or even just define $DAEMONS in puppet? It's remarkable that such a simple configuration syntax in rc.conf (an ordered list of daemons) is causing these issues, it seems as if puppet tries to abstract (too much?) and now we must work around the limitations of that abstraction?

BTW: I was right across Nigel at the devops dinner in Brussels. small world :)

Dieter

Dieter, the problem is not so much in parsing the DAEMONS array, as it is the fact that in puppet the services are configured from a different logical perspective, rather than configuring what services are to start, you define services individually and then "realize" certain services for certain servers, this makes it hard to implicitly define the daemons array. On the other hand, if require statements could be used to determine the order of the services in the daemons array, then the order would still be explicitly determined by the admin. I mentioned the idea of having an index => param, maybe we could have a before => Service['foo'], or before => foo, to specify that the service should run before the other service. of course the require statement could still fulfill that.

On Mon, Mar 14, 2011 at 11:37 AM, Dieter Plaetinck wrote:

On Mon, 14 Mar 2011 11:26:08 -0600 Thomas S Hatch wrote:

...

On Mon, Mar 14, 2011 at 11:09 AM, Dieter Plaetinck wrote:

...

On Mon, 14 Mar 2011 10:58:19 -0600 Thomas S Hatch wrote:

...

But regardless, this should support the Arch style runlevel.

maybe... in theory it's possible that in a month we switch to systemd as official init system. this will probably not happen (soon), but just saying.

...

I do think that if it is possible to determine the ordering based on the requires statements of other services that would be great

the key thing imho is that currently on a normal, single Arch install it's the admin who configures the ordering, very explicitly. Automatically guessing an implicit ordering based on whatever heuristics is not how Arch currently does it, and probably neither should puppet. I don't have any puppet experience, but is it not possible to just make the admin configure the order in puppet? Or even just define $DAEMONS in puppet? It's remarkable that such a simple configuration syntax in rc.conf (an ordered list of daemons) is causing these issues, it seems as if puppet tries to abstract (too much?) and now we must work around the limitations of that abstraction?

BTW: I was right across Nigel at the devops dinner in Brussels. small world :)

Dieter

Dieter, the problem is not so much in parsing the DAEMONS array, as it is the fact that in puppet the services are configured from a different logical perspective, rather than configuring what services are to start, you define services individually and then "realize" certain services for certain servers, this makes it hard to implicitly define the daemons array.

On the other hand, if require statements could be used to determine the order of the services in the daemons array, then the order would still be explicitly determined by the admin.

I mentioned the idea of having an index => param, maybe we could have a before => Service['foo'],

or

before => foo,

to specify that the service should run before the other service.

of course the require statement could still fulfill that.

maybe, instead of tens of verbose require/verbose/... statements, how about 1 long list of all possible initscripts the admin will ever use (or maybe even of all initscripts in Arch, less duplication of efforts), and then, this list can be consulted to know the order of the daemons which you want in rc.conf

Just throwing the idea out there. Dieter

I think that maintaining a list like that in Arch would be a good idea, or at least some sort of dependency document or application, say a small program that you ask it what other services are required for service x and it returns those services? This would also be useful in debugging ordering problems in an admin may have in the rc.conf. What do you think?

Append is okay (most likely dependencies are already in place for a service), but it would be nice to specify dependencies / ensure necessary services are running. We should allow the module writer / administrator to specify these dependencies so they can specify custom depends/etc. The logic should allow one to specify logical "or" statements (say wicd vs network manager vs netcfg vs network) and should try to place the puppet configured service after all of its dependencies adding them to the line if possible (OR statements may make this hard... perhaps a default to add if one isn't present?) or producing some sort of error/exception. The point should be to give the admin full control of what happens to his / her rc.conf ;) -Lee On Mon, Mar 14, 2011 at 12:58, Thomas S Hatch wrote:

On Mon, Mar 14, 2011 at 10:43 AM, C Anthony Risinger wrote:

...

...

On Mon, Mar 14, 2011 at 11:15 AM, Thomas S Hatch wrote:

...

I am posting this to venture an opinion, and see if anyone has any ideas on how best to handle this situation.

I am working with the puppet developers to help improve Arch Linux support in puppet (for those of you unfamiliar with puppet, it is an amazingly crucial component in datacenter management - http://www.puppetlabs.com ).

Puppet can automate the starting and stopping of systems services as well as managing if they are set to start on boot. In most Linux distributions flagging a system service to start on boot is done with an application

...

...

chkconfig and the order in which the system services start up is pre determined by the distribution. Arch Linux uses a much simpler approach to system services, the DAEMONS array in the rc.conf. The problem faced with an automation system in

...

...

scope is that the boot order of services is not predetermined.

Right now the best idea we have been able to come up with is to have

On Mon, Mar 14, 2011 at 11:32 AM, C Anthony Risinger wrote: like this puppet

...

...

simply append the named services to the end of the DAEMONS array in the rc.conf, but I wanted to ask the community if anyone had any alternative ideas on how this could be done.

The ticket containing the discussion on the matter can be found here: https://projects.puppetlabs.com/issues/6697

i haven't had a chance to really sink time into it quite yet, but i have a moderately sized network i manage at w3rk and implementing puppet is high on my list of improvements -- am also considering using arch for hypervisor deployments due to the simplicity of spinning a custom environment.

anyway ... could we just bypass rc.conf and add some additional logic to initscripts to load from a directory?  example:

/et/rc.daemons/@002-ntpd -> /etc/rc.d/ntpd /et/rc.daemons/@004-sshd -> /etc/rc.d/sshd

initscripts could `echo /et/rc.daemons/*` if the DAEMONS variable is missing? does that simplify things or cause more issue?

the other idea i had is a little more heavyweight, but we could use the augeas tool ( http://augeas.net/ ) to perform the rc.conf update directly; i'm fairly confident either that tool or something very similar is the future of all automated config editing, at least i hope it is, because `sed` is just a bad bad solution.  i don't know puppets plans, but i wouldn't be surprised to see puppet depend on it, because it would be a good fit.

... oooooor we use systemd which provides things like `systemctl enable foo` ...

i have deployed it to my private servers and my various laptops/netbooks/etc with great success; quite frankly, it's pretty awesome, and simple to manage ... should be hitting the main repos any day now (if not already?) due to recent inclusion of util-linux-2.19.

not exactly OOTB though (yet ... bwa hahahahaha >:-)

C Anthony

As for augeas, puppet will not have any problems parsing the array in the rc.conf file. I agree with Aaron, that adding extra components to the Arch runlevel would be something we should avoid.

As for systemd, after working with the insides of Upstart, I am VERY cautious about systemd, and I am honestly a little upset about it, I don't like the idea that Red Hat changing the runlevel space could pressure us into having to use systemd, I like the Arch runlevel the way it is, and I think that managing service startup with something like systemd will most likely be a bad thing. But I have not spent as much time as I should with systemd to make a final conclusion there. But regardless, this should support the Arch style runlevel.

I do think that if it is possible to determine the ordering based on the requires statements of other services that would be great, or maybe even to implicitly state the index of the service:

index => 4,

would set the value as the 4th thing to start.

-- Lee Burton lburton@mrow.org 301 910 0246

On Mon, Mar 14, 2011 at 12:29 PM, Thomas S Hatch wrote:

On Mon, Mar 14, 2011 at 11:24 AM, Lee Burton wrote:

...

Append is okay (most likely dependencies are already in place for a service), but it would be nice to specify dependencies / ensure necessary services are running.  We should allow the module writer / administrator to specify these dependencies so they can specify custom depends/etc.  The logic should allow one to specify logical "or" statements (say wicd vs network manager vs netcfg vs network) and should try to place the puppet configured service after all of its dependencies adding them to the line if possible (OR statements may make this hard... perhaps a default to add if one isn't present?) or producing some sort of error/exception.  The point should be to give the admin full control of what happens to his / her rc.conf ;) -Lee

 I agree, the admin should have the ability to manage the service loading sequence. I think that this could be done with require statements, simply saying that if a service requires another service then it is placed at an earlier index in the DAEMON array.

yeah but then you have the (common) condition where services are not connected to the same hierarchy of dependencies, but you sinply want one after/before the other for whatever reason. systemd handles this with Require/Conflicts, AND Before/After keywords (as you earlier mentioned) ... because they are orthogonal. Require/Conflicts is traditional deps, Before/After is essentially "should these two services happen to be started at the same time, do X before Y". it doesn't matter if puppet actually starts them one at a time; the order is ambiguous. in short, you'd need both unless you want things to get fuzzy w/Requires only ... ... but at that point your replicating the existing, _working_, functionality of systemd; a problem that is being solved by a much larger group at a widespread level (eg. upstream), with the intention of eradicating all this nonsense amongst distros. i'm not trying to beat the drums here, just saying that it's nothing like upstart, and IMO ... is a welcomed addition ... as a _long_ overdue face lift to the crusty/error-prone black-box bash gigantamatron we call "the init process" (and while it does work well enough, we _do_ want to move forward, yes?). anyways ... i'm really trying not to go on a tangent here, but you explicitly say: "you define services individually and then "realize" certain services [on the target server]" and that's exactly what systemd would do for you -- with no additional work on puppet or arch's end; i'm just trying to suggest the best tool for the job, that's all, even if it's not (yet/ever) officially endorsed by the arch high commanders ... i'm a lazy [pragmatic] developer/sysadmin ... and the init process simply is destine for change. ... yes, yes -- i'll just stand aside and listen from this point on :-) C Anthony

Hi Thomas, On Mon, Mar 14, 2011 at 7:07 PM, Thomas S Hatch wrote:

I think that the unfolding systemd issues here and the fact that we may be required to move to systemd might rewrite this problem altogether, in which case systemd would translate ordering and the order of services as they are placed in the daemons array may be irrelevant.

I will do some more research.

I don't think there are any immediate plans to move Arch to systemd, systemd might soon move to [community] though. However, if you are interested in knowing more about how systemd works on Arch, let me know as I'd be happy to help. There is a wiki here: http://wiki.archlinux.org/index.php/Systemd. I'd like to point out one thing which might not be obvious at first glance: systemd has support for the DAEMONS array in rc.conf (partially written by me). However, when this is used, it might happen that systemd will not know the required ordering of daemons, in which case it will fall back to using the order from the DAEMONS array (so if this is incorrect you might get trouble). To be more specific: when systemd has a native service file for a daemon, this will allow it to figure out the dependencies by itself and the position of the service in the DAEMONS array is ignored. If it does not have the service file for a given daemon, it will generate a default one, i.e. it will assume the ordering of the DAEMONS array is correct and order the daemon after the previous non-background daemon in the array. All of this can be avoided by not relying on the DAEMONS array at all, but just adding generic systemd support to Puppet (if it is not there already), and then adding the needed service files to our systemd-arch-units package (the latter is very easy, and I'd be happy to help). Cheers, Tom