[arch-general] Arch Linux on servers?

[Paul Gideon Dann]

On Wednesday 10 Jul 2013 13:59:07 Sébastien Luttringer wrote:
> 3) Use a versioned kernel
> One of the most wanted expectation on a server is to avoid reboot.
> Arch official kernel is too often updated for a server _and_ cannot be
> installed without breaking the running kernel (modules mismatch).
> To workaround this I build custom kernels, with the version in the
> name[4] and I use a meta package[5] which push new version
> automatically and clean the old one. So I can update my server, and at
> the next reboot the last kernel will be selected.
> 
> 4) Detect daemon upgrade
> When you update your system, some libraries or binaries can be updated
>  and your running programs still use the old version.
> This give the bad feeling that your software are up-to-date. But it's false.
> Of course you can reboot your server to be sure after each update. It's too
> long and give the feeling to hunt fly with a tank.
> 
> I use the following script[6] which list services (systemd speaking)
> which need to be restarted.
> 
> # checkservcies -l # list services to restart
> # checkservices -r # restart it
> 
> 6) Use your repository to spread your custom packages
> For personal packages or taken from AUR, using a custom repository[8}
> will simplify your job.
> You compile your soft in one place, no need  to have gcc or base-devel
> on your servers.
> Update is automatically propagate as official repository.
> You can easily override official package (not recommended).
> You could use a base meta package[9] to have all the basics software
> on all your servers.
> This will prepare you to become an archlinux TU or Dev.

The above three points in particular are incredibly insightful; thank you.  
This is definitely helpful to have in the back of my head for if the stakes 
are raised for the machines I'm responsible for.

FYI, for monitoring my servers I use a combination of Monit and Ganglia, which 
I'm pretty happy with.  I wish Ganglia had the ability to notify on certain 
conditions as some other monitoring tools can, but that's not a biggie, as 
that can be dealt with with cron or Monit if necessary.

Some of the functionality I used Monit for became redundant when Systemd came 
along: most importantly ensuring that processes were still running and hadn't 
crashed.  Systemd is capable of re-starting failed services, so long as it's 
enabled in the unit file, but it doesn't (yet) have any notification 
capability, which is a shame.  I also use a cron job to check for any systemd 
units that are in a failed state.  (In case something unusual happens that 
Monit isn't looking for.)

Paul