[arch-general] Arch Linux on servers?

[Sébastien Luttringer]

On Tue, Jul 9, 2013 at 12:13 PM, M Saunders <okachi at gmail.com> wrote:
> which has some useful tips. But it'd be interesting to hear from
> people running Arch on production servers, how well it works for them
> and what (if any) problems they've faced.
>

I've 9 personal servers running Archlinux (previously under debian)
and I plan to move about ~250 professional hypervisor under Arch this
year.
Let me share the following experiences with you.

1) Use the minimum set of packages
This will save you from updating useless packages and give you a
better view of what your server use.
As there is only few packages, don't rush to update them when there is
major change on it.

2) Do your sysadmin homework
Before updating, check archlinux.org for announcements.
During update  read pacman output.[1]
After updating, look for pacsave/pacorig/pacnew files.

Supervise your packages. I use munin with the following plugins[2][3]

3) Use a versioned kernel
One of the most wanted expectation on a server is to avoid reboot.
Arch official kernel is too often updated for a server _and_ cannot be
installed without breaking the running kernel (modules mismatch).
To workaround this I build custom kernels, with the version in the
name[4] and I use a meta package[5] which push new version
automatically and clean the old one. So I can update my server, and at
the next reboot the last kernel will be selected.

4) Detect daemon upgrade
When you update your system, some libraries or binaries can be updated
 and your running programs still use the old version.
This give the bad feeling that your software are up-to-date. But it's false.
Of course you can reboot your server to be sure after each update.
It's too long and give the feeling to hunt fly with a tank.

I use the following script[6] which list services (systemd speaking)
which need to be restarted.

# checkservcies -l # list services to restart
# checkservices -r # restart it

5) Detect server reboot
I track my server reboot with the following software[7]. Btw, this is
not a solution for 250 servers.

6) Use your repository to spread your custom packages
For personal packages or taken from AUR, using a custom repository[8}
will simplify your job.
You compile your soft in one place, no need  to have gcc or base-devel
on your servers.
Update is automatically propagate as official repository.
You can easily override official package (not recommended).
You could use a base meta package[9] to have all the basics software
on all your servers.
This will prepare you to become an archlinux TU or Dev.

7) Security
Debian is not more secure because their softwares are old. It's a lie.
Check the number of open flaw in the security bug tracker[10].
If you want to be in a secure environment stay up-to-date, don't use
debian stable, use debian sid. So Archlinux is a good alternative.

Regards,

[1] Please note, that is not a pleasure for a package maintainer to
add a message in his package. So read it.
[2] https://github.com/seblu/archutils/blob/master/archlinux-pacfiles
[3] https://github.com/seblu/archutils/blob/master/archlinux-packages
[4] https://github.com/seblu/archpkg/blob/master/linux-seblu/PKGBUILD
[5] https://github.com/seblu/archpkg/blob/master/linux-seblu-meta/PKGBUILD
[6] https://github.com/seblu/archutils/blob/master/checkservices
[7] https://github.com/seblu/mailboot
[8] https://seblu.net/a/seblu/x86_64/
[9] https://github.com/seblu/archpkg/blob/master/base-seblu/PKGBUILD
[10] https://security-tracker.debian.org/tracker/status/release/stable

--
Sébastien "Seblu" Luttringer
https://www.seblu.net
GPG: 0x2072D77A