Hi Mike, On 2013-07-09 11:13, M Saunders wrote:

I'm writing a feature about Arch for Linux Format, a UK-based newsstand Linux magazine. I've been using Arch myself for a while for testing new app releases, and it's brilliant for that purpose.

I'm still left wondering though: who uses it on production servers? I mean, the distro's overall simplicity and trimmed-down base installation are plus points here, but surely a rolling release poses problems. After installation you just want security and critical bug fix updates for software, and not major version bumps, right?

I only use it to manage small production environments (although these are not corporate deployments). IMO it is suitable for servers in limited cases, where neither of the following are true: - The server will be running obscure services with limited eyes-on - You will be running a lot of services I ran my entire personal development infrastructure on Arch Linux for a good while, and only stopped because I've outsourced it all now so there's no need for the installation in the first place -- that being, a CI, git hosting, HTTP server, a few other things.

www.archserver.org seems to be on hold, and I've also seen this page: https://wiki.archlinux.org/index.php/Enhancing_Arch_Linux_Stability

The only thing I did was have linux-lts installed in addition to the linux package. I never had a problem, and I ran that server for years.

which has some useful tips. But it'd be interesting to hear from people running Arch on production servers, how well it works for them and what (if any) problems they've faced.

I never had a problem that was due to the packaging, which limits the opportunities for breakage to upstream (mainly). That just means you have to have an eye on things. I was subscribed to the announcement mailing lists for all the stuff I was using (Jenkins, nginx, git, cgit, et al). If you're running a very complex server then it can become a bit complicated to go down this road, especially if you're used to your distribution providing deprecation guidance for you. Generally things don't just "break" on Arch, there is [testing] after all. If things break, it's usually because people didn't pay attention to configuration changes or important details prior to upgrading. If you aren't willing to keep an eye on upstream and on the Arch mailing lists, it will not end well. Good luck with the article. I'm not living in the UK any more, else I would still be buying Linux Format. Chris

Obviously, I am biased, but... [2013-07-09 11:13:08 +0100] M Saunders:

After installation you just want security and critical bug fix updates for software, and not major version bumps, right?

If you are prepared to stick to a given feature set, maybe. Then, you might be able to achieve near-absolute stability with a Debian-stable- like system. Now, in my experience, that makes any upgrade extremely painful, so I would definitely not recommend this approach to anyone with an interest in recent software. On the other hand, Arch's continuous flow of upgrades proves to be quite reliable, so long as the system is updated with a minimum degree of attention. Besides, it avoids duplicating upstream work at the distro level (such as backporting security fixes): such work can never be perfect, and has indeed been the cause of major problems in the past (Debian's openssl entropy issue likely being the most famous). I run two small-scale servers with Arch, which I only upgrade with care and when I have available time to fix potential problems. My upgrading frequency goes from a couple of times a week to once a month. About once a year, I hit an upgrade that is not straightforward, and it takes me an hour or two to fix arising issue and perform it; that is about it. -- Gaetan

On 09.07.2013 12:13, M Saunders wrote:

I'm still left wondering though: who uses it on production servers? I mean, the distro's overall simplicity and trimmed-down base installation are plus points here, but surely a rolling release poses problems. After installation you just want security and critical bug fix updates for software, and not major version bumps, right?

I've seen at least 2 or 3 kernel exploits that were mitigated by newer kernel versions (which we had, debian didn't). Obviously there have been other issues which could only be exploited in more recent kernel versions which didn't affect debian. Then there are those issues where there is a patch but no new release so it might not get fixed in arch until the next release (no security team nor policy for such patches). In terms of updating breakage it doesn't matter what you use, updating will eventually result in breakage, but if you know the system well enough you will have a much easier time fixing it. I had a case where a few debian servers got upgraded after something like 1.5 years and spamassassin suddenly used a lot more resources. Since basically every package jumped lots of versions finding the package responsible for that was kind of impossible so they just bought a bunch more servers to deal with the higher load. On arch you could probably narrow it down and fix the software. Might not be cheaper and might not be what you want (cool new feature causing the issue maybe), but at least you aren't left in the dark. I'm not sure if either distro is more time intensive, I think you will just spend your time differently. Also investing time in anything will result in knowledge so I'm not sure if that's a bad thing. If you don't know what you are doing, don't run a server with arch. But then you shouldn't be running a server in that case anyway. As Allan once said: "If you have to ask, then no". I'd say neither solution (rolling-release vs "stable and secure") is better, they are just different. Get to know your tool (distro) and decide for yourself.

On 07/09/13 06:13, M Saunders wrote:

Hi all,

I'm writing a feature about Arch for Linux Format, a UK-based newsstand Linux magazine. I've been using Arch myself for a while for testing new app releases, and it's brilliant for that purpose.

I'm still left wondering though: who uses it on production servers? I mean, the distro's overall simplicity and trimmed-down base installation are plus points here, but surely a rolling release poses problems. After installation you just want security and critical bug fix updates for software, and not major version bumps, right?

www.archserver.org seems to be on hold, and I've also seen this page: https://wiki.archlinux.org/index.php/Enhancing_Arch_Linux_Stability

which has some useful tips. But it'd be interesting to hear from people running Arch on production servers, how well it works for them and what (if any) problems they've faced.

Thanks! Mike

I run two production Arch server boxes. I update them each about twice a month unless I hear about a vulnerability in a critical internet facing component on the system (WordPress, etc). I just make sure to do it on Fridays and Saturdays. If something goes wrong, it can often take a little more time to back out than a traditional distribution. As a security professional, I appreciate always being up to date and having more detailed control over exactly what code is on my system. I have massively reduced my exposure to flaws in packages included by the distro that I never needed or even wanted. Get Linux? Get Arch Linux. Not so sure about Linux? Call Rackspace. Chester

On Tue, Jul 9, 2013 at 12:13 PM, M Saunders <okachi@gmail.com> wrote:

which has some useful tips. But it'd be interesting to hear from people running Arch on production servers, how well it works for them and what (if any) problems they've faced.

I've 9 personal servers running Archlinux (previously under debian) and I plan to move about ~250 professional hypervisor under Arch this year. Let me share the following experiences with you. 1) Use the minimum set of packages This will save you from updating useless packages and give you a better view of what your server use. As there is only few packages, don't rush to update them when there is major change on it. 2) Do your sysadmin homework Before updating, check archlinux.org for announcements. During update read pacman output.[1] After updating, look for pacsave/pacorig/pacnew files. Supervise your packages. I use munin with the following plugins[2][3] 3) Use a versioned kernel One of the most wanted expectation on a server is to avoid reboot. Arch official kernel is too often updated for a server _and_ cannot be installed without breaking the running kernel (modules mismatch). To workaround this I build custom kernels, with the version in the name[4] and I use a meta package[5] which push new version automatically and clean the old one. So I can update my server, and at the next reboot the last kernel will be selected. 4) Detect daemon upgrade When you update your system, some libraries or binaries can be updated and your running programs still use the old version. This give the bad feeling that your software are up-to-date. But it's false. Of course you can reboot your server to be sure after each update. It's too long and give the feeling to hunt fly with a tank. I use the following script[6] which list services (systemd speaking) which need to be restarted. # checkservcies -l # list services to restart # checkservices -r # restart it 5) Detect server reboot I track my server reboot with the following software[7]. Btw, this is not a solution for 250 servers. 6) Use your repository to spread your custom packages For personal packages or taken from AUR, using a custom repository[8} will simplify your job. You compile your soft in one place, no need to have gcc or base-devel on your servers. Update is automatically propagate as official repository. You can easily override official package (not recommended). You could use a base meta package[9] to have all the basics software on all your servers. This will prepare you to become an archlinux TU or Dev. 7) Security Debian is not more secure because their softwares are old. It's a lie. Check the number of open flaw in the security bug tracker[10]. If you want to be in a secure environment stay up-to-date, don't use debian stable, use debian sid. So Archlinux is a good alternative. Regards, [1] Please note, that is not a pleasure for a package maintainer to add a message in his package. So read it. [2] https://github.com/seblu/archutils/blob/master/archlinux-pacfiles [3] https://github.com/seblu/archutils/blob/master/archlinux-packages [4] https://github.com/seblu/archpkg/blob/master/linux-seblu/PKGBUILD [5] https://github.com/seblu/archpkg/blob/master/linux-seblu-meta/PKGBUILD [6] https://github.com/seblu/archutils/blob/master/checkservices [7] https://github.com/seblu/mailboot [8] https://seblu.net/a/seblu/x86_64/ [9] https://github.com/seblu/archpkg/blob/master/base-seblu/PKGBUILD [10] https://security-tracker.debian.org/tracker/status/release/stable -- Sébastien "Seblu" Luttringer https://www.seblu.net GPG: 0x2072D77A

On 07/09/2013 05:13 AM, M Saunders wrote:

Hi all,

I'm writing a feature about Arch for Linux Format, a UK-based newsstand Linux magazine. I've been using Arch myself for a while for testing new app releases, and it's brilliant for that purpose.

I'm still left wondering though: who uses it on production servers? I mean, the distro's overall simplicity and trimmed-down base installation are plus points here, but surely a rolling release poses problems. After installation you just want security and critical bug fix updates for software, and not major version bumps, right?

www.archserver.org seems to be on hold, and I've also seen this page: https://wiki.archlinux.org/index.php/Enhancing_Arch_Linux_Stability

which has some useful tips. But it'd be interesting to hear from people running Arch on production servers, how well it works for them and what (if any) problems they've faced.

Thanks! Mike

M, I have run two offices with Arch production servers. The primary consideration for looking at Arch for production boxes was the rolling-release model which promised to eliminate the forced reinstall from version X to version X.1 in traditional release based distros. Arch was stellar from 2009 through mid 2012 in providing seamless updates without an excessive amount of time-robbing intervention required. Stability of the distro in this regard was a primary consideration in remaining with Arch. Beginning with the clib change and continuing through the initscript->systemd migration, the changes really began to impact Arch's suitability for use in production. In my case, the last killer was the loss of dmraid (nvidia based controller) support which was not provided by mdraid. Whereas before, there had always been a stable upgrade path, it ended at that point. Arch is just as suitable for server use as any other distro and is rock-solid. However, Arch's priority to maintain a cutting-edge distro far outweighs any thought of providing a broad stable upgrade path for all currently supported hardware. If Arch has a chance to move forward to the latest "released" version of a package that will break hardware support for some, then those 'some' are just out of luck. This is something to be aware of before committing to build a production server platform around Arch. Reinstalling a desktop if a change is necessary is painless compared to tuning a new server. -- David C. Rankin, J.D.,P.E.