[arch-dev-public] Story time: Gerolde breakage

[Aaron Griffin]

What follows below is a recounting of the Great Gerolde Breakage of
'08. Details are provided by eyewitness accounts, and names have been
changed to protect the innocent.

It was a dark and gloomy night, that Thursday. I had come home late,
in a bit of a dour mood, and had been drinking a bit. I sat down at my
computer to check some emails and do some things before bed. When I
concluded my events for the night, I decided it was time to -Syu and
reboot. So I closed everything down except one terminal. The events
below were all internal monologue:

Hmm, Syu time. Woah, that's a lot of packages.... ah well. ENTER
Huh, why do I have apache on this laptop? I'll have to remove that later.
Hmm... mailman? I KNOW I didn't install that... what the? OH MY GOD
THIS IS GEROLDE!

I immediately Ctrl-C'd because I had a "sudo reboot" queued up for
when the -Syu finished. I paused for a moment, kicking myself for
doing that. I checked apache, and it was still running. ftp was
working too.

Ok, good. Exit SSH. Let me -Syu locally.

Re-ssh to gerolde... immediate "permission denied". Oh god. Oh god. Oh god.

Yeah, I Ctrl-C'd before a kerberos upgrade was finished. This broke
ssh, and something else goofy broke mailman.

I emailed Dale, and he fixed SSH in the morning (whew, thanks a lot
Dale). Mailman was a bit harder, but considering everything was down
already, we switched from exim to postfix, as we'd been talking about
it for ages. Hooray for silver linings!

So. It's all back up now. Mailman is working, postfix is working, and
everything else should be in working order. If something is broken,
please let me know.

Sorry for the inconvenience.

Gerolde, hacked by phrak

- Aaron