[arch-dev-public] Story time: Gerolde breakage

[Dusty Phillips]

2008/8/1 Aaron Griffin <aaronmgriffin at gmail.com>:
> What follows below is a recounting of the Great Gerolde Breakage of
> '08. Details are provided by eyewitness accounts, and names have been
> changed to protect the innocent.
>
> It was a dark and gloomy night, that Thursday. I had come home late,
> in a bit of a dour mood, and had been drinking a bit. I sat down at my
> computer to check some emails and do some things before bed. When I
> concluded my events for the night, I decided it was time to -Syu and
> reboot. So I closed everything down except one terminal. The events
> below were all internal monologue:
>
> Hmm, Syu time. Woah, that's a lot of packages.... ah well. ENTER
> Huh, why do I have apache on this laptop? I'll have to remove that later.
> Hmm... mailman? I KNOW I didn't install that... what the? OH MY GOD
> THIS IS GEROLDE!
>
> I immediately Ctrl-C'd because I had a "sudo reboot" queued up for
> when the -Syu finished. I paused for a moment, kicking myself for
> doing that. I checked apache, and it was still running. ftp was
> working too.
>
> Ok, good. Exit SSH. Let me -Syu locally.
>
> Re-ssh to gerolde... immediate "permission denied". Oh god. Oh god. Oh god.
>
> Yeah, I Ctrl-C'd before a kerberos upgrade was finished. This broke
> ssh, and something else goofy broke mailman.
>
> I emailed Dale, and he fixed SSH in the morning (whew, thanks a lot
> Dale). Mailman was a bit harder, but considering everything was down
> already, we switched from exim to postfix, as we'd been talking about
> it for ages. Hooray for silver linings!
>
> So. It's all back up now. Mailman is working, postfix is working, and
> everything else should be in working order. If something is broken,
> please let me know.
>
> Sorry for the inconvenience.
>
> Gerolde, hacked by phrak

Well written and all but I seriously think we should forward this to
Skoal for some bardification. =)

Well done! Now you can get the hacked by phrak t-shirt!

Dusty