Hey, just wanted to give a post-mortim on this downtime incident.
At 1am ET on Thursday morning, HTTP requests on either port 80 or 443
started timing out on the server running the mirror.
My automated downtime monitors did not alert me, since the server could
still be pinged, just not accessed on 80/443. Doing some diagnosis this
morning into the issue revealed that the disk had filled, and apache was
not able to respond to requests appropriately.
Suffice to say, I'm going to implement better monitoring in the future for
this case, but the server is back up now and appears to be synching
properly. I will be monitoring this for the next few hours to ensure
stability.