23 Sep
2017
23 Sep
'17
1:15 p.m.
Hey, just wanted to give a post-mortim on this downtime incident. At 1am ET on Thursday morning, HTTP requests on either port 80 or 443 started timing out on the server running the mirror. My automated downtime monitors did not alert me, since the server could still be pinged, just not accessed on 80/443. Doing some diagnosis this morning into the issue revealed that the disk had filled, and apache was not able to respond to requests appropriately. Suffice to say, I'm going to implement better monitoring in the future for this case, but the server is back up now and appears to be synching properly. I will be monitoring this for the next few hours to ensure stability.