On Sat, Jun 27, 2020, 19:37 Jelle van der Waa
On 27/06/2020 04:01, Sven-Hendrik Haase via arch-devops wrote:
On 26.06.20 19:07, Sven-Hendrik Haase wrote:
Hey all,
During a small routine check I noticed that we have a broken disk on vostok. In fact, according to the log, we've had it for 7 months at least which is a bit embarrassing given that this is RAID1 only and it's also our primary backup box. This really goes to show that we need better monitoring/more meaningful alerts.
Anyway, I had Hetzner replace the disk and the array is now rebuilding.
Cheers, Sven
The array is now resynced and appears to be happy. Yay!
Thanks for handling and finding the issue sven! However how can our monitoring not catch this? Especially for gemini it would be nice to know :-)
Greetings,
Jelle van der Waa
Yeah, well I dunno. That's gonna get really embarrassing if we don't notice for too long at some point.