On Sat, Jun 27, 2020, 19:37 Jelle van der Waa <jelle@vdwaa.nl> wrote:
On 27/06/2020 04:01, Sven-Hendrik Haase via arch-devops wrote:
> On 26.06.20 19:07, Sven-Hendrik Haase wrote:
>> Hey all,
>>
>> During a small routine check I noticed that we have a broken disk on
>> vostok. In fact, according to the log, we've had it for 7 months at
>> least which is a bit embarrassing given that this is RAID1 only and it's
>> also our primary backup box. This really goes to show that we need
>> better monitoring/more meaningful alerts.
>>
>> Anyway, I had Hetzner replace the disk and the array is now rebuilding.
>>
>> Cheers,
>> Sven
>>
>
> The array is now resynced and appears to be happy. Yay!

Thanks for handling and finding the issue sven! However how can our
monitoring not catch this? Especially for gemini it would be nice to
know :-)

Greetings,

Jelle van der Waa

Yeah, well I dunno. That's gonna get really embarrassing if we don't notice for too long at some point.