[arch-general] Strange mdadm fail to activate root raid1 array - resulting in loss of libraries and pacman data?

[David C. Rankin]

All,

   I saw TB's patch posts to the kernel-raid mailing list. I believe I 
experienced the same systemd-mdadm error which resulted in ~440 libraries being 
left as 0-byte files and pacman losing all reference to what files where owned 
by what packages. In my case dmesg showed the system never made any attempt to 
activate sda7 as part of the root array (md1) on boot. Following a --fail, 
--remove and --add to re-add sda7 worked as advertised. However, on next reboot, 
grub was found and launched the kernel, only to have the system crash due to 
/usr/lib/libkmod.so "file too short" (0-byte).

   Attempting to re-install the kernel to attempt to address the issue resulted in:

[2015-08-31 15:58] [ALPM-SCRIPTLET] ==> WARNING: No modules were added to the 
image. This is probably not what you want.
[2015-08-31 15:58] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: 
/boot/initramfs-linux.img
[2015-08-31 15:58] [ALPM-SCRIPTLET] ==> WARNING: errors were encountered during 
the build. The image may not be complete.
[2015-08-31 15:58] [ALPM-SCRIPTLET] ==> Building image from preset: 
/etc/mkinitcpio.d/linux.preset: 'fallback'
[2015-08-31 15:58] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -c 
/etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
[2015-08-31 15:58] [ALPM-SCRIPTLET] ==> Starting build: 4.1.6-1-ARCH
[2015-08-31 15:58] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2015-08-31 15:58] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2015-08-31 15:58] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2015-08-31 15:58] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2015-08-31 15:58] [ALPM-SCRIPTLET] ==> ERROR: module not found: `eata'
[2015-08-31 15:58] [ALPM-SCRIPTLET] ==> ERROR: module not found: `a100u2w'
[2015-08-31 15:58] [ALPM-SCRIPTLET] ==> ERROR: module not found: `3w_sas'
[2015-08-31 15:58] [ALPM-SCRIPTLET] ==> ERROR: module not found: `bnx2fc'
<snip> lots of modules not found

   Booting the install media, chrooting and investigating revealed some 440 
libraries in /usr/lib that were 0-byte. pacman had also lost its package-file 
database. E.g.

   # pacman -Ql unixodbc
   # (showed no files)

all of this occurred with mdadm 3.3.2-2, which is still the current package.

   TB's original post at the kernel raid list regarding the patch is:

[PATCH 0/2] mdadm: Fix udev rule interaction with systemd,
http://www.spinics.net/lists/raid/msg49678.html

   The explanation of the losses on my system posted to the list are:

Re-add of raid1 drive resulted in strange loss of data on Archlinux?
http://www.spinics.net/lists/raid/msg49704.html

   Are these the symptoms (or related symptoms) that the TB's patch is intended 
to address?

   This has only occurred once since the current mdadm was installed back on 
2015-05-26. What is the recommended work-around until a fix is in place? 
(validate all arrays with /proc/mdstat) But then how to handle a degraded boot? 
I did --fail, --remove --add, and ended up with a mess. Is there some other 
suggested way to handle this?


-- 
David C. Rankin, J.D.,P.E.