[arch-general] Update to 4.15.8 on dual quad-core box locked on ( 3/16) Install DKMS modules, need help resurecting

[David C. Rankin]

All,

  I experienced a hard lockup during kernel update to 4.15.8 on a Supermicro
Dual Opteron Quad-core box. I've updated this box 50 times without issue, but
something caused a hardlock. The filesystems are on mdadm linux-raid 1 partitions.

  The hardlock occurred after the packages were installed and it was in
postprocessing at:

( 3/16) Install DKMS modules

  Now on boot I receive:

Warning: /lib/modules/4.15.8-1-ARCH/modules.devname not found - ignoring
starting version 237
ERROR: devide `UUID=c7492ac0-e805...` not found. Skipping fsck.
mount: /new_root: can't find UUID UUID=c7492ac0-e805...
You are being dropped into an emergency shell.
sh: can't access tty; job control turned off
[rootfs ]#

(and the box hardlocks)

  So I downloaded the 201803 iso to try and fix the box. I have to boot from
CD, since this box does not boot from USB. So I burn the .iso to CD (making
sure the CD Label is `ARCH_201803`) and boot the box again in attempt to fix it:

  All goes well until...

:: Mounting '/dev/disk/by-label/ARCH_201803' to '/run/archiso/bootmnt'
Waiting 30 seconds for device /dev/disk/by-label/ARCH_201803 ...
ERROR: '/dev/disk/by-label/ARCH_201803' device did not show up after 30
seconds ...
  Falling back to interactive prompt
  You can try to fix the problem manually, log out when you are finished
sh: can't access tty; job control tuned off
[rootfs ]#

(thankfully this prompt is not hardlocked)

This is bizarre, I've created the iso, sha1sums are correct, CD label is
'ARCH_201803', but the iso won't boot. I've researched, but these solutions
don't solve the problem:

https://bbs.archlinux.org/viewtopic.php?id=195671
https://superuser.com/questions/519784/error-installing-arch-linux
https://bugs.launchpad.net/bugs/1318400

  Check /dev/disk from the recovery prompt, there is no "by-label" directory
under /dev/disk to begin with. Attempting to create 'by-label' and softlinking
/dev/sr0 to /dev/disk/by-label/ARCH_201803 does create a series of additional
errors I/O errors concluding with,

mount: /run/archiso/bootmnt: wrong fs type, bad option, bad superblock on
/dev/sr0, missing codepage or helper system ....

So I'm snakebit and need help. I've never had the system lock during kernel
update before and it has left part of the system thinking it has 4.15.7 and
the rest thinking it is 4.15.8 (but the 4.15.8 update never finished)

(1) How do I go about recovering? 4.15.7 was A-OK. I'm not sure what part of
the install is still 4.15.7 and what's 4.15.8. 59 packages were updated,
including the kernel and lts-kernel, but the initramfs was never regenerated
due to the failure at the 'Install DKMS modules' phase. If I can get the
ARCH_201803 install media to boot properly -- what next?

(2) How do I get around the ERROR: '/dev/disk/by-label/ARCH_201803' device did
not show up after 30 seconds problem? The disk label is correct, it's just not
being seen and mounted by the installer to /run/archiso/bootmnt

Any help greatly appreciated.


-- 
David C. Rankin, J.D.,P.E.