Hello all! tl;dr version: I've got some local filesystems, all within the same LVM LV, some of which will fail to mount on boot, requiring me to do so by hand instead. Which filesystems don't mount can vary from boot to boot (and sometimes they all work), and the ones that *don't* mount aren't even mentioned anywhere by systemctl/journalctl -- I would expect that if the mount was *attempted* but failed, I'd see some logs. Anyone have a clue about this? Many details follow: So I've got some admittedly-improperly-placed filesystems that I mount into /usr/local, specifically: /usr/local/dv, /usr/local/winex, and /usr/local/vmware (in addition to /usr/local itself being its own filesystem). I've got them all in /etc/fstab basically identical to each other:
# grep /usr/local /etc/fstab /dev/mapper/arrakis-local /usr/local ext4 noatime 1 2 /dev/mapper/arrakis-winex /usr/local/winex ext4 noatime 1 2 /dev/mapper/arrakis-vmware /usr/local/vmware ext4 noatime 1 2 /dev/mapper/arrakis-dv /usr/local/dv ext4 noatime 1 2
Those all exist inside the same LVM LV, as you might guess from the device paths. I've noticed that on initial boot, I don't always get all of them properly mounted, and I've got to manually mount them myself. For instance, after a reboot just now, /usr/local/dv did not get mounted:
# mount | grep /usr/local /dev/mapper/arrakis-local on /usr/local type ext4 (rw,noatime) /dev/mapper/arrakis-vmware on /usr/local/vmware type ext4 (rw,noatime) /dev/mapper/arrakis-winex on /usr/local/winex type ext4 (rw,noatime,stripe=27879)
Or, looking at it via systemctl:
# systemctl list-units --all | grep usr-local usr-local-dv.mount loaded inactive dead /usr/local/dv usr-local-vmware.mount loaded active mounted /usr/local/vmware usr-local-winex.mount loaded active mounted /usr/local/winex usr-local.mount loaded active mounted /usr/local
Querying the usr-local-dv unit directly, I don't see anything particularly telling:
# systemctl status usr-local-dv.mount ○ usr-local-dv.mount - /usr/local/dv Loaded: loaded (/etc/fstab; generated) Active: inactive (dead) Where: /usr/local/dv What: /dev/mapper/arrakis-dv Docs: man:fstab(5) man:systemd-fstab-generator(8)
And there's no mention of the "dv" mount in journalctl either:
# journalctl -b 0 | grep -E '(usr-local|/usr/local)' Jul 19 14:03:13 arrakis systemd[1]: usr-local.mount: Directory /usr/local to mount over is not empty, mounting anyway. Jul 19 14:03:13 arrakis systemd[1]: Mounting /usr/local... Jul 19 14:03:13 arrakis systemd[1]: Mounted /usr/local. Jul 19 14:03:13 arrakis systemd[1]: Mounting /usr/local/vmware... Jul 19 14:03:13 arrakis systemd[1]: Mounting /usr/local/winex... Jul 19 14:03:13 arrakis systemd[1]: Mounted /usr/local/vmware. Jul 19 14:03:13 arrakis systemd[1]: Mounted /usr/local/winex.
(Obvs. I've got some data in /usr/local on the root filesystem which gets mounted over, but that doesn't interfere with the /usr/local mount at all.) I even checked the device creation times, to see if maybe /dev/mapper/arrakis-dv was created too "late" or something, but it seems as though it was actually the first of that bunch:
# ls -lrt --full-time /dev/mapper/arrakis-{local,dv,vmware,winex} lrwxrwxrwx 1 root root 7 2021-07-19 14:03:12.406271238 -0500 /dev/mapper/arrakis-dv -> ../dm-4 lrwxrwxrwx 1 root root 7 2021-07-19 14:03:13.649604518 -0500 /dev/mapper/arrakis-winex -> ../dm-5 lrwxrwxrwx 1 root root 7 2021-07-19 14:03:13.696271181 -0500 /dev/mapper/arrakis-local -> ../dm-0 lrwxrwxrwx 1 root root 7 2021-07-19 14:03:13.809604509 -0500 /dev/mapper/arrakis-vmware -> ../dm-3
As I mentioned above, I can mount it just fine manually, either with `mount` or `systemctl`. Here's me mounting it w/ `mount`, and confirming its status w/ `systemctl`:
# mount /usr/local/dv # mount | grep /usr/local/dv /dev/mapper/arrakis-dv on /usr/local/dv type ext4 (rw,noatime) # systemctl status usr-local-dv.mount ● usr-local-dv.mount - /usr/local/dv Loaded: loaded (/etc/fstab; generated) Active: active (mounted) since Mon 2021-07-19 14:31:22 CDT; 10s ago Where: /usr/local/dv What: /dev/mapper/arrakis-dv Docs: man:fstab(5) man:systemd-fstab-generator(8)
The specific filesystem that fails to mount changes from boot to boot, and sometimes I do get all of them mounted as I'd expect. I've got a few other non-/usr/local filesystems as well, and sometimes *those* fail to mount too. (For instance, on this particular boot, I just noticed that /games, a root-level filesystem, had failed to mount.) Anyway, I'm at a loss. If systemd were *trying* to mount the filesystem but failing, I'd expect to see some log entries somewhere to that effect. Instead, it's just like they don't even exist at boot time. This has actually been happening for quite awhile now; I've gotten in the habit of just doing a `mount -a` after booting, to get everything mounted properly. It'd be nice to not have to do that, though. If anyone has any ideas, I'm all ears! Thx, CJ