[arch-general] systemd not mounting some filesystems on boot?

[CJ Kucera]

Hello all!

tl;dr version: I've got some local filesystems, all within the same LVM
LV, some of which will fail to mount on boot, requiring me to do so by
hand instead.  Which filesystems don't mount can vary from boot to boot
(and sometimes they all work), and the ones that *don't* mount aren't
even mentioned anywhere by systemctl/journalctl -- I would expect that
if the mount was *attempted* but failed, I'd see some logs.  Anyone have
a clue about this?

Many details follow:

So I've got some admittedly-improperly-placed filesystems that I mount
into /usr/local, specifically: /usr/local/dv, /usr/local/winex, and
/usr/local/vmware (in addition to /usr/local itself being its own
filesystem).  I've got them all in /etc/fstab basically identical to
each other:

> # grep /usr/local /etc/fstab
> /dev/mapper/arrakis-local /usr/local          ext4    noatime 1 2
> /dev/mapper/arrakis-winex /usr/local/winex    ext4    noatime 1 2
> /dev/mapper/arrakis-vmware /usr/local/vmware  ext4    noatime 1 2
> /dev/mapper/arrakis-dv /usr/local/dv          ext4    noatime 1 2

Those all exist inside the same LVM LV, as you might guess from the
device paths.  I've noticed that on initial boot, I don't always get all of
them properly mounted, and I've got to manually mount them myself.  For
instance, after a reboot just now, /usr/local/dv did not get mounted:

> # mount | grep /usr/local
> /dev/mapper/arrakis-local on /usr/local type ext4 (rw,noatime)
> /dev/mapper/arrakis-vmware on /usr/local/vmware type ext4 (rw,noatime)
> /dev/mapper/arrakis-winex on /usr/local/winex type ext4 (rw,noatime,stripe=27879)

Or, looking at it via systemctl:

> # systemctl list-units --all | grep usr-local
>   usr-local-dv.mount       loaded    inactive dead      /usr/local/dv
>   usr-local-vmware.mount   loaded    active   mounted   /usr/local/vmware
>   usr-local-winex.mount    loaded    active   mounted   /usr/local/winex
>   usr-local.mount          loaded    active   mounted   /usr/local

Querying the usr-local-dv unit directly, I don't see anything
particularly telling:

> # systemctl status usr-local-dv.mount
> ○ usr-local-dv.mount - /usr/local/dv
>      Loaded: loaded (/etc/fstab; generated)
>      Active: inactive (dead)
>       Where: /usr/local/dv
>        What: /dev/mapper/arrakis-dv
>        Docs: man:fstab(5)
>              man:systemd-fstab-generator(8)

And there's no mention of the "dv" mount in journalctl either:

> # journalctl -b 0 | grep -E '(usr-local|/usr/local)'
> Jul 19 14:03:13 arrakis systemd[1]: usr-local.mount: Directory /usr/local to mount over is not empty, mounting anyway.
> Jul 19 14:03:13 arrakis systemd[1]: Mounting /usr/local...
> Jul 19 14:03:13 arrakis systemd[1]: Mounted /usr/local.
> Jul 19 14:03:13 arrakis systemd[1]: Mounting /usr/local/vmware...
> Jul 19 14:03:13 arrakis systemd[1]: Mounting /usr/local/winex...
> Jul 19 14:03:13 arrakis systemd[1]: Mounted /usr/local/vmware.
> Jul 19 14:03:13 arrakis systemd[1]: Mounted /usr/local/winex.

(Obvs. I've got some data in /usr/local on the root filesystem which
gets mounted over, but that doesn't interfere with the /usr/local mount
at all.)

I even checked the device creation times, to see if maybe
/dev/mapper/arrakis-dv was created too "late" or something, but it seems
as though it was actually the first of that bunch:

> # ls -lrt --full-time /dev/mapper/arrakis-{local,dv,vmware,winex}
> lrwxrwxrwx 1 root root 7 2021-07-19 14:03:12.406271238 -0500 /dev/mapper/arrakis-dv -> ../dm-4
> lrwxrwxrwx 1 root root 7 2021-07-19 14:03:13.649604518 -0500 /dev/mapper/arrakis-winex -> ../dm-5
> lrwxrwxrwx 1 root root 7 2021-07-19 14:03:13.696271181 -0500 /dev/mapper/arrakis-local -> ../dm-0
> lrwxrwxrwx 1 root root 7 2021-07-19 14:03:13.809604509 -0500 /dev/mapper/arrakis-vmware -> ../dm-3

As I mentioned above, I can mount it just fine manually, either with
`mount` or `systemctl`.  Here's me mounting it w/ `mount`, and
confirming its status w/ `systemctl`:

> # mount /usr/local/dv
> # mount | grep /usr/local/dv
> /dev/mapper/arrakis-dv on /usr/local/dv type ext4 (rw,noatime)
> # systemctl status usr-local-dv.mount
> ● usr-local-dv.mount - /usr/local/dv
>      Loaded: loaded (/etc/fstab; generated)
>      Active: active (mounted) since Mon 2021-07-19 14:31:22 CDT; 10s ago
>       Where: /usr/local/dv
>        What: /dev/mapper/arrakis-dv
>        Docs: man:fstab(5)
>              man:systemd-fstab-generator(8)

The specific filesystem that fails to mount changes from boot to boot,
and sometimes I do get all of them mounted as I'd expect.  I've got a
few other non-/usr/local filesystems as well, and sometimes *those* fail
to mount too.  (For instance, on this particular boot, I just noticed
that /games, a root-level filesystem, had failed to mount.)

Anyway, I'm at a loss.  If systemd were *trying* to mount the filesystem
but failing, I'd expect to see some log entries somewhere to that
effect.  Instead, it's just like they don't even exist at boot time.
This has actually been happening for quite awhile now; I've gotten in
the habit of just doing a `mount -a` after booting, to get everything
mounted properly.  It'd be nice to not have to do that, though.

If anyone has any ideas, I'm all ears!

Thx,
CJ