[arch-general] Mounting /var early in systemd

newer
[arch-general] need advices for...

Paul Gideon Dann

11 Dec 2012 11 Dec '12

3:36 p.m.

Hello all, I'm migrating a diskless cluster from initscripts to systemd. The problem I have is that the nodes each need to mount their own separate /var (identified by hostname) from the NFS server when they boot. With initscripts, I used a sysinit_end hook to do the job, which worked pretty well. However, with systemd I've really hit a brick wall. I can't get the unit to start early enough, because systemd creates so many sockets (in /var) so early, and there's no target that is ordered before the socket units are started. This means that the early sockets get masked when the "real" /var is mounted, and things break. I think the only solution with systemd is to explicitly specify each socket in "Before=" for my unit, which is likely to break if anything changes in future. It might also be possible to create a generator that automates this, but that's still pretty ugly. Right now, I'm thinking the only sensible solution is to forget doing it with systemd and push the mounting of /var up into initcpio, but that's admitting defeat. Any ideas? Paul

Show replies by thread

Tom Gundersen

11 Dec 11 Dec

11:40 p.m.

On Tue, Dec 11, 2012 at 4:36 PM, Paul Gideon Dann wrote:

...

I'm migrating a diskless cluster from initscripts to systemd. The problem I have is that the nodes each need to mount their own separate /var (identified by hostname) from the NFS server when they boot.

I have never done something like this with systemd, but it should definitely be possible.

...

With initscripts, I used a sysinit_end hook to do the job, which worked pretty well. However, with systemd I've really hit a brick wall. I can't get the unit to start early enough, because systemd creates so many sockets (in /var) so early, and there's no target that is ordered before the socket units are started. This means that the early sockets get masked when the "real" /var is mounted, and things break.

I think the only solution with systemd is to explicitly specify each socket in "Before=" for my unit, which is likely to break if anything changes in future. It might also be possible to create a generator that automates this, but that's still pretty ugly.

Sockets in /var should automatically be ordered After=var.mount, so this should in theory just work. How are you mounting /var? I assume an fstab entry would not do in your setting, so I guess you somehow generate a custom var.mount file? Please link to the code so I could have a look.

...

Right now, I'm thinking the only sensible solution is to forget doing it with systemd and push the mounting of /var up into initcpio, but that's admitting defeat.

That should definitely not be necessary. -t

Paul Gideon Dann

12 Dec 12 Dec

11:10 a.m.

On Wednesday 12 Dec 2012 00:40:43 Tom Gundersen wrote:

...

Sockets in /var should automatically be ordered After=var.mount, so this should in theory just work. How are you mounting /var? I assume an fstab entry would not do in your setting, so I guess you somehow generate a custom var.mount file? Please link to the code so I could have a look.

That's what I hoped too. I've tried several approaches. I'm trying a mount unit here, because I was hoping there might be a bit more magic to it. However, it does mean that I had to hardcode the mount path (%H doesn't seem to work), but if I can get this working, I have a oneshot unit that should take care of that. I created a "sockets-pre.target" unit, ordered before "sockets.target", and the following unit is ordered before that, because I was hoping that might help. It doesn't, presumably because the socket units and this unit are all "before" sockets.target, and all get started at the same time. If the sockets were set "after" sockets-pre.target, this would probably work. (But in that case they might as well be specified directly in the unit, and the sockets-pre target can be dropped.) [Unit] Description=/var directory for the node DefaultDependencies=false Requires=sockets-pre.target Before=sockets-pre.target [Mount] What=192.168.0.1:/srv/nfs/cluster-store/vars/node07 Where=/var Type=nfs Options=v3,nolock [Install] RequiredBy=local-fs.target Bootchart is available here: http://giddie.homeip.net/screenshots/cluster-node-var-mount-boot-chart.png Thanks, Paul

Dave Reisner

1:05 p.m.

On Wed, Dec 12, 2012 at 11:10:24AM +0000, Paul Gideon Dann wrote:

...

On Wednesday 12 Dec 2012 00:40:43 Tom Gundersen wrote:

...
Sockets in /var should automatically be ordered After=var.mount, so this should in theory just work. How are you mounting /var? I assume an fstab entry would not do in your setting, so I guess you somehow generate a custom var.mount file? Please link to the code so I could have a look.

That's what I hoped too. I've tried several approaches. I'm trying a mount unit here, because I was hoping there might be a bit more magic to it. However, it does mean that I had to hardcode the mount path (%H doesn't seem to work), but if I can get this working, I have a oneshot unit that should take care of that.

I'd sooner use a generator than a oneshot unit to perform a mount.

...

I created a "sockets-pre.target" unit, ordered before "sockets.target", and the following unit is ordered before that, because I was hoping that might help. It doesn't, presumably because the socket units and this unit are all "before" sockets.target, and all get started at the same time. If the sockets were set "after" sockets-pre.target, this would probably work. (But in that case they might as well be specified directly in the unit, and the sockets-pre target can be dropped.)

If sockets.target is too late, then order it before that. man bootup shows a synchronization point at sysinit.target before jobs for sockets.target are even dispatched.

...

[Unit] Description=/var directory for the node DefaultDependencies=false Requires=sockets-pre.target Before=sockets-pre.target

[Mount] What=192.168.0.1:/srv/nfs/cluster-store/vars/node07 Where=/var Type=nfs Options=v3,nolock

[Install] RequiredBy=local-fs.target

Bootchart is available here: http://giddie.homeip.net/screenshots/cluster-node-var-mount-boot-chart.png

Thanks, Paul

Tom Gundersen

1:17 p.m.

On Wed, Dec 12, 2012 at 12:10 PM, Paul Gideon Dann wrote:

...

I'm trying a mount unit here, because I was hoping there might be a bit more magic to it.

Yes, it is.

...

However, it does mean that I had to hardcode the mount path (%H doesn't seem to work)

At the moment %H and friends only work on Exec*, I think it would make sense to extend it to work in more places (including Where=).

...

but if I can get this working, I have a oneshot unit that should take care of that.

As Dave said, maybe a generator would be better.

...

I created a "sockets-pre.target" unit, ordered before "sockets.target", and the following unit is ordered before that, because I was hoping that might help. It doesn't, presumably because the socket units and this unit are all "before" sockets.target, and all get started at the same time. If the sockets were set "after" sockets-pre.target, this would probably work. (But in that case they might as well be specified directly in the unit, and the sockets-pre target can be dropped.)

[Unit] Description=/var directory for the node DefaultDependencies=false Requires=sockets-pre.target Before=sockets-pre.target

[Mount] What=192.168.0.1:/srv/nfs/cluster-store/vars/node07 Where=/var Type=nfs Options=v3,nolock

[Install] RequiredBy=local-fs.target

I'm not able to reproduce this problem. I don't see the need for sockets-pre.target. It should be possible to simply specify your mount in /etc/fstab (obviously this only works in this test case, as the hostname will be hardcoded), and all sockets should get ordered correctly. Could you try that (remember doing "systemctl daemon-reload" after editing fstab) and then doing "systemctl show <problematic>.socket | grep After" to see if it worked? If it did not, please post the socket unit in question. Cheers, Tom

Paul Gideon Dann

3:02 p.m.

On Wednesday 12 Dec 2012 14:17:34 Tom Gundersen wrote:

...

I'm not able to reproduce this problem. I don't see the need for sockets-pre.target. It should be possible to simply specify your mount in /etc/fstab (obviously this only works in this test case, as the hostname will be hardcoded), and all sockets should get ordered correctly.

Could you try that (remember doing "systemctl daemon-reload" after editing fstab) and then doing "systemctl show <problematic>.socket | grep After" to see if it worked? If it did not, please post the socket unit in question.

Thanks for your suggestions. I added /var to fstab and rebooted, but it was ordered lower down that specifying "Before=sockets.target": http://giddie.homeip.net/screenshots/cluster-node-var-mount-boot-chart- fstab.png # systemctl show syslog.socket | grep After After=-.mount Paul

Paul Gideon Dann

3:03 p.m.

On Wednesday 12 Dec 2012 08:05:26 Dave Reisner wrote:

...

...
That's what I hoped too. I've tried several approaches. I'm trying a mount unit here, because I was hoping there might be a bit more magic to it. However, it does mean that I had to hardcode the mount path (%H doesn't seem to work), but if I can get this working, I have a oneshot unit that should take care of that.

I'd sooner use a generator than a oneshot unit to perform a mount.

Yes, you're absolutely right. I just haven't looked into implementing generators so far.

...

If sockets.target is too late, then order it before that. man bootup shows a synchronization point at sysinit.target before jobs for sockets.target are even dispatched.

I was already ordering before sockets.target, but the problem is that the other sockets are also ordered before sockets.target, so my unit isn't necessarily ordered before any of those socket units. I wasn't aware of the bootup manpage. That's incredibly helpful; thank you. I'll try ordering before a few more of those targets and see if I get anywhere. However, based on the bootchart, none of the targets appear before the first socket unit is activated. Paul

Paul Gideon Dann

4:36 p.m.

On Wednesday 12 Dec 2012 15:03:47 you wrote:

...

I wasn't aware of the bootup manpage. That's incredibly helpful; thank you. I'll try ordering before a few more of those targets and see if I get anywhere. However, based on the bootchart, none of the targets appear before the first socket unit is activated.

I couldn't actually figure out how the boot manpage connects with what actually happens. I certainly couldn't get /var to mount by ordering before any of those. Paul

Tom Gundersen

4:53 p.m.

On Wed, Dec 12, 2012 at 4:02 PM, Paul Gideon Dann wrote:

...

Thanks for your suggestions. I added /var to fstab and rebooted, but it was ordered lower down that specifying "Before=sockets.target":

So no socket is After=var.mount. The question is: which one should be? On my system there should be none.

...

http://giddie.homeip.net/screenshots/cluster-node-var-mount-boot-chart- fstab.png

# systemctl show syslog.socket | grep After After=-.mount

Is syslog.socket giving you problems? Could you paste the unit file? On my system the socket is located at /run/systemd/journal/syslog, so After=-.mount is correct. -t

Paul Gideon Dann

13 Dec 13 Dec

10:51 a.m.

On Wednesday 12 Dec 2012 17:53:23 Tom Gundersen wrote:

...

On Wed, Dec 12, 2012 at 4:02 PM, Paul Gideon Dann wrote: Is syslog.socket giving you problems? Could you paste the unit file? On my system the socket is located at /run/systemd/journal/syslog, so After=-.mount is correct.

That's a very good point; probably all of these sockets will end up in /run rather than /var (as they once did). Maybe the issues I'm seeing after boot are not related to sockets being masked as I assumed. I have no idea what else it could be, though. I've got the unit ordered before all the sockets, but NFS is still behaving badly (losing visibility of /home and other mounts), and it works fine if the proper NFS /var is not mounted at all and the /var from the NFS root is used (but then it's not using the right /var). Any idea if systemd does something else in /var in very early boot? Paul

Paul Gideon Dann

17 Dec 17 Dec

12:28 p.m.

On Thursday 13 Dec 2012 10:51:02 you wrote:

...

That's a very good point; probably all of these sockets will end up in /run rather than /var (as they once did). Maybe the issues I'm seeing after boot are not related to sockets being masked as I assumed. I have no idea what else it could be, though.

I've got the unit ordered before all the sockets, but NFS is still behaving badly (losing visibility of /home and other mounts), and it works fine if the proper NFS /var is not mounted at all and the /var from the NFS root is used (but then it's not using the right /var).

I'm thinking it might be something to do with /var/lib/nfs? Given that /home is mounted significantly later than /var, though, I really don't get why the system ends up losing /home with "Device or resource busy". Root and /var remain fine, though... /home is mounted from fstab, and is fine if a new /var is not mounted. Paul

4150

Age (days ago)

4156

Last active (days ago)

List overview

Download

10 comments

3 participants

participants (3)

Dave Reisner
Paul Gideon Dann
Tom Gundersen

[arch-general] Mounting /var early in systemd

tags

participants (3)