[arch-general] remote poweroff with systemd
Hello all, In one installation I manage I have a script that (among other actions) remotely shuts down a number of headless machines. Until recently these machines used the traditional init scripts, and the commands in the script were ssh -t remote1 "sudo /sbin/init 0" ssh -t remote2 "sudo /sbin/init 0" etc. This has worked well for years. Recently the machines were upgraded and mow use systemd. I replaced the original commands by ssh -t remote1 "sudo telinit 0" etc. This shuts down the first one OK, but then the script hangs. Apparently the ssh connection isn't closed cleanly. I didn't yet try 'systemctl poweroff', but according the man page, 'telinit 0' is equivalent to it. Another one I didn't yet try (just discovered it in the man pages but I don't have access to the installation ATM) is systemctl's --host option, in other words, let systemctl do the ssh. Can this be expected to work ? If not, any other ideas ? TIA for any help or hints. -- FA A world of exhaustive, reliable metadata would be an utopia. It's also a pipe-dream, founded on self-delusion, nerd hubris and hysterically inflated market opportunities. (Cory Doctorow)
On Monday 05 Aug 2013 18:39:22 Fons Adriaensen wrote:
ssh -t remote1 "sudo /sbin/init 0" ssh -t remote2 "sudo /sbin/init 0"
etc. This has worked well for years.
Recently the machines were upgraded and mow use systemd. I replaced the original commands by
ssh -t remote1 "sudo telinit 0"
i've had problems with clean shutdown since systemd, too, and what's working for me i "shutdown -h now" instead of the systemd commands i've come across. i'm using this on virtual machines only, but don't see why it shouldn't work on real remote boxes as well. this doesn't hang the ssh session, or cause any other problems i've seen yet. -- phani.
On 08/06/2013 09:01 AM, phanisvara wrote:
On Monday 05 Aug 2013 18:39:22 Fons Adriaensen wrote:
ssh -t remote1 "sudo /sbin/init 0" ssh -t remote2 "sudo /sbin/init 0"
etc. This has worked well for years.
Recently the machines were upgraded and mow use systemd. I replaced the original commands by
ssh -t remote1 "sudo telinit 0"
i've had problems with clean shutdown since systemd, too, and what's working for me i "shutdown -h now" instead of the systemd commands i've come across. i'm using this on virtual machines only, but don't see why it shouldn't work on real remote boxes as well.
this doesn't hang the ssh session, or cause any other problems i've seen yet.
Why not poweroff? Anyways, telinit and shutdown are symlinks to systemctl, so you are actualy using systemd commands :P
On Tuesday 06 Aug 2013 11:59:10 Armin K. wrote:
Why not poweroff? Anyways, telinit and shutdown are symlinks to systemctl, so you are actualy using systemd commands
thank you, good to know. i didn't get into this any further after finding something that worked...too lazy for my own good. -- phani.
On Tue, Aug 06, 2013 at 12:31:20PM +0530, phanisvara wrote:
On Monday 05 Aug 2013 18:39:22 Fons Adriaensen wrote:
ssh -t remote1 "sudo /sbin/init 0" ssh -t remote2 "sudo /sbin/init 0"
etc. This has worked well for years.
Recently the machines were upgraded and mow use systemd. I replaced the original commands by
ssh -t remote1 "sudo telinit 0"
i've had problems with clean shutdown since systemd, too, and what's working for me i "shutdown -h now" instead of the systemd commands i've come across. i'm using this on virtual machines only, but don't see why it shouldn't work on real remote boxes as well.
this doesn't hang the ssh session, or cause any other problems i've seen yet.
Tried it, it does hang the ssh session here. Also tried systemctl --host. This makes systemd complain rather clearly about unit files being modified and other havoc. But I found a very systemd-ish solution: On the remote machines I have /etc/systemd/system/poweroff.timer ---- [Unit] Description=Delayed poweroff [Timer] OnActiveSec=5 Unit=poweroff.target ---- and the script does ssh -t remote1 "sudo systemctl start poweroff.timer" As this is the first systemd unit file I wrote it can probably be improved, all suggestions from the experts are welcome. Ciao, -- FA A world of exhaustive, reliable metadata would be an utopia. It's also a pipe-dream, founded on self-delusion, nerd hubris and hysterically inflated market opportunities. (Cory Doctorow)
On Tuesday 06 Aug 2013 10:09:33 Fons Adriaensen wrote:
On the remote machines I have /etc/systemd/system/poweroff.timer
---- [Unit] Description=Delayed poweroff
[Timer] OnActiveSec=5 Unit=poweroff.target ----
and the script does
ssh -t remote1 "sudo systemctl start poweroff.timer"
Have you tried something like: ssh -t remote1 "sudo systemctl poweroff && exit" Technically I guess it's a race condition, but the command should terminate and close the SSH session before OpenSSH is shut down, so you shouldn't get the hang. Alternatively, you could try "systemctl start poweroff.target" to see if that returns quicker. I'm not sure if the poweroff command returns straight away or waits. Basically, I'm thinking that if you can get the first command to return quickly, your shell will close nicely before the shutdown kills the connection. Paul
On Tue, Aug 6, 2013 at 2:56 PM, Paul Gideon Dann <pdgiddie@gmail.com> wrote:
On Tuesday 06 Aug 2013 10:09:33 Fons Adriaensen wrote:
On the remote machines I have /etc/systemd/system/poweroff.timer
---- [Unit] Description=Delayed poweroff
[Timer] OnActiveSec=5 Unit=poweroff.target ----
and the script does
ssh -t remote1 "sudo systemctl start poweroff.timer"
Have you tried something like:
ssh -t remote1 "sudo systemctl poweroff && exit"
Technically I guess it's a race condition, but the command should terminate and close the SSH session before OpenSSH is shut down, so you shouldn't get the hang.
Alternatively, you could try "systemctl start poweroff.target" to see if that returns quicker. I'm not sure if the poweroff command returns straight away or waits. Basically, I'm thinking that if you can get the first command to return quickly, your shell will close nicely before the shutdown kills the connection.
Or maybe "systemctl --no-block poweroff".
On Tue, Aug 06, 2013 at 01:56:09PM +0100, Paul Gideon Dann wrote:
ssh -t remote1 "sudo systemctl poweroff && exit"
Technically I guess it's a race condition, but the command should terminate and close the SSH session before OpenSSH is shut down, so you shouldn't get the hang.
Alternatively, you could try "systemctl start poweroff.target" to see if that returns quicker. I'm not sure if the poweroff command returns straight away or waits. Basically, I'm thinking that if you can get the first command to return quickly, your shell will close nicely before the shutdown kills the connection.
Thanks to all who suggested solutions. What difference should the '&& exit' make ? The ssh will terminate anyway when poweroff returns. Problem is that by then it's too late. I think the fundamental problem is that you just can't expect 'systemctl poweroff' or whatever variation of it, to return and still have a clean system at that point. So the explicit delay provided by the poweroff.timer unit seems the right solution. And I like it because it doesn't rely on anything undocumented - it just uses systemd's features. Ciao, -- FA A world of exhaustive, reliable metadata would be an utopia. It's also a pipe-dream, founded on self-delusion, nerd hubris and hysterically inflated market opportunities. (Cory Doctorow)
(Resending, used the wrong email before. Again. -_-' ) On 6 August 2013 16:18, Fons Adriaensen wrote:
What difference should the '&& exit' make ? The ssh will terminate anyway when poweroff returns. Problem is that by then it's too late.
I think the fundamental problem is that you just can't expect 'systemctl poweroff' or whatever variation of it, to return and still have a clean system at that point.
How about something like 'shutdown -t 5'? I know if I put that into an SSH session manually, it'll print me a wall message and then I have some time to hit CTRL-D (or exit if I'm quick), so why would an automated command not be able to do the same? For this reason I believe the above, plus the '&& exit' might actually work. The command will return, the wall message will print (obviously we won't see that) and then the exit will happen, before the system actually tries to shutdown. I don't have a box handy to test with as I write this though, sorry. -- *:wq!*
On Wed, Aug 7, 2013 at 11:23 AM, Joe Eaves <jinux@alluha.net> wrote:
How about something like 'shutdown -t 5'? I know if I put that into an SSH session manually, it'll print me a wall message and then I have some time to hit CTRL-D (or exit if I'm quick), so why would an automated command not be able to do the same?
Would it be possible to delay the reboot command whilst connected in an ssh session by using something like: # at now + 2 minutes systemctl reboot then exit the ssh session and wait until the remote machine has rebooted? I haven't tested it but is there a reason why this would not work? -- mike c
On Wednesday 07 Aug 2013 14:50:53 Mike Cloaked wrote:
Would it be possible to delay the reboot command whilst connected in an ssh session by using something like:
# at now + 2 minutes systemctl reboot
then exit the ssh session and wait until the remote machine has rebooted?
I haven't tested it but is there a reason why this would not work?
The "at" daemon has to be installed & running for this to work. I believe the latest systemd has functionality that is intended to replace "at", but I can't remember what the syntax is for that, or indeed if it's been released yet. Paul
On Tuesday 06 Aug 2013 15:18:39 Fons Adriaensen wrote:
Thanks to all who suggested solutions.
What difference should the '&& exit' make ? The ssh will terminate anyway when poweroff returns. Problem is that by then it's too late.
I think the fundamental problem is that you just can't expect 'systemctl poweroff' or whatever variation of it, to return and still have a clean system at that point.
So the explicit delay provided by the poweroff.timer unit seems the right solution. And I like it because it doesn't rely on anything undocumented - it just uses systemd's features.
Yeah, it's a nice, practical solution. The analytic part of me wonders if it's truly a "correct" solution: is there a genuine guarantee that the shell session will close and SSH will close the connection gracefully in 5 seconds? In practice, this should always happen, but it's probably still technically a race just like before. Just an academic exercise, though. Paul
On Tue, Aug 6, 2013 at 5:18 PM, Fons Adriaensen <fons@linuxaudio.org> wrote:
On Tue, Aug 06, 2013 at 01:56:09PM +0100, Paul Gideon Dann wrote:
ssh -t remote1 "sudo systemctl poweroff && exit"
Technically I guess it's a race condition, but the command should terminate and close the SSH session before OpenSSH is shut down, so you shouldn't get the hang.
Alternatively, you could try "systemctl start poweroff.target" to see if that returns quicker. I'm not sure if the poweroff command returns straight away or waits. Basically, I'm thinking that if you can get the first command to return quickly, your shell will close nicely before the shutdown kills the connection.
Thanks to all who suggested solutions.
What difference should the '&& exit' make ? The ssh will terminate anyway when poweroff returns. Problem is that by then it's too late.
That's why I suggested "systemctl --no-block poweroff". That should return immediately and finish the ssh session well within the grace time period that systemd gives to every service in poweroff. As I see it, the problem with a simple "systemctl poweroff" (or equivalent) is not a race-condition, but a dead-lock: systemd waits for the ssh session to finish to go on with the poweroff, and the ssh session waits for the poweroff to finish until it returns. And those are precisely the situations where "--no-block" is useful. -- Rodrigo
On Wed, Aug 7, 2013 at 12:58 PM, Rodrigo Rivas <rodrigorivascosta@gmail.com> wrote:
On Tue, Aug 6, 2013 at 5:18 PM, Fons Adriaensen <fons@linuxaudio.org> wrote:
On Tue, Aug 06, 2013 at 01:56:09PM +0100, Paul Gideon Dann wrote:
ssh -t remote1 "sudo systemctl poweroff && exit"
Technically I guess it's a race condition, but the command should terminate and close the SSH session before OpenSSH is shut down, so you shouldn't get the hang.
Alternatively, you could try "systemctl start poweroff.target" to see if that returns quicker. I'm not sure if the poweroff command returns straight away or waits. Basically, I'm thinking that if you can get the first command to return quickly, your shell will close nicely before the shutdown kills the connection.
Thanks to all who suggested solutions.
What difference should the '&& exit' make ? The ssh will terminate anyway when poweroff returns. Problem is that by then it's too late.
That's why I suggested "systemctl --no-block poweroff". That should return immediately and finish the ssh session well within the grace time period that systemd gives to every service in poweroff.
As I see it, the problem with a simple "systemctl poweroff" (or equivalent) is not a race-condition,
It is correct that systemtl poweroff is synchronous, but using telinit or --no-block will avoid that.
but a dead-lock: systemd waits for the ssh session to finish to go on with the poweroff, and the ssh session waits for the poweroff to finish until it returns. And those are precisely the situations where "--no-block" is useful.
Actually, you don't get this deadlock. ssh will shutdown just fine even if you have some command (such as systemctl) running. The reason for the problem you are seeing here is that the network is torn down before ssh is shutdown. This isn't actually a big problem, as the remote machine is shutting down just fine. The only problem is that the ssh client ends up hanging, which is annoying, but not really critical (kill it/ignore it/whatever). I have an idea how to solve this at the systemd/networkmanager/openssh side, but haven't gotten around to sorting all that out yet (we need to order the network tear-down after user-sessions are closed, but we don't want to order network setup before user-sessions are allowed). Cheers, Tom
On Wed, Aug 07, 2013 at 01:15:28PM +0200, Tom Gundersen wrote:
It is correct that systemtl poweroff is synchronous, but using telinit or --no-block will avoid that.
Are you sure about telinit ? It was the first thing I tried, assuming it would asynchronous. But the man page says nothing about it using --no-block.
but a dead-lock: systemd waits for the ssh session to finish to go on with the poweroff, and the ssh session waits for the poweroff to finish until it returns. And those are precisely the situations where "--no-block" is useful.
Actually, you don't get this deadlock. ssh will shutdown just fine even if you have some command (such as systemctl) running.
The reason for the problem you are seeing here is that the network is torn down before ssh is shutdown. This isn't actually a big problem, as the remote machine is shutting down just fine. The only problem is that the ssh client ends up hanging, which is annoying, but not really critical (kill it/ignore it/whatever).
The shutdowns are called from a script which is supposed to continue... Ciao, -- FA A world of exhaustive, reliable metadata would be an utopia. It's also a pipe-dream, founded on self-delusion, nerd hubris and hysterically inflated market opportunities. (Cory Doctorow)
On Wed, Aug 7, 2013 at 2:17 PM, Fons Adriaensen <fons@linuxaudio.org> wrote:
On Wed, Aug 07, 2013 at 01:15:28PM +0200, Tom Gundersen wrote:
It is correct that systemtl poweroff is synchronous, but using telinit or --no-block will avoid that.
Are you sure about telinit ? It was the first thing I tried, assuming it would asynchronous. But the man page says nothing about it using --no-block.
If you use /bin/init (which is a symlink to systemctl) it is synchronous, but if you change runlevel via the compatibility /dev/initctl it is asynchronous. I couldn't find that we ship the telinit binary in Arch, so don't know exactly what you were using, but I assume it was the legacy one using /dev/initctl, so it should have been async (I guess the man page should specify this).
The reason for the problem you are seeing here is that the network is torn down before ssh is shutdown. This isn't actually a big problem, as the remote machine is shutting down just fine. The only problem is that the ssh client ends up hanging, which is annoying, but not really critical (kill it/ignore it/whatever).
The shutdowns are called from a script which is supposed to continue...
I'd make some hack like forking it off. Or better yet, use systemctl -H (which I hope works...). -t
On Wed, Aug 07, 2013 at 04:34:41PM +0200, Tom Gundersen wrote:
If you use /bin/init (which is a symlink to systemctl) it is synchronous, but if you change runlevel via the compatibility /dev/initctl it is asynchronous. I couldn't find that we ship the telinit binary in Arch, so don't know exactly what you were using, but I assume it was the legacy one using /dev/initctl, so it should have been async (I guess the man page should specify this).
If the man page corresponds to the actual code it was the one based on systemd, provided only for compatibility. Ciao, -- FA A world of exhaustive, reliable metadata would be an utopia. It's also a pipe-dream, founded on self-delusion, nerd hubris and hysterically inflated market opportunities. (Cory Doctorow)
I didn't yet try 'systemctl poweroff', but according the man page, 'telinit 0' is equivalent to it.
Another one I didn't yet try (just discovered it in the man pages but I don't have access to the installation ATM) is systemctl's --host option, in other words, let systemctl do the ssh. Can this be expected to work ? If not, any other ideas ?
ssh host 'nohup systemctl poweroff &' -- дамјан
On 05.08.2013 20:39, Fons Adriaensen wrote:
ssh -t remote1 "sudo telinit 0"
etc. This shuts down the first one OK, but then the script hangs. Apparently the ssh connection isn't closed cleanly.
Additionally to the approach you took or the ones proposed by others, using ssh -o TCPKeepAlive=yes should help to terminate the SSH connection to a dead peer at some point. Cheers, Daniel
participants (10)
-
Armin K.
-
Damjan
-
Daniel Albers
-
Fons Adriaensen
-
Joe Eaves
-
Mike Cloaked
-
Paul Gideon Dann
-
phanisvara
-
Rodrigo Rivas
-
Tom Gundersen