On Monday 05 Aug 2013 18:39:22 Fons Adriaensen wrote:

ssh -t remote1 "sudo /sbin/init 0" ssh -t remote2 "sudo /sbin/init 0"

etc. This has worked well for years.

Recently the machines were upgraded and mow use systemd. I replaced the original commands by

ssh -t remote1 "sudo telinit 0"

i've had problems with clean shutdown since systemd, too, and what's working for me i "shutdown -h now" instead of the systemd commands i've come across. i'm using this on virtual machines only, but don't see why it shouldn't work on real remote boxes as well. this doesn't hang the ssh session, or cause any other problems i've seen yet. -- phani.

On Tue, Aug 06, 2013 at 12:31:20PM +0530, phanisvara wrote:

On Monday 05 Aug 2013 18:39:22 Fons Adriaensen wrote:

...

ssh -t remote1 "sudo /sbin/init 0" ssh -t remote2 "sudo /sbin/init 0"

etc. This has worked well for years.

Recently the machines were upgraded and mow use systemd. I replaced the original commands by

ssh -t remote1 "sudo telinit 0"

i've had problems with clean shutdown since systemd, too, and what's working for me i "shutdown -h now" instead of the systemd commands i've come across. i'm using this on virtual machines only, but don't see why it shouldn't work on real remote boxes as well.

this doesn't hang the ssh session, or cause any other problems i've seen yet.

Tried it, it does hang the ssh session here. Also tried systemctl --host. This makes systemd complain rather clearly about unit files being modified and other havoc. But I found a very systemd-ish solution: On the remote machines I have /etc/systemd/system/poweroff.timer ---- [Unit] Description=Delayed poweroff [Timer] OnActiveSec=5 Unit=poweroff.target ---- and the script does ssh -t remote1 "sudo systemctl start poweroff.timer" As this is the first systemd unit file I wrote it can probably be improved, all suggestions from the experts are welcome. Ciao, -- FA A world of exhaustive, reliable metadata would be an utopia. It's also a pipe-dream, founded on self-delusion, nerd hubris and hysterically inflated market opportunities. (Cory Doctorow)

On Tuesday 06 Aug 2013 10:09:33 Fons Adriaensen wrote:

On the remote machines I have /etc/systemd/system/poweroff.timer

---- [Unit] Description=Delayed poweroff

[Timer] OnActiveSec=5 Unit=poweroff.target ----

and the script does

ssh -t remote1 "sudo systemctl start poweroff.timer"

Have you tried something like: ssh -t remote1 "sudo systemctl poweroff && exit" Technically I guess it's a race condition, but the command should terminate and close the SSH session before OpenSSH is shut down, so you shouldn't get the hang. Alternatively, you could try "systemctl start poweroff.target" to see if that returns quicker. I'm not sure if the poweroff command returns straight away or waits. Basically, I'm thinking that if you can get the first command to return quickly, your shell will close nicely before the shutdown kills the connection. Paul

On Tue, Aug 06, 2013 at 01:56:09PM +0100, Paul Gideon Dann wrote:

ssh -t remote1 "sudo systemctl poweroff && exit"

Technically I guess it's a race condition, but the command should terminate and close the SSH session before OpenSSH is shut down, so you shouldn't get the hang.

Alternatively, you could try "systemctl start poweroff.target" to see if that returns quicker. I'm not sure if the poweroff command returns straight away or waits. Basically, I'm thinking that if you can get the first command to return quickly, your shell will close nicely before the shutdown kills the connection.

Thanks to all who suggested solutions. What difference should the '&& exit' make ? The ssh will terminate anyway when poweroff returns. Problem is that by then it's too late. I think the fundamental problem is that you just can't expect 'systemctl poweroff' or whatever variation of it, to return and still have a clean system at that point. So the explicit delay provided by the poweroff.timer unit seems the right solution. And I like it because it doesn't rely on anything undocumented - it just uses systemd's features. Ciao, -- FA A world of exhaustive, reliable metadata would be an utopia. It's also a pipe-dream, founded on self-delusion, nerd hubris and hysterically inflated market opportunities. (Cory Doctorow)

On Tuesday 06 Aug 2013 15:18:39 Fons Adriaensen wrote:

Thanks to all who suggested solutions.

What difference should the '&& exit' make ? The ssh will terminate anyway when poweroff returns. Problem is that by then it's too late.

I think the fundamental problem is that you just can't expect 'systemctl poweroff' or whatever variation of it, to return and still have a clean system at that point.

So the explicit delay provided by the poweroff.timer unit seems the right solution. And I like it because it doesn't rely on anything undocumented - it just uses systemd's features.

Yeah, it's a nice, practical solution. The analytic part of me wonders if it's truly a "correct" solution: is there a genuine guarantee that the shell session will close and SSH will close the connection gracefully in 5 seconds? In practice, this should always happen, but it's probably still technically a race just like before. Just an academic exercise, though. Paul

On Wed, Aug 7, 2013 at 12:58 PM, Rodrigo Rivas wrote:

On Tue, Aug 6, 2013 at 5:18 PM, Fons Adriaensen wrote:

...

On Tue, Aug 06, 2013 at 01:56:09PM +0100, Paul Gideon Dann wrote:

...

ssh -t remote1 "sudo systemctl poweroff && exit"

Technically I guess it's a race condition, but the command should terminate and close the SSH session before OpenSSH is shut down, so you shouldn't get the hang.

Alternatively, you could try "systemctl start poweroff.target" to see if that returns quicker. I'm not sure if the poweroff command returns straight away or waits. Basically, I'm thinking that if you can get the first command to return quickly, your shell will close nicely before the shutdown kills the connection.

Thanks to all who suggested solutions.

What difference should the '&& exit' make ? The ssh will terminate anyway when poweroff returns. Problem is that by then it's too late.

That's why I suggested "systemctl --no-block poweroff". That should return immediately and finish the ssh session well within the grace time period that systemd gives to every service in poweroff.

As I see it, the problem with a simple "systemctl poweroff" (or equivalent) is not a race-condition,

It is correct that systemtl poweroff is synchronous, but using telinit or --no-block will avoid that.

but a dead-lock: systemd waits for the ssh session to finish to go on with the poweroff, and the ssh session waits for the poweroff to finish until it returns. And those are precisely the situations where "--no-block" is useful.

Actually, you don't get this deadlock. ssh will shutdown just fine even if you have some command (such as systemctl) running. The reason for the problem you are seeing here is that the network is torn down before ssh is shutdown. This isn't actually a big problem, as the remote machine is shutting down just fine. The only problem is that the ssh client ends up hanging, which is annoying, but not really critical (kill it/ignore it/whatever). I have an idea how to solve this at the systemd/networkmanager/openssh side, but haven't gotten around to sorting all that out yet (we need to order the network tear-down after user-sessions are closed, but we don't want to order network setup before user-sessions are allowed). Cheers, Tom

On Wed, Aug 07, 2013 at 01:15:28PM +0200, Tom Gundersen wrote:

It is correct that systemtl poweroff is synchronous, but using telinit or --no-block will avoid that.

Are you sure about telinit ? It was the first thing I tried, assuming it would asynchronous. But the man page says nothing about it using --no-block.

...

but a dead-lock: systemd waits for the ssh session to finish to go on with the poweroff, and the ssh session waits for the poweroff to finish until it returns. And those are precisely the situations where "--no-block" is useful.

Actually, you don't get this deadlock. ssh will shutdown just fine even if you have some command (such as systemctl) running.

The reason for the problem you are seeing here is that the network is torn down before ssh is shutdown. This isn't actually a big problem, as the remote machine is shutting down just fine. The only problem is that the ssh client ends up hanging, which is annoying, but not really critical (kill it/ignore it/whatever).

The shutdowns are called from a script which is supposed to continue... Ciao, -- FA A world of exhaustive, reliable metadata would be an utopia. It's also a pipe-dream, founded on self-delusion, nerd hubris and hysterically inflated market opportunities. (Cory Doctorow)

On Wed, Aug 7, 2013 at 2:17 PM, Fons Adriaensen wrote:

On Wed, Aug 07, 2013 at 01:15:28PM +0200, Tom Gundersen wrote:

...

It is correct that systemtl poweroff is synchronous, but using telinit or --no-block will avoid that.

Are you sure about telinit ? It was the first thing I tried, assuming it would asynchronous. But the man page says nothing about it using --no-block.

If you use /bin/init (which is a symlink to systemctl) it is synchronous, but if you change runlevel via the compatibility /dev/initctl it is asynchronous. I couldn't find that we ship the telinit binary in Arch, so don't know exactly what you were using, but I assume it was the legacy one using /dev/initctl, so it should have been async (I guess the man page should specify this).

...

The reason for the problem you are seeing here is that the network is torn down before ssh is shutdown. This isn't actually a big problem, as the remote machine is shutting down just fine. The only problem is that the ssh client ends up hanging, which is annoying, but not really critical (kill it/ignore it/whatever).

The shutdowns are called from a script which is supposed to continue...

I'd make some hack like forking it off. Or better yet, use systemctl -H (which I hope works...). -t

On 05.08.2013 20:39, Fons Adriaensen wrote:

ssh -t remote1 "sudo telinit 0"

etc. This shuts down the first one OK, but then the script hangs. Apparently the ssh connection isn't closed cleanly.

Additionally to the approach you took or the ones proposed by others, using ssh -o TCPKeepAlive=yes should help to terminate the SSH connection to a dead peer at some point. Cheers, Daniel