[arch-general] netcl leaves network down until reenable performed - do we need note?
Archdevs, I have two computers using netctl, one for a static and one for a dhcp address. After the last update on May, 3, netctl (1.22-1 -> 1.23-1) all remained fine and old profiles generated as subservices in /etc/systemd/system, such as: netctl\@rlf_network\\x2dstatic.service continued to bring the network up. However within the past week there was a change, likely to systemd 246, that no longer respects the subservices originally created by netctl. Now attempting at reboot causes the network to fail completely. Not good for remotely adminned boxes. What is required is a "reenable" of the netctl profile. Doing so will remove the old subservice and symlinks and replace then with a directory in /etc/systemd/system of same name as the old subservice, but with a ".d" appended that now contains a "profile.conf", e.g. netctl@rlf_network\x2dstatic.service.d/ └── profile.conf However, there is no note or warning during update that any manual intervention will be needed. That will leave anyone adminning a remote arch install with netctl with a box that is unreachable and has no network. Shouldn't there be a warning about this change generated on update? Arch is always pretty good about warning when manual interaction is required -- and this is a biggie. Couldn't there also be a post install that does a reenable for each netctl profile found in /etc/systemd/system as another option to avoid this SNAFU? -- David C. Rankin, J.D.,P.E.
On 8/17/20 5:25 PM, David C. Rankin wrote:
Archdevs,
I have two computers using netctl, one for a static and one for a dhcp address. After the last update on May, 3,
netctl (1.22-1 -> 1.23-1)
all remained fine and old profiles generated as subservices in /etc/systemd/system, such as:
netctl\@rlf_network\\x2dstatic.service
continued to bring the network up. However within the past week there was a change, likely to systemd 246, that no longer respects the subservices originally created by netctl. Now attempting at reboot causes the network to fail completely. Not good for remotely adminned boxes.
What is required is a "reenable" of the netctl profile. Doing so will remove the old subservice and symlinks and replace then with a directory in /etc/systemd/system of same name as the old subservice, but with a ".d" appended that now contains a "profile.conf", e.g.
By "subservice" I think you mean ".include" stanzas? Your analysis is correct, this got removed from systemd here: https://github.com/systemd/systemd/commit/7ade8982ca1969e251a29589ae918c3b5c... and systemd 246 is the first release with its removal. However, netctl migrated over to *.d dropins here: https://git.archlinux.org/netctl.git/commit/?id=04d39b2573bd34d4159837afdb47... So I think you should be fine if you've re-enabled the netctl profile with 1.18 or higher (released on 2018-08-07). That's 2 years. Granted, if it's been working for years, why would anyone care about manually re-enabling their netctl profile... but... There should also probably have been a warning logged in journalctl for this, if your service was still using the old method.
netctl@rlf_network\x2dstatic.service.d/ └── profile.conf
However, there is no note or warning during update that any manual intervention will be needed. That will leave anyone adminning a remote arch install with netctl with a box that is unreachable and has no network.
Shouldn't there be a warning about this change generated on update? Arch is always pretty good about warning when manual interaction is required -- and this is a biggie.
I'm not sure it merits a news post for something that old which is only now becoming fatal. Hopefully anyone with remotely adminned boxes caught this while monitoring journalctl logs. But, thanks for posting this to the list -- at the very least, people in the same situation will be able to figure out what happened by reading here. And hopefully they will see this *before* upgrading.
Couldn't there also be a post install that does a reenable for each netctl profile found in /etc/systemd/system as another option to avoid this SNAFU? That might have been an interesting precautionary measure for netctl 1.18, at least for printing a message advising people to reenable the service.
I'm not sure it makes sense to do that automatically, since disabling a profile removes customizations and the netctl manpage explicitly warns you to be careful about doing so. -- Eli Schwartz Bug Wrangler and Trusted User
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi list, I got hit by this, too. On Mon, 17 Aug 2020, Eli Schwartz via arch-general wrote:
On 8/17/20 5:25 PM, David C. Rankin wrote:
By "subservice" I think you mean ".include" stanzas?
Your analysis is correct, this got removed from systemd here: https://github.com/systemd/systemd/commit/7ade8982ca1969e251a29589ae918c3b5c...
and systemd 246 is the first release with its removal.
However, netctl migrated over to *.d dropins here: https://git.archlinux.org/netctl.git/commit/?id=04d39b2573bd34d4159837afdb47...
So I think you should be fine if you've re-enabled the netctl profile with 1.18 or higher (released on 2018-08-07). That's 2 years. Granted, if it's been working for years, why would anyone care about manually re-enabling their netctl profile... but...
There should also probably have been a warning logged in journalctl for this, if your service was still using the old method.
That's true, however, when this warning first appeared, `netctl reenable my-profile` did not help - same was true several days later (probably also weeks/months). So I forgot about the warning (I'm not keen to put `for profile in $list; do netctl reenable $profile; done` into my common update routine for all my arch boxes).
netctl@rlf_network\x2dstatic.service.d/ └── profile.conf
However, there is no note or warning during update that any manual intervention will be needed. That will leave anyone adminning a remote arch install with netctl with a box that is unreachable and has no network.
Shouldn't there be a warning about this change generated on update? Arch is always pretty good about warning when manual interaction is required -- and this is a biggie.
I'm not sure it merits a news post for something that old which is only now becoming fatal.
In my opinion, a news post would have been appropriate. But now it's "too late" (see below).
Hopefully anyone with remotely adminned boxes caught this while monitoring journalctl logs.
Logs were unhelpful at that time, see above.
But, thanks for posting this to the list -- at the very least, people in the same situation will be able to figure out what happened by reading here. And hopefully they will see this *before* upgrading.
Additionally, people on the list might catch this issue in advance (if they don't update too often and have not yet been hit by this issue). This makes a news post obsolete, now, I believe.
Couldn't there also be a post install that does a reenable for each netctl profile found in /etc/systemd/system as another option to avoid this SNAFU? That might have been an interesting precautionary measure for netctl 1.18, at least for printing a message advising people to reenable the service.
I'm not sure it makes sense to do that automatically, since disabling a profile removes customizations and the netctl manpage explicitly warns you to be careful about doing so.
I think, it's not arch's way to do such things. Arch rather says "your config is broken/old, run `xyz` to fix it" than simply running `xyz`.
-- Eli Schwartz Bug Wrangler and Trusted User
regards, Erich -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE3p92iMrPBP64GmxZCu7JB1Xae1oFAl87ZzQACgkQCu7JB1Xa e1qq7w/+KHd/vyL9AloIgirCeEcYf/+JxcCOngmROoxbM/Gs3rtxynewpwqV6iCi VJGIlrH5Xl2/NasKONRQuzNyGD0qqIMMRbAiNOZq3r37RAb8vmWmRxJmY5rkymO7 yoG4ox4HSgfeTyCvQhUnMZWE8Qm7INjcI/nHvs//wIwqB5ppeLkkFLHK9LCWBpaF +dkKP5vrsvjrD9785RjjMg46iHDCJ3AEAPuZmWes7IgUR161MD7Ujf07YMDfit8u 1SrdtxsncPSR0OX/m4DKC31tBP5WjS4kGL61NjUfQm354YtkjhyULQ7ecobRNBqA xQ4i6nLpbbZJSx38S0jfOgsAqKxM5HV6pFXRUR1GTT6a0MTxXG9bERmVm6oaFQeN 9cm3b4MbeBDVXp5x7MI2VGlG2mmNaIxZWpMKtUufQwBOxSkDSp5rifqy+NUgPhLX H/WcdK8pCQfIVfiopENVA9aNrb25/ST79io0KfUNmON3HHEnCWaJGHmCFMB/yOe7 2frLccn6TP17KkSSL5cJev5tpB+epH9k3BUxln7BT9ZDGVNJwbhlksaKXWncj8Hg OEvt6vqHCuswwvcrAvTJwDeCVtQRU4dt3cqSqCHuJjKzNg6egE1gU9KOxvFkZE0T 5dNq9DrcixQoXbG59DhNsOkM8b186HvvXOxTVBJvsFwS46Nnn4g= =/I9c -----END PGP SIGNATURE-----
On 18/08/2020 07.29, Erich Eckner via arch-general wrote:
Hi list,
I got hit by this, too.
Me too. Fortunately I was doing update on-site for a box without a BMC. Regards, Łukasz
On 8/17/20 5:56 PM, Eli Schwartz wrote:
Couldn't there also be a post install that does a reenable for each netctl profile found in /etc/systemd/system as another option to avoid this SNAFU?
That might have been an interesting precautionary measure for netctl 1.18, at least for printing a message advising people to reenable the service.
Oh, for the record -- it looks like netctl already did this: https://git.archlinux.org/svntogit/packages.git/tree/trunk/netctl.install?h=... post_upgrade() { if [[ $(vercmp 1.18 "$2") -gt 0 ]]; then grep -ls '^.include ' /etc/systemd/system/netctl@*.service | \ while read -r unit; do profile=$(systemd-escape --unescape "${unit:27:-8}") echo ":: The unit for profile '$profile' uses deprecated features." echo " Consider running: netctl reenable $(printf '%q' "$profile")" done fi } -- Eli Schwartz Bug Wrangler and Trusted User
On 8/21/20 12:25 AM, Eli Schwartz via arch-general wrote:
That might have been an interesting precautionary measure for netctl 1.18, at least for printing a message advising people to reenable the service. Oh, for the record -- it looks like netctl already did this:
https://git.archlinux.org/svntogit/packages.git/tree/trunk/netctl.install?h=...
post_upgrade() { if [[ $(vercmp 1.18 "$2") -gt 0 ]]; then grep -ls '^.include ' /etc/systemd/system/netctl@*.service | \ while read -r unit; do profile=$(systemd-escape --unescape "${unit:27:-8}") echo ":: The unit for profile '$profile' uses deprecated features." echo " Consider running: netctl reenable $(printf '%q' "$profile")" done fi }
The fallibility of human infallibility... You can lead a horse to water, but you can't make 'em drink :) Thank you Eli, at least we know now the distribution was 1-step ahead. -- David C. Rankin, J.D.,P.E.
participants (4)
-
David C. Rankin
-
Eli Schwartz
-
Erich Eckner
-
Łukasz Michalski