On 6/28/25 4:45 AM, Ralph Corderoy wrote:

Hi David,

...

the update tonight (an hour ago), failed to start dhcpd4. I can't ping the box, I have to log in locally and I find the network interface enp5s0 'down' - that's not happened since Arch was installed.

Did it reboot because of that update? If so, could the damage have been caused by some other change made since the previous boot which only took effect on booting?

Thanks Ralf, No, this was just a normal pacman -Syu and then a manual reboot because of the kernel update. Nothing strange. I watch the rss feed for kernel updates in core and always update for each new kernel and then manually reboot. There is something that definitely changed in how the network interface is being handled. It is brought up fine, but then is taken down??? See last post for details and config files, but the crux of it from the journal is: # journalctl -b --no-pager --full | grep enp5 | head -n50 Jun 28 02:17:05 valkyrie systemd[1]: Expecting device /sys/subsystem/net/devices/enp5s0... Jun 28 02:17:08 valkyrie kernel: r8169 0000:05:00.0 enp5s0: renamed from eth0 Jun 28 02:17:12 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:17:15 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx ************************************************************************ Jun 28 02:17:15 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:17:15 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv4 for mDNS. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 28 02:17:15 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down ************************************************************************ Jun 28 02:17:15 valkyrie avahi-daemon[711]: Interface enp5s0.IPv4 no longer relevant for mDNS. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Leaving mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Withdrawing address record for 192.168.6.14 on enp5s0. Jun 28 02:17:18 valkyrie named[801]: listening on IPv4 interface enp5s0, 192.168.6.14#53 Jun 28 02:40:10 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:40:10 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv4 for mDNS. Jun 28 02:40:10 valkyrie avahi-daemon[711]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 28 02:40:10 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:40:13 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 28 02:40:14 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv6 with address fe80::feaa:14ff:fe9e:3319. Jun 28 02:40:14 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv6 for mDNS. Jun 28 02:40:14 valkyrie avahi-daemon[711]: Registering new address record for fe80::feaa:14ff:fe9e:3319 on enp5s0.*. Jun 28 02:41:43 valkyrie dhcpd[1781]: Listening on LPF/enp5s0/fc:aa:14:9e:33:19/192.168.6.0/24 Jun 28 02:41:43 valkyrie dhcpd[1781]: Sending on LPF/enp5s0/fc:aa:14:9e:33:19/192.168.6.0/24 Huh??? Where did the IPv4 interface go? And why? -- David C. Rankin, J.D.,P.E.

On 6/28/25 11:04 AM, David C Rankin wrote:

here has definitely been a change in how (whatever) looks at IPv4 and IPv6 and chooses the interface to use. Here is the same journal entries from the last boot of kernel 6.15.3 and IPv4 is configured and stays UP. So whatever happened, it looks like the logic governing whether to use IPv4 or IPv6 changed in this last update and that is what is taking my IPv4 interface down, e.g.

No, it's not an IPv4/IPv6 logic choice issue, it is simply that on boot the interfaces is configured for IPv4 and brought up, but then is immediately taken down again by something. The later manual bringing up of the interface configured it as it has always been with mdns registered for both IPv4 and IPv6. The difference being on manual activation the interface stays up. It's looking like there may have been a change in how systemd is managing the interface. Something takes the interface back down during boot. It is the boot activation where something has changed. The boot entries are the Jun 28 02:17:05 journal entries: Jun 28 02:17:05 valkyrie systemd[1]: Expecting device /sys/subsystem/net/devices/enp5s0... Jun 28 02:17:08 valkyrie kernel: r8169 0000:05:00.0 enp5s0: renamed from eth0 Jun 28 02:17:12 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:17:15 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 28 02:17:15 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:17:15 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv4 for mDNS. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 28 02:17:15 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:17:15 valkyrie avahi-daemon[711]: Interface enp5s0.IPv4 no longer relevant for mDNS. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Leaving mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Withdrawing address record for 192.168.6.14 on enp5s0. Jun 28 02:17:18 valkyrie named[801]: listening on IPv4 interface enp5s0, 192.168.6.14#53 Comparing the later manual activation of the link on Jun 28 02:40:10, with "ip link set enp5s0 up", the device comes right up as it always did on boot before this last update: Jun 28 02:40:10 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:40:10 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv4 for mDNS. Jun 28 02:40:10 valkyrie avahi-daemon[711]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 28 02:40:10 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:40:13 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 28 02:40:14 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv6 with address fe80::feaa:14ff:fe9e:3319. Jun 28 02:40:14 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv6 for mDNS. Jun 28 02:40:14 valkyrie avahi-daemon[711]: Registering new address record for fe80::feaa:14ff:fe9e:3319 on enp5s0.*. That's confirmed by the journal entries from the boot after the June 25 update to 6.15.3, e.g. Jun 25 03:49:36 valkyrie systemd[1]: Expecting device /sys/subsystem/net/devices/enp5s0... Jun 25 03:49:44 valkyrie kernel: r8169 0000:05:00.0 enp5s0: renamed from eth0 Jun 25 03:49:48 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 25 03:49:50 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 25 03:49:50 valkyrie avahi-daemon[729]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 25 03:49:50 valkyrie avahi-daemon[729]: New relevant interface enp5s0.IPv4 for mDNS. Jun 25 03:49:50 valkyrie avahi-daemon[729]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 25 03:49:52 valkyrie avahi-daemon[729]: Joining mDNS multicast group on interface enp5s0.IPv6 with address fe80::feaa:14ff:fe9e:3319. Jun 25 03:49:52 valkyrie avahi-daemon[729]: New relevant interface enp5s0.IPv6 for mDNS. Jun 25 03:49:52 valkyrie avahi-daemon[729]: Registering new address record for fe80::feaa:14ff:fe9e:3319 on enp5s0.*. The part that is different now is on boot after the device is brought up, it's being brought back down, and it occurs with the LTS kernel and the new 6.15.4 - so that should eliminate a kernel issue. The only other changes that look relevant and would be common between LTS boot and 6.15.4 are the firmware and systemd (257.6-1 -> 257.7-1) updates. (this is my best educated guess). How would we confirm this and then fix it? -- David C. Rankin, J.D.,P.E.

On Sat, Jun 28, 2025 at 01:24:44PM -0500, David C Rankin wrote:

On 6/28/25 12:09 PM, David C Rankin wrote:

<snip> I've been following this thread; and although I don't have any firm advice to give, I will say that I've *never* had any good results using those cruddy 'enpxxxx' interface names. I'm not a big fan of avahi, either. Merell -- <Mercury> You don't have to be crazy to be a member of the project, but you will be.. <=:]

On 6/28/25 1:24 PM, David C Rankin wrote:

On 6/28/25 12:09 PM, David C Rankin wrote:

...

   The only other changes that look relevant and would be common between LTS boot and 6.15.4 are the firmware and systemd (257.6-1 -> 257.7-1) updates. (this is my best educated guess). How would we confirm this and then fix it?

  Can I just downgrade systemd to test, or is that likely to break things (https://wiki.archlinux.org/title/Downgrading_packages)?

  The three packages I have in cache are:

systemd-257.6-1-x86_64.pkg.tar.zst systemd-257.7-1-x86_64.pkg.tar.zst

systemd-libs-257.6-1-x86_64.pkg.tar.zst systemd-libs-257.7-1-x86_64.pkg.tar.zst

systemd-sysvcompat-257.6-1-x86_64.pkg.tar.zst systemd-sysvcompat-257.7-1-x86_64.pkg.tar.zst

  If that won't hurt anything else, then we can at least rule-in/out systemd. Seems like that should be okay, but I'd rather ask, than compound the problem doing something that isn't advisable. If needed I can use the package archive to downgrade back to the 25th.

It's not the systemd update. I downgraded and same result, e.g. the interface is brought up and then brought back down by something? # pacman -Q | grep systemd systemd 257.6-1 systemd-libs 257.6-1 systemd-sysvcompat 257.6-1 The journal: # journalctl -b --no-pager --full | grep enp5 | head -n50 ** ******** boot ** Jun 28 23:58:21 valkyrie systemd[1]: Expecting device /sys/subsystem/net/devices/enp5s0... Jun 28 23:58:27 valkyrie kernel: r8169 0000:05:00.0 enp5s0: renamed from eth0 Jun 28 23:58:30 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 23:58:33 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 28 23:58:33 valkyrie avahi-daemon[723]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 23:58:33 valkyrie avahi-daemon[723]: New relevant interface enp5s0.IPv4 for mDNS. Jun 28 23:58:33 valkyrie avahi-daemon[723]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 28 23:58:33 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 23:58:33 valkyrie avahi-daemon[723]: Interface enp5s0.IPv4 no longer relevant for mDNS. Jun 28 23:58:33 valkyrie avahi-daemon[723]: Leaving mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 23:58:33 valkyrie avahi-daemon[723]: Withdrawing address record for 192.168.6.14 on enp5s0. Jun 28 23:58:36 valkyrie named[812]: listening on IPv4 interface enp5s0, 192.168.6.14#53 ** ******** manual activation ** Jun 29 00:01:11 valkyrie avahi-daemon[723]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 29 00:01:11 valkyrie avahi-daemon[723]: New relevant interface enp5s0.IPv4 for mDNS. Jun 29 00:01:11 valkyrie avahi-daemon[723]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 29 00:01:12 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 29 00:01:14 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 29 00:01:15 valkyrie avahi-daemon[723]: Joining mDNS multicast group on interface enp5s0.IPv6 with address fe80::feaa:14ff:fe9e:3319. Jun 29 00:01:15 valkyrie avahi-daemon[723]: New relevant interface enp5s0.IPv6 for mDNS. Jun 29 00:01:15 valkyrie avahi-daemon[723]: Registering new address record for fe80::feaa:14ff:fe9e:3319 on enp5s0.*. Again, manually bringing up the interface and restoring the default route and starting dhcpd4 and all is fine. At least in my mind, that pretty much leaves a problem created by the recent "churn" in the firmware and some impact that has had at least on the Realtek r8169 binary parts. Is it worth downgrading the firmware to check? -- David C. Rankin, J.D.,P.E.

David C Rankin <drankinatty@gmail.com> wrote:

On 6/29/25 12:19 AM, David C Rankin wrote:

...

Again, manually bringing up the interface and restoring the default route and starting dhcpd4 and all is fine. At least in my mind, that pretty much leaves a problem created by the recent "churn" in the firmware and some impact that has had at least on the Realtek r8169 binary parts.

  Is it worth downgrading the firmware to check?

Okay, I give... It's not the firmware (sorry gromit). I downgraded the firmware:

# pmu *firmware*20250613*zst loading packages... warning: downgrading package linux-firmware (20250627-1 => 20250613.12fe085f-9) warning: downgrading package linux-firmware-amdgpu (20250627-1 => 20250613.12fe085f-9) warning: downgrading package linux-firmware-atheros (20250627-1 => 20250613.12fe085f-9) warning: downgrading package linux-firmware-broadcom (20250627-1 => 20250613.12fe085f-9) warning: downgrading package linux-firmware-cirrus (20250627-1 => 20250613.12fe085f-9) warning: downgrading package linux-firmware-intel (20250627-1 => 20250613.12fe085f-9) warning: downgrading package linux-firmware-mediatek (20250627-1 => 20250613.12fe085f-9) warning: downgrading package linux-firmware-nvidia (20250627-1 => 20250613.12fe085f-9) warning: downgrading package linux-firmware-other (20250627-1 => 20250613.12fe085f-9) warning: downgrading package linux-firmware-radeon (20250627-1 => 20250613.12fe085f-9) warning: downgrading package linux-firmware-realtek (20250627-1 => 20250613.12fe085f-9) warning: downgrading package linux-firmware-whence (20250627-1 => 20250613.12fe085f-9) resolving dependencies... looking for conflicting packages...

Packages (12) linux-firmware-20250613.12fe085f-9 linux-firmware-amdgpu-20250613.12fe085f-9 linux-firmware-atheros-20250613.12fe085f-9 linux-firmware-broadcom-20250613.12fe085f-9 linux-firmware-cirrus-20250613.12fe085f-9 linux-firmware-intel-20250613.12fe085f-9 linux-firmware-mediatek-20250613.12fe085f-9 linux-firmware-nvidia-20250613.12fe085f-9 linux-firmware-other-20250613.12fe085f-9 linux-firmware-radeon-20250613.12fe085f-9 linux-firmware-realtek-20250613.12fe085f-9 linux-firmware-whence-20250613.12fe085f-9

Total Installed Size: 349.27 MiB Net Upgrade Size: -1.04 MiB

and it failed in the exact same way, enp5s0 is brought up as normal and then enp5s0 is mysterously taken back down. I'm all out of ideas at this point. I can't even think of anything else that changed between 6/25 and 6/28 except the kernel, and that wouldn't also effect booting LTS?

I went ahead and updated to current (pacman -Syu) and the problem persists. So the easiest workaround at this point is to write a short script that runs after boot to bring the interface back up, add the default route and start dhcpd4, e.g.

#!/bin/sh

## bring up the interface ip link set enp5s0 up

## set default route ip route add default via 192.168.6.11 dev enp5s0

## start dhcpd4 systemctl start dhcpd4

I guess systemd has a after-boot type service that could call the script. Anybody have any other guesses to investigate at why the interface gets taken back down on boot?

-- David C. Rankin, J.D.,P.E.

1) What you need the firmware for? Can you leave out only the firmware package you are actually using? Just trying to narrow the factors involved. 2) Can you share your network config? 3) systemd has debugging features. Which I am not familiar with. Perhaps using them could reveal why the interface is taken down. 4) Can you roll backward? I mean, not only systemd. Not only the firmware. Not only the kernel. Just roll back to a full state that was prevoiusly working. -- u34

David C Rankin <drankinatty@gmail.com> wrote:

On 6/29/25 12:55 AM, David C Rankin wrote:

...

  I went ahead and updated to current (pacman -Syu) and the problem persists. So the easiest workaround at this point is to write a short script that runs after boot to bring the interface back up, add the default route and start dhcpd4, e.g. <snip>

Could this be something new with netctl? systemd is showing the profile failed due to "resolvconf: signature mismatch: /etc/resolv.conf". I don't not use systemd-resolvconf and never have. The unit file failure shows the following:

# systemctl --no-pager --full status 'netctl@ethernet\x2dstatis\x2dvalkyrie.service' × netctl@ethernet\x2dstatis\x2dvalkyrie.service - A basic static ethernet connection Loaded: loaded (/usr/lib/systemd/system/netctl@.service; enabled; preset: 5:185mdisabled) Drop-In: /etc/systemd/system/netctl@ethernet\x2dstatis\x2dvalkyrie.service.d └─profile.conf Active: failed (Result: exit-code) since Sun 2025-06-29 00:44:59 CDT; 2h 6min ago Duration: 1.157s Invocation: 07b4efd73c69447db4d62112dccf3ce3 Docs: man:netctl.profile(5) Process: 779 ExecStart=/usr/lib/netctl/network start ethernet-statis-valkyrie (code=exited, status=1/FAILURE) Main PID: 779 (code=exited, status=1/FAILURE) Mem peak: 7.3M CPU: 213ms

Jun 29 00:44:55 valkyrie systemd[1]: Starting A basic static ethernet connection... Jun 29 00:44:55 valkyrie network[779]: Starting network profile 'ethernet-statis-valkyrie'... Jun 29 00:44:57 valkyrie systemd[1]: Started A basic static ethernet connection. Jun 29 00:44:58 valkyrie network[847]: resolvconf: signature mismatch: /etc/resolv.conf Jun 29 00:44:58 valkyrie network[847]: resolvconf: run `resolvconf -u` to update Jun 29 00:44:59 valkyrie systemd[1]: netctl@ethernet\x2dstatis\x2dvalkyrie.service: Main process exited, code=exited, status=1/FAILURE Jun 29 00:44:59 valkyrie network[779]: Failed to bring the network up for profile 'ethernet-statis-valkyrie' Jun 29 00:44:59 valkyrie systemd[1]: 5:185m5:185mnetctl@ethernet\x2dstatis\x2dvalkyrie.service: Failed with result 'exit-code'.

This failure is new since the Jun 28 update. The resolve.conf file hasn't changed since 2022 (the way I set it and it provides the backup nameserver as well), e.g.

$ l /etc/resolv.conf -rw-r--r-- 1 root root 313 Dec 10 2022 /etc/resolv.conf

$ cat /etc/resolv.conf ## Generated by dcr # set 1 sec timeout to use alternate NS and prevent parallel requests options timeout:1 options single-request # all lookup using hostnames rather than FQDN domain 3111skyline.com search 3111skyline.com # nameservers to use nameserver 127.0.0.1 nameserver 192.168.6.14 nameserver 192.168.6.17

The netctl config (based on the /example for static) hasn't changed Nov 6 2022:

$ cat cat ethernet-statis-valkyrie Description='A basic static ethernet connection' Interface=enp5s0 Connection=ethernet IP=static Address=('192.168.6.14/24') #Routes=('192.168.0.0/24 via 192.168.1.2') Gateway='192.168.6.11' DNS=('192.168.6.14')

## For IPv6 autoconfiguration #IP6=stateless

## For IPv6 static address configuration #IP6=static #Address6=('1234:5678:9abc:def::1/64' '1234:3456::123/96') #Routes6=('abcd::1234') #Gateway6='1234:0:123::abcd'

The profile hasn't changed since that same date:

$ l /etc/systemd/system/netctl@ethernet\\x2dstatis\\x2dvalkyrie.service.d/ total 12 drwxr-xr-x 2 root root 4096 Nov 6 2022 . drwxr-xr-x 10 root root 4096 May 6 14:01 .. -rw-r--r-- 1 root root 148 Nov 6 2022 profile.conf

$ cat /etc/systemd/system/netctl@ethernet\\x2dstatis\\x2dvalkyrie.service.d/profile.conf [Unit] Description=A basic static ethernet connection BindsTo=sys-subsystem-net-devices-enp5s0.device After=sys-subsystem-net-devices-enp5s0.device

Why would this netctl unit suddenly start failing? Is the 'resolvconf: signature mismatch: /etc/resolv.conf' now something that causes the unit to fail? That is the only thing shown for the unit and then systemd seems to be taking the interface down after that?

If it is systemd and netctl - why didn't downgrading systemd fix it? netctl hasn't changed in almost a year:

[2024-07-06T18:27:27-0500] [ALPM] upgraded netctl (1.29-1 -> 1.29-2)

I have got to get this sorted out. The network being taken down causes havoc in the journal with named and the repeated restart attempts. Not to mention not being able to reboot into a running configuration. Manually bringing the interface up and restoring the default route is a quick work-around and not a solution.

What package would a bug even be written against here? systemd? netctl? Any/all thoughts welcome. I've been through 1000s lines of dmesg, the journal, the network configs, and my eyes are starting to cross. I'm done for the night and will check back tomorrow. Frustrating...

netctl itself? openresolv? What dhcp package are you using? -- u34

-- David C. Rankin, J.D.,P.E.

On 6/29/25 4:27 AM, David C Rankin wrote:

  Ironically (and also hiding the problem) when the interface is brought up manually with "ip set enp5s0 up" there must be no check of resolv.conf. Otherwise, if that is what is causing the issue, I would expect bringing the interface up to result in the same error and it to go back down. But it doesn't. After being brought up, the server is running perfectly, just as it did before the Jun 28 update.

  Hah! we may have just found it. openresolv was updated in the Jun 28 update. That may be the culprit. I'll downgrade reboot and report.

I'll be damned! It's openresolv and the update on 6/28 broke the netctl unit startup: [2025-06-28T01:54:53-0500] [ALPM] upgraded openresolv (3.16.5-1 -> 3.17.0-1) I downgraded openresolv, rebooted and BINGO, server comes up perfectly, network is up, all it good in the world. # pmu openresolv-3.16.5-1-any.pkg.tar.zst loading packages... warning: downgrading package openresolv (3.17.0-1 => 3.16.5-1) resolving dependencies... looking for conflicting packages... Packages (1) openresolv-3.16.5-1 Total Installed Size: 0.08 MiB Net Upgrade Size: 0.00 MiB :: Proceed with installation? [Y/n] (1/1) checking keys in keyring [########################################] 100% (1/1) checking package integrity [########################################] 100% (1/1) loading package files [########################################] 100% (1/1) checking for file conflicts [########################################] 100% (1/1) checking available disk space [########################################] 100% :: Processing package changes... (1/1) downgrading openresolv [########################################] 100% :: Running post-transaction hooks... (1/1) Arming ConditionNeedsUpdate... (reboot) We have a winner! openresolv gets the bug. One hell of a process of elimination :) Thanks to all for suggestions that got this identified. I'm going to bed... -- David C. Rankin, J.D.,P.E.

Do you still have problems if you disable the resolved.conf update? <https://man.archlinux.org/man/resolvconf.conf.5>

The following variables affect resolv.conf(5) directly:

resolv_conf Defaults to /etc/resolv.conf if not set. Set to /dev/null to stop resolvconf(8) from changing it.

Am 30. Juni 2025 00:01:59 MESZ schrieb David C Rankin <drankinatty@gmail.com>:

On 6/29/25 5:17 AM, David C Rankin wrote:

...

On 6/29/25 4:37 AM, David C Rankin wrote:

...

   We have a winner! openresolv gets the bug. One hell of a process of elimination :)  Thanks to all for suggestions that got this identified. I'm going to bed...

Upstream bug:

openresolv 3.17.0-1 breaks netctl unit startup if resolv.conf has immutable attribute set https://github.com/NetworkConfiguration/openresolv/issues/38

Gitlab issue opened on openresolv to track upstream bug report:

https://gitlab.archlinux.org/archlinux/packaging/packages/openresolv/-/issue...

-- David C. Rankin, J.D.,P.E.

On 7/1/25 12:53 PM, David C Rankin wrote:

On 6/30/25 12:31 AM, ProgAndy wrote:

...

Do you still have problems if you disable the resolved.conf update?

Downgrading openresolv 3.17 => 3.16.5 fixes the problem.

The crux of the issue for me is that the /etc/netctl/examples/ethernet- static example included a DNS entry which I filled in as the example expected.

The openresolv code was changed between 3.16.5 and 3.17.0 so that if that DNS entry is in your netctl config file and thee signature at the top of /etc/resolv.conf isn't one that openresolv recognized, or openresolv couldn't write to /etc/resolv.conf -- the 3.17 version marked that network startup unit as FAILED and took down the running network.

This also impacted users or wiregruard.

On suggestion by the openresolv maintainer, I removed, the DNS entry from the netctl configuration file, and that caused openresolv to ignore the fact it couldn't write to /etc/resolv.conf. So I can use 3.17.0 now.

There are a couple solutions being considered, but simply having some documentation explaining the side-effect of the DNS entry in the netctl config would have been sufficient for my issue.

The big issue is there is no clearly documented way to turn off modification of /etc/resolv.conf for those who self-admin that file on servers and the like. I'm pushing for the solution to include that option as well.

If you have a netctl config file in /etc/netctl and you don't want resolvconf to treat your network startup as FAILED, you can try removing the DNS entry. It worked fine for me with my fixed-IP setup for the server. It should work fine for all type of network configs as long as you proper set your DNS domains /etc/resolv.conf.

All of the "nanny" dynamic update of resolv.conf is supposed to help with boxes that move between different networks, different WiFi locations, etc... It is trying to apply what works there are a "one size fits all" solution and that has caused issues for servers in fixed locations where /etc/resolv.conf is self-adminned. Simply providing a documented way to turn this behavior off would solve 90% of the issues seen.

openresolv keys off the "signature' at the top of /etc/resolv.conf to determine who wrote it and whether it can be modified. The problem in my case was for signatures not know to openresolve, the new update was marking the network startup used as failed.

A better solutions would be for openresolv to only attempt dynamic modification and error reporting for /etc/resolv.conf files where the signature is *known* to it. Otherwise for /etc/resolv.conf file with no signature or a signature not know to openresolv, it should do nothing regarding those files and presume they are self-administered. It could throw a warning if it just had to do something, but not mark the network startup as failed.

I'm sure they will come up with a workable solutions. Good discussion in the upstream bug report. For now, just removing the DNS entry from the config file in /etc/netctl/xyz... allows the unit file startup to succeed.

The trouble you have/had with networking is exactly why I removed network manager and openresolv, in fact I removed dhcp clients and all the other newtorking things and I just use systemd-networkd. I also use systemd-resolved on some systems that don't use my pi-hole server. Since I use all of systemd networking,I have zero networking issues. systemd-timedated.service systemd-networkd.service systemd-resolved.service -- Hindi madali ang maging ako

On 6/28/25 6:29 AM, Andy Pieters wrote:

On Sat, 28 Jun 2025 at 09:18, David C Rankin <drankinatty@gmail.com <mailto:drankinatty@gmail.com>> wrote:

All,

   I have my Arch server that has been running for a decade, updates weekly, never misses a beat. It has been so long since I've had an issue with it, I thought Arch had finally reached Linux rolling release nirvana.

   Then, for reasons I can't explain, the update tonight

Whilst we wait for other insight, can we assume that you run pacdiff after every update, to make sure all the configuration files are compatible with updated software?

Given this is a server and the configs highly tweaked, I manually merge them, but yes, I check regularly (and scan the pacman -Syu update for notes about any manual changes needed) I don't use pacdiff as it is impossible to read with the colors on traditional Linux color back background in konsole/xterm. I also have LTS on this server, so I booted the LTS kernel and it failed in the exact same way. Network interface down, dhcpd4 failed to start. (all other services are running, iptables, ipset, fail2ban, named, avahi, apache/PHP, postfix, dovecot, sshd, samba, nut, mariadb, postgresql, mongodb, etc..) In dmesg output, curiously, the network device is brought up, but then gets taken down again until I manually start it, e.g. # dmesg | grep -i enp5 [ 23.138144] systemd[1]: Expecting device /sys/subsystem/net/devices/enp5s0... [ 26.867849] r8169 0000:05:00.0 enp5s0: renamed from eth0 [ 31.170658] r8169 0000:05:00.0 enp5s0: Link is Down [ 33.382008] r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx [ 34.212934] r8169 0000:05:00.0 enp5s0: Link is Down ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [ 1409.078729] r8169 0000:05:00.0 enp5s0: Link is Down [ 1411.306823] r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx The journal entries shows the interface coming up, and then being taken down with no explanation for why it goes down, then it shows IPv6 coming up as well when I manually bring the interface up a second time: # journalctl -b --no-pager --full | grep enp5 | head -n50 Jun 28 02:17:05 valkyrie systemd[1]: Expecting device /sys/subsystem/net/devices/enp5s0... Jun 28 02:17:08 valkyrie kernel: r8169 0000:05:00.0 enp5s0: renamed from eth0 Jun 28 02:17:12 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:17:15 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 28 02:17:15 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:17:15 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv4 for mDNS. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 28 02:17:15 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 28 02:17:15 valkyrie avahi-daemon[711]: Interface enp5s0.IPv4 no longer relevant for mDNS. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Leaving mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Withdrawing address record for 192.168.6.14 on enp5s0. Jun 28 02:17:18 valkyrie named[801]: listening on IPv4 interface enp5s0, 192.168.6.14#53 Jun 28 02:40:10 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:40:10 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv4 for mDNS. Jun 28 02:40:10 valkyrie avahi-daemon[711]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 28 02:40:10 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:40:13 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 28 02:40:14 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv6 with address fe80::feaa:14ff:fe9e:3319. Jun 28 02:40:14 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv6 for mDNS. Jun 28 02:40:14 valkyrie avahi-daemon[711]: Registering new address record for fe80::feaa:14ff:fe9e:3319 on enp5s0.*. Jun 28 02:41:43 valkyrie dhcpd[1781]: Listening on LPF/enp5s0/fc:aa:14:9e:33:19/192.168.6.0/24 Jun 28 02:41:43 valkyrie dhcpd[1781]: Sending on LPF/enp5s0/fc:aa:14:9e:33:19/192.168.6.0/24 Jun 28 02:41:47 valkyrie dhcpd[1781]: DHCPDISCOVER from 1c:93:c4:5b:30:87 (amazon-90e3375b8) via enp5s0 Jun 28 02:41:47 valkyrie dhcpd[1781]: DHCPOFFER on 192.168.6.113 to 1c:93:c4:5b:30:87 (amazon-90e3375b8) via enp5s0 Jun 28 02:41:47 valkyrie dhcpd[1781]: DHCPREQUEST for 192.168.6.113 (192.168.6.14) from 1c:93:c4:5b:30:87 (amazon-90e3375b8) via enp5s0 Jun 28 02:41:47 valkyrie dhcpd[1781]: DHCPACK on 192.168.6.113 to 1c:93:c4:5b:30:87 (amazon-90e3375b8) via enp5s0 Jun 28 02:41:58 valkyrie dhcpd[1781]: DHCPREQUEST for 192.168.6.114 from b8:27:eb:67:78:cc (rp3bz) via enp5s0 <snip> This has never happened before. Let me know what else you want to see and I'll get it. Why is my enp5s0 device being brought up and then being taken down? Here are the relevant configs: /etc/dhcpd.conf option domain-name "mydomain.tld"; option routers gw.mydomain.tld; option ntp-servers ns1.mydomain.tld; option netbios-name-servers ns1.mydomain.tld; option domain-name-servers 192.168.6.14, 192.168.6.17; default-lease-time 3600; max-lease-time 28800; ddns-update-style interim; ddns-updates on; ignore client-updates; include "/etc/named.keys"; include "/etc/dhcp_updater.key"; authoritative; subnet 192.168.6.0 netmask 255.255.255.0 { option domain-name "mydomain.tld"; ddns-domainname "mydomain.tld"; option subnet-mask 255.255.255.0; option broadcast-address 192.168.6.255; range dynamic-bootp 192.168.6.110 192.168.6.224; default-lease-time 7200; max-lease-time 28800; zone mydomain.tld. {primary 127.0.0.1; key dhcp_updater; } zone 6.168.192.in-addr.arpa. { primary 127.0.0.1; key dhcp_updater; } host wizard.mydomain.tld { hardware ethernet 24:77:03:21:56:3c; fixed-address 192.168.6.104; } host wiz.mydomain.tld { hardware ethernet e4:11:5b:25:bf:d0; #eth0 fixed-address 192.168.6.105; } host picow1.mydomain.tld { hardware ethernet fe:ed:de:ad:ba:be; fixed-address 192.168.6.106; } } /etc/systemd/system/netctl@ethernet\\x2dstatis\\x2dvalkyrie.service.d/profile.conf [Unit] Description=A basic static ethernet connection BindsTo=sys-subsystem-net-devices-enp5s0.device After=sys-subsystem-net-devices-enp5s0.device There is nothing in /etc/systemd/network on any of my arch boxes. The network config in /etc/netctl/ethernet-static-valkyrie (unchanged since Nov 6 2022) is: Description='A basic static ethernet connection' Interface=enp5s0 Connection=ethernet IP=static Address=('192.168.6.14/24') Gateway='192.168.6.11' DNS=('192.168.6.14') ## For IPv6 autoconfiguration #IP6=stateless ## For IPv6 static address configuration #IP6=static #Address6=('1234:5678:9abc:def::1/64' '1234:3456::123/96') #Routes6=('abcd::1234') #Gateway6='1234:0:123::abcd' Any ideas? -- David C. Rankin, J.D.,P.E.