On 6/28/25 4:45 AM, Ralph Corderoy wrote:

Hi David,

...

the update tonight (an hour ago), failed to start dhcpd4. I can't ping the box, I have to log in locally and I find the network interface enp5s0 'down' - that's not happened since Arch was installed.

Did it reboot because of that update? If so, could the damage have been caused by some other change made since the previous boot which only took effect on booting?

Thanks Ralf, No, this was just a normal pacman -Syu and then a manual reboot because of the kernel update. Nothing strange. I watch the rss feed for kernel updates in core and always update for each new kernel and then manually reboot. There is something that definitely changed in how the network interface is being handled. It is brought up fine, but then is taken down??? See last post for details and config files, but the crux of it from the journal is: # journalctl -b --no-pager --full | grep enp5 | head -n50 Jun 28 02:17:05 valkyrie systemd[1]: Expecting device /sys/subsystem/net/devices/enp5s0... Jun 28 02:17:08 valkyrie kernel: r8169 0000:05:00.0 enp5s0: renamed from eth0 Jun 28 02:17:12 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:17:15 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx ************************************************************************ Jun 28 02:17:15 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:17:15 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv4 for mDNS. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 28 02:17:15 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down ************************************************************************ Jun 28 02:17:15 valkyrie avahi-daemon[711]: Interface enp5s0.IPv4 no longer relevant for mDNS. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Leaving mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Withdrawing address record for 192.168.6.14 on enp5s0. Jun 28 02:17:18 valkyrie named[801]: listening on IPv4 interface enp5s0, 192.168.6.14#53 Jun 28 02:40:10 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:40:10 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv4 for mDNS. Jun 28 02:40:10 valkyrie avahi-daemon[711]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 28 02:40:10 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:40:13 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 28 02:40:14 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv6 with address fe80::feaa:14ff:fe9e:3319. Jun 28 02:40:14 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv6 for mDNS. Jun 28 02:40:14 valkyrie avahi-daemon[711]: Registering new address record for fe80::feaa:14ff:fe9e:3319 on enp5s0.*. Jun 28 02:41:43 valkyrie dhcpd[1781]: Listening on LPF/enp5s0/fc:aa:14:9e:33:19/192.168.6.0/24 Jun 28 02:41:43 valkyrie dhcpd[1781]: Sending on LPF/enp5s0/fc:aa:14:9e:33:19/192.168.6.0/24 Huh??? Where did the IPv4 interface go? And why? -- David C. Rankin, J.D.,P.E.

On 6/28/25 11:04 AM, David C Rankin wrote:

here has definitely been a change in how (whatever) looks at IPv4 and IPv6 and chooses the interface to use. Here is the same journal entries from the last boot of kernel 6.15.3 and IPv4 is configured and stays UP. So whatever happened, it looks like the logic governing whether to use IPv4 or IPv6 changed in this last update and that is what is taking my IPv4 interface down, e.g.

No, it's not an IPv4/IPv6 logic choice issue, it is simply that on boot the interfaces is configured for IPv4 and brought up, but then is immediately taken down again by something. The later manual bringing up of the interface configured it as it has always been with mdns registered for both IPv4 and IPv6. The difference being on manual activation the interface stays up. It's looking like there may have been a change in how systemd is managing the interface. Something takes the interface back down during boot. It is the boot activation where something has changed. The boot entries are the Jun 28 02:17:05 journal entries: Jun 28 02:17:05 valkyrie systemd[1]: Expecting device /sys/subsystem/net/devices/enp5s0... Jun 28 02:17:08 valkyrie kernel: r8169 0000:05:00.0 enp5s0: renamed from eth0 Jun 28 02:17:12 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:17:15 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 28 02:17:15 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:17:15 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv4 for mDNS. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 28 02:17:15 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:17:15 valkyrie avahi-daemon[711]: Interface enp5s0.IPv4 no longer relevant for mDNS. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Leaving mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Withdrawing address record for 192.168.6.14 on enp5s0. Jun 28 02:17:18 valkyrie named[801]: listening on IPv4 interface enp5s0, 192.168.6.14#53 Comparing the later manual activation of the link on Jun 28 02:40:10, with "ip link set enp5s0 up", the device comes right up as it always did on boot before this last update: Jun 28 02:40:10 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:40:10 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv4 for mDNS. Jun 28 02:40:10 valkyrie avahi-daemon[711]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 28 02:40:10 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:40:13 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 28 02:40:14 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv6 with address fe80::feaa:14ff:fe9e:3319. Jun 28 02:40:14 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv6 for mDNS. Jun 28 02:40:14 valkyrie avahi-daemon[711]: Registering new address record for fe80::feaa:14ff:fe9e:3319 on enp5s0.*. That's confirmed by the journal entries from the boot after the June 25 update to 6.15.3, e.g. Jun 25 03:49:36 valkyrie systemd[1]: Expecting device /sys/subsystem/net/devices/enp5s0... Jun 25 03:49:44 valkyrie kernel: r8169 0000:05:00.0 enp5s0: renamed from eth0 Jun 25 03:49:48 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 25 03:49:50 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 25 03:49:50 valkyrie avahi-daemon[729]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 25 03:49:50 valkyrie avahi-daemon[729]: New relevant interface enp5s0.IPv4 for mDNS. Jun 25 03:49:50 valkyrie avahi-daemon[729]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 25 03:49:52 valkyrie avahi-daemon[729]: Joining mDNS multicast group on interface enp5s0.IPv6 with address fe80::feaa:14ff:fe9e:3319. Jun 25 03:49:52 valkyrie avahi-daemon[729]: New relevant interface enp5s0.IPv6 for mDNS. Jun 25 03:49:52 valkyrie avahi-daemon[729]: Registering new address record for fe80::feaa:14ff:fe9e:3319 on enp5s0.*. The part that is different now is on boot after the device is brought up, it's being brought back down, and it occurs with the LTS kernel and the new 6.15.4 - so that should eliminate a kernel issue. The only other changes that look relevant and would be common between LTS boot and 6.15.4 are the firmware and systemd (257.6-1 -> 257.7-1) updates. (this is my best educated guess). How would we confirm this and then fix it? -- David C. Rankin, J.D.,P.E.

On Sat, Jun 28, 2025 at 01:24:44PM -0500, David C Rankin wrote:

On 6/28/25 12:09 PM, David C Rankin wrote:

<snip> I've been following this thread; and although I don't have any firm advice to give, I will say that I've *never* had any good results using those cruddy 'enpxxxx' interface names. I'm not a big fan of avahi, either. Merell -- <Mercury> You don't have to be crazy to be a member of the project, but you will be.. <=:]

On 6/28/25 6:29 AM, Andy Pieters wrote:

On Sat, 28 Jun 2025 at 09:18, David C Rankin <drankinatty@gmail.com <mailto:drankinatty@gmail.com>> wrote:

All,

   I have my Arch server that has been running for a decade, updates weekly, never misses a beat. It has been so long since I've had an issue with it, I thought Arch had finally reached Linux rolling release nirvana.

   Then, for reasons I can't explain, the update tonight

Whilst we wait for other insight, can we assume that you run pacdiff after every update, to make sure all the configuration files are compatible with updated software?

Given this is a server and the configs highly tweaked, I manually merge them, but yes, I check regularly (and scan the pacman -Syu update for notes about any manual changes needed) I don't use pacdiff as it is impossible to read with the colors on traditional Linux color back background in konsole/xterm. I also have LTS on this server, so I booted the LTS kernel and it failed in the exact same way. Network interface down, dhcpd4 failed to start. (all other services are running, iptables, ipset, fail2ban, named, avahi, apache/PHP, postfix, dovecot, sshd, samba, nut, mariadb, postgresql, mongodb, etc..) In dmesg output, curiously, the network device is brought up, but then gets taken down again until I manually start it, e.g. # dmesg | grep -i enp5 [ 23.138144] systemd[1]: Expecting device /sys/subsystem/net/devices/enp5s0... [ 26.867849] r8169 0000:05:00.0 enp5s0: renamed from eth0 [ 31.170658] r8169 0000:05:00.0 enp5s0: Link is Down [ 33.382008] r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx [ 34.212934] r8169 0000:05:00.0 enp5s0: Link is Down ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [ 1409.078729] r8169 0000:05:00.0 enp5s0: Link is Down [ 1411.306823] r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx The journal entries shows the interface coming up, and then being taken down with no explanation for why it goes down, then it shows IPv6 coming up as well when I manually bring the interface up a second time: # journalctl -b --no-pager --full | grep enp5 | head -n50 Jun 28 02:17:05 valkyrie systemd[1]: Expecting device /sys/subsystem/net/devices/enp5s0... Jun 28 02:17:08 valkyrie kernel: r8169 0000:05:00.0 enp5s0: renamed from eth0 Jun 28 02:17:12 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:17:15 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 28 02:17:15 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:17:15 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv4 for mDNS. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 28 02:17:15 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 28 02:17:15 valkyrie avahi-daemon[711]: Interface enp5s0.IPv4 no longer relevant for mDNS. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Leaving mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:17:15 valkyrie avahi-daemon[711]: Withdrawing address record for 192.168.6.14 on enp5s0. Jun 28 02:17:18 valkyrie named[801]: listening on IPv4 interface enp5s0, 192.168.6.14#53 Jun 28 02:40:10 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 28 02:40:10 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv4 for mDNS. Jun 28 02:40:10 valkyrie avahi-daemon[711]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 28 02:40:10 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 28 02:40:13 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 28 02:40:14 valkyrie avahi-daemon[711]: Joining mDNS multicast group on interface enp5s0.IPv6 with address fe80::feaa:14ff:fe9e:3319. Jun 28 02:40:14 valkyrie avahi-daemon[711]: New relevant interface enp5s0.IPv6 for mDNS. Jun 28 02:40:14 valkyrie avahi-daemon[711]: Registering new address record for fe80::feaa:14ff:fe9e:3319 on enp5s0.*. Jun 28 02:41:43 valkyrie dhcpd[1781]: Listening on LPF/enp5s0/fc:aa:14:9e:33:19/192.168.6.0/24 Jun 28 02:41:43 valkyrie dhcpd[1781]: Sending on LPF/enp5s0/fc:aa:14:9e:33:19/192.168.6.0/24 Jun 28 02:41:47 valkyrie dhcpd[1781]: DHCPDISCOVER from 1c:93:c4:5b:30:87 (amazon-90e3375b8) via enp5s0 Jun 28 02:41:47 valkyrie dhcpd[1781]: DHCPOFFER on 192.168.6.113 to 1c:93:c4:5b:30:87 (amazon-90e3375b8) via enp5s0 Jun 28 02:41:47 valkyrie dhcpd[1781]: DHCPREQUEST for 192.168.6.113 (192.168.6.14) from 1c:93:c4:5b:30:87 (amazon-90e3375b8) via enp5s0 Jun 28 02:41:47 valkyrie dhcpd[1781]: DHCPACK on 192.168.6.113 to 1c:93:c4:5b:30:87 (amazon-90e3375b8) via enp5s0 Jun 28 02:41:58 valkyrie dhcpd[1781]: DHCPREQUEST for 192.168.6.114 from b8:27:eb:67:78:cc (rp3bz) via enp5s0 <snip> This has never happened before. Let me know what else you want to see and I'll get it. Why is my enp5s0 device being brought up and then being taken down? Here are the relevant configs: /etc/dhcpd.conf option domain-name "mydomain.tld"; option routers gw.mydomain.tld; option ntp-servers ns1.mydomain.tld; option netbios-name-servers ns1.mydomain.tld; option domain-name-servers 192.168.6.14, 192.168.6.17; default-lease-time 3600; max-lease-time 28800; ddns-update-style interim; ddns-updates on; ignore client-updates; include "/etc/named.keys"; include "/etc/dhcp_updater.key"; authoritative; subnet 192.168.6.0 netmask 255.255.255.0 { option domain-name "mydomain.tld"; ddns-domainname "mydomain.tld"; option subnet-mask 255.255.255.0; option broadcast-address 192.168.6.255; range dynamic-bootp 192.168.6.110 192.168.6.224; default-lease-time 7200; max-lease-time 28800; zone mydomain.tld. {primary 127.0.0.1; key dhcp_updater; } zone 6.168.192.in-addr.arpa. { primary 127.0.0.1; key dhcp_updater; } host wizard.mydomain.tld { hardware ethernet 24:77:03:21:56:3c; fixed-address 192.168.6.104; } host wiz.mydomain.tld { hardware ethernet e4:11:5b:25:bf:d0; #eth0 fixed-address 192.168.6.105; } host picow1.mydomain.tld { hardware ethernet fe:ed:de:ad:ba:be; fixed-address 192.168.6.106; } } /etc/systemd/system/netctl@ethernet\\x2dstatis\\x2dvalkyrie.service.d/profile.conf [Unit] Description=A basic static ethernet connection BindsTo=sys-subsystem-net-devices-enp5s0.device After=sys-subsystem-net-devices-enp5s0.device There is nothing in /etc/systemd/network on any of my arch boxes. The network config in /etc/netctl/ethernet-static-valkyrie (unchanged since Nov 6 2022) is: Description='A basic static ethernet connection' Interface=enp5s0 Connection=ethernet IP=static Address=('192.168.6.14/24') Gateway='192.168.6.11' DNS=('192.168.6.14') ## For IPv6 autoconfiguration #IP6=stateless ## For IPv6 static address configuration #IP6=static #Address6=('1234:5678:9abc:def::1/64' '1234:3456::123/96') #Routes6=('abcd::1234') #Gateway6='1234:0:123::abcd' Any ideas? -- David C. Rankin, J.D.,P.E.