On 6/29/25 3:57 AM, u34@net9.cf wrote:
What package would a bug even be written against here? systemd? netctl? Any/all thoughts welcome. I've been through 1000s lines of dmesg, the journal, the network configs, and my eyes are starting to cross. I'm done for the night and will check back tomorrow. Frustrating...
netctl itself? openresolv? What dhcp package are you using?
Thank you for the follow-up: I'm using ISC dhcp: dhcp 4.4.3.P1-4 along with bind for name resolution: bind 9.20.10-1 That provides the cleanest way for dhcp to provide dynamic updates to DNS and have a secondary server configured for backup DNS that takes zone transfers from the primary. I don't they dhcp is at fault. It is a symptom of the netctl unit failing rather than the cause. This is a good Sherlock Holmes mystery as far as "Who dunnit?"'. If this were a new setup, then I could see having to work though everything, but with this box running in the same configuration for almost a decade - something changed... but what? The only other piece that may somehow now be causing issues with resolv.conf that never did before is that the file has the immutable attribute set to keep it from changing. There were significant problems with it being overwritten, and taking from the 2022 date it would have been sometime late November 2022 (maybe early December) when that started happening. To prevent it, the immutable attribute was set on resolv.conf (we can probably find that thread in the archives) The latest change to anything network related on this box was Dec 9 2022 and it has run perfectly since with updates and reboots for every kernel. Literally that is the only time the server is rebooted (except for hurricanes and ice-storms where power-outages extend beyond the UPS battery life) So I'm completely perplexed and at a lost as the WTF happened to start causing the enp5s0 interface to start normally but then be taken done and the unit to be marked as FAILED. That's just not something that is possible to overlook on a reboot :) I summarized the journal picking though it line by line (as well as dmesg -- which is basically reflected in the journal) and the relevant entries seem to be: journal: Jun 29 00:44:52 valkyrie kernel: r8169 0000:05:00.0: can't disable ASPM; OS doesn't have ASPM control Jun 29 00:44:52 valkyrie kernel: r8169 0000:05:00.0 eth0: RTL8168evl/8111evl, fc:aa:14:9e:33:19, XID 2c9, IRQ 36 Jun 29 00:44:52 valkyrie kernel: r8169 0000:05:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] Jun 29 00:44:52 valkyrie kernel: r8169 0000:05:00.0 enp5s0: renamed from eth0 Jun 29 00:44:47 valkyrie systemd[1]: Started Network Time Synchronization. Jun 29 00:44:47 valkyrie systemd[1]: Reached target System Time Set. Jun 29 00:44:48 valkyrie systemd[1]: Found device RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (Onboard Ethernet). Jun 29 00:44:51 valkyrie systemd-timesyncd[383]: Network configuration changed, trying to establish connection. Jun 29 00:44:51 valkyrie systemd[1]: Listening on System Extension Image Management. Jun 29 00:44:51 valkyrie systemd-timesyncd[383]: Network configuration changed, trying to establish connection. Jun 29 00:44:53 valkyrie systemd-timesyncd[383]: Network configuration changed, trying to establish connection. Jun 29 00:44:53 valkyrie systemd-timesyncd[383]: Network configuration changed, trying to establish connection. Jun 29 00:44:53 valkyrie systemd-timesyncd[383]: Network configuration changed, trying to establish connection. Jun 29 00:44:53 valkyrie systemd-timesyncd[383]: Network configuration changed, trying to establish connection. Jun 29 00:44:54 valkyrie systemd[1]: Listening on Avahi mDNS/DNS-SD Stack Activation Socket. Jun 29 00:44:54 valkyrie systemd[1]: Listening on D-Bus System Message Bus Socket. Jun 29 00:44:54 valkyrie systemd[1]: Listening on GnuPG network certificate management daemon for /etc/pacman.d/gnupg. Jun 29 00:44:54 valkyrie systemd[1]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers) for /etc/pacman.d/gnupg. Jun 29 00:44:54 valkyrie systemd[1]: Listening on GnuPG cryptographic agent and passphrase cache (restricted) for /etc/pacman.d/gnupg. Jun 29 00:44:54 valkyrie systemd[1]: Listening on GnuPG cryptographic agent (ssh-agent emulation) for /etc/pacman.d/gnupg. Jun 29 00:44:54 valkyrie systemd[1]: Listening on GnuPG cryptographic agent and passphrase cache for /etc/pacman.d/gnupg. Jun 29 00:44:54 valkyrie systemd[1]: Listening on GnuPG public key management service for /etc/pacman.d/gnupg. Jun 29 00:44:54 valkyrie systemd[1]: Listening on OpenSSH Server Socket (systemd-ssh-generator, AF_UNIX Local). Jun 29 00:44:54 valkyrie systemd[1]: Listening on Hostname Service Socket. Jun 29 00:44:54 valkyrie systemd[1]: Reached target Socket Units. Jun 29 00:44:54 valkyrie systemd[1]: Starting Avahi mDNS/DNS-SD Stack... Jun 29 00:44:54 valkyrie systemd[1]: Starting Loading IP Sets... Jun 29 00:44:54 valkyrie systemd[1]: Starting IPv4 Packet Filtering Framework... Jun 29 00:44:54 valkyrie avahi-daemon[724]: Found user 'avahi' (UID 84) and group 'avahi' (GID 84). Jun 29 00:44:54 valkyrie avahi-daemon[724]: Successfully dropped root privileges. Jun 29 00:44:54 valkyrie avahi-daemon[724]: avahi-daemon 0.8 starting up. Jun 29 00:44:54 valkyrie avahi-daemon[724]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns! Jun 29 00:44:54 valkyrie systemd[1]: Started Avahi mDNS/DNS-SD Stack. Jun 29 00:44:54 valkyrie avahi-daemon[724]: Successfully called chroot(). Jun 29 00:44:54 valkyrie avahi-daemon[724]: Successfully dropped remaining capabilities. Jun 29 00:44:54 valkyrie avahi-daemon[724]: Loading service file /services/LaserJet.service. Jun 29 00:44:54 valkyrie avahi-daemon[724]: Loading service file /services/dcr4100n.service. Jun 29 00:44:54 valkyrie avahi-daemon[724]: Joining mDNS multicast group on interface lo.IPv6 with address ::1. Jun 29 00:44:54 valkyrie avahi-daemon[724]: New relevant interface lo.IPv6 for mDNS. Jun 29 00:44:54 valkyrie avahi-daemon[724]: Joining mDNS multicast group on interface lo.IPv4 with address 127.0.0.1. Jun 29 00:44:54 valkyrie avahi-daemon[724]: New relevant interface lo.IPv4 for mDNS. Jun 29 00:44:54 valkyrie avahi-daemon[724]: Network interface enumeration completed. Jun 29 00:44:54 valkyrie avahi-daemon[724]: Registering new address record for ::1 on lo.*. Jun 29 00:44:54 valkyrie avahi-daemon[724]: Registering new address record for 127.0.0.1 on lo.IPv4. Jun 29 00:44:55 valkyrie systemd[1]: Finished IPv4 Packet Filtering Framework. Jun 29 00:44:55 valkyrie systemd[1]: Reached target Preparation for Network. Jun 29 00:44:55 valkyrie systemd[1]: Starting A basic static ethernet connection... Jun 29 00:44:55 valkyrie network[779]: Starting network profile 'ethernet-statis-valkyrie'... Jun 29 00:44:55 valkyrie kernel: RTL8211E Gigabit Ethernet r8169-0-500:00: attached PHY driver (mii_bus:phy_addr=r8169-0-500:00, irq=MAC) Jun 29 00:44:55 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 29 00:44:55 valkyrie avahi-daemon[724]: Server startup complete. Host name is valkyrie.local. Local service cookie is 357913226. Jun 29 00:44:56 valkyrie avahi-daemon[724]: Service "dcr4100n" (/services/dcr4100n.service) successfully established. Jun 29 00:44:56 valkyrie avahi-daemon[724]: Service "LaserJet" (/services/LaserJet.service) successfully established. Jun 29 00:44:57 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx Jun 29 00:44:57 valkyrie systemd[1]: Started A basic static ethernet connection. Jun 29 00:44:57 valkyrie systemd[1]: Reached target Network. Jun 29 00:44:57 valkyrie systemd[1]: Reached target Network is Online. Jun 29 00:44:57 valkyrie avahi-daemon[724]: Joining mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 29 00:44:57 valkyrie avahi-daemon[724]: New relevant interface enp5s0.IPv4 for mDNS. Jun 29 00:44:57 valkyrie avahi-daemon[724]: Registering new address record for 192.168.6.14 on enp5s0.IPv4. Jun 29 00:44:57 valkyrie systemd[1]: Started ClamAV virus database updater. Jun 29 00:44:57 valkyrie systemd[1]: Starting IPv4 DHCP server... Jun 29 00:44:57 valkyrie systemd[1]: Starting Dovecot IMAP/POP3 email server... Jun 29 00:44:57 valkyrie systemd[1]: Starting Fail2Ban Service... Jun 29 00:44:57 valkyrie systemd[1]: Started Apache Web Server. Jun 29 00:44:57 valkyrie systemd[1]: Starting MariaDB 11.8.2 database server... Jun 29 00:44:57 valkyrie systemd[1]: Started Internet domain name server. Jun 29 00:44:57 valkyrie systemd[1]: Starting Network UPS Tools - power device driver controller... Jun 29 00:44:57 valkyrie systemd[1]: Starting Postfix Mail Transport Agent... Jun 29 00:44:57 valkyrie systemd[1]: Starting PostgreSQL database server... Jun 29 00:44:58 valkyrie systemd[1]: Starting Samba SMB Daemon... Jun 29 00:44:58 valkyrie systemd[1]: Started Spamassassin daemon. Jun 29 00:44:58 valkyrie systemd[1]: Starting OpenSSH Daemon... Jun 29 00:44:58 valkyrie systemd[1]: Starting Permit User Sessions... Jun 29 00:44:58 valkyrie systemd[1]: Started Fail2Ban Service. Jun 29 00:44:58 valkyrie systemd[1]: Finished Permit User Sessions. Jun 29 00:44:58 valkyrie systemd[1]: Started Command Scheduler. Jun 29 00:44:58 valkyrie systemd[1]: Starting Light Display Manager... Jun 29 00:44:58 valkyrie network[847]: resolvconf: signature mismatch: /etc/resolv.conf Jun 29 00:44:58 valkyrie network[847]: resolvconf: run `resolvconf -u` to update Jun 29 00:44:59 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down Jun 29 00:44:59 valkyrie avahi-daemon[724]: Interface enp5s0.IPv4 no longer relevant for mDNS. Jun 29 00:44:59 valkyrie systemd[1]: netctl@ethernet\x2dstatis\x2dvalkyrie.service: Main process exited, code=exited, status=1/FAILURE Jun 29 00:44:59 valkyrie network[779]: Failed to bring the network up for profile 'ethernet-statis-valkyrie' Jun 29 00:44:59 valkyrie avahi-daemon[724]: Leaving mDNS multicast group on interface enp5s0.IPv4 with address 192.168.6.14. Jun 29 00:44:59 valkyrie systemd[1]: netctl@ethernet\x2dstatis\x2dvalkyrie.service: Failed with result 'exit-code'. Jun 29 00:44:59 valkyrie avahi-daemon[724]: Withdrawing address record for 192.168.6.14 on enp5s0. So it very much looks like either systemd or netctl are now taking the inability to update resolv.conf as a FAILURE of the netctl@ethernet\x2dstatis\x2dvalkyrie.service which shouldn't be happening unless something changed. It has been in the exact same configuration since 2015 and has used the exact same files since November 2022 when the immutable attribute was set on resolv.conf. And again, that unit failing isn't something you could ever possibly overlook on boot. Bad situation for a server. No network, no server. Ironically (and also hiding the problem) when the interface is brought up manually with "ip set enp5s0 up" there must be no check of resolv.conf. Otherwise, if that is what is causing the issue, I would expect bringing the interface up to result in the same error and it to go back down. But it doesn't. After being brought up, the server is running perfectly, just as it did before the Jun 28 update. Hah! we may have just found it. openresolv was updated in the Jun 28 update. That may be the culprit. I'll downgrade reboot and report. -- David C. Rankin, J.D.,P.E.