All, I have my Arch server that has been running for a decade, updates weekly, never misses a beat. It has been so long since I've had an issue with it, I thought Arch had finally reached Linux rolling release nirvana. Then, for reasons I can't explain, the update tonight (an hour ago), failed to start dhcpd4. I can't ping the box, I have to log in locally and I find the network interface enp5s0 'down' - that's not happened since Arch was installed. So I bring the interface up: # ip link set enp5s0 up Presto, network is up, and well. So I try starting dhcpd4 and it fails "cannot create /run/dhcpd4/dhcpd.pid". There is no /run/dhcpd4 directory. So I created it and manually started dhcpd4 without -q and all starts fine and it is now happily handing out addresses and updating the bind DNS zone for about 60 clients. What happened? I did not install or uninstall anything new. I simply did a "pacman -Syu" to update the system. The following packages were upgraded: [2025-06-28T01:54:00-0500] [PACMAN] Running 'pacman -Syu' [2025-06-28T01:54:00-0500] [PACMAN] synchronizing package lists [2025-06-28T01:54:01-0500] [PACMAN] starting full system upgrade [2025-06-28T01:54:21-0500] [ALPM] running '60-mkinitcpio-remove.hook'... [2025-06-28T01:54:21-0500] [ALPM] running '71-dkms-remove.hook'... [2025-06-28T01:54:23-0500] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod nvidia/390.157 -k 6.15.3-arch1-1 [2025-06-28T01:54:24-0500] [ALPM-SCRIPTLET] ==> depmod 6.15.3-arch1-1 [2025-06-28T01:54:31-0500] [ALPM] transaction started [2025-06-28T01:54:31-0500] [ALPM] upgraded broot (1.46.5-3 -> 1.47.0-1) [2025-06-28T01:54:31-0500] [ALPM] upgraded util-linux-libs (2.41-4 -> 2.41.1-1) [2025-06-28T01:54:31-0500] [ALPM] upgraded systemd-libs (257.6-1 -> 257.7-1) [2025-06-28T01:54:31-0500] [ALPM] upgraded cryptsetup (2.7.5-2 -> 2.8.0-1) [2025-06-28T01:54:32-0500] [ALPM] upgraded docker (1:28.2.2-1 -> 1:28.3.0-1) [2025-06-28T01:54:32-0500] [ALPM] upgraded libopenmpt (0.8.0-1 -> 0.8.1-1) [2025-06-28T01:54:32-0500] [ALPM] upgraded libjpeg-turbo (3.1.0-1 -> 3.1.1-1) [2025-06-28T01:54:32-0500] [ALPM] upgraded util-linux (2.41-4 -> 2.41.1-1) [2025-06-28T01:54:32-0500] [ALPM] upgraded systemd (257.6-1 -> 257.7-1) [2025-06-28T01:54:36-0500] [ALPM] upgraded firefox (140.0-1 -> 140.0.1-1) [2025-06-28T01:54:36-0500] [ALPM] upgraded freetds (1.5.3-1 -> 1.5.4-1) [2025-06-28T01:54:36-0500] [ALPM] upgraded gstreamer (1.26.2-2 -> 1.26.3-1) [2025-06-28T01:54:36-0500] [ALPM] upgraded gst-plugins-base-libs (1.26.2-2 -> 1.26.3-1) [2025-06-28T01:54:36-0500] [ALPM] upgraded gst-libav (1.26.2-2 -> 1.26.3-1) [2025-06-28T01:54:36-0500] [ALPM] upgraded gst-plugins-bad-libs (1.26.2-2 -> 1.26.3-1) [2025-06-28T01:54:36-0500] [ALPM] upgraded libpipewire (1:1.4.5-1 -> 1:1.4.6-1) [2025-06-28T01:54:36-0500] [ALPM] upgraded gst-plugins-bad (1.26.2-2 -> 1.26.3-1) [2025-06-28T01:54:36-0500] [ALPM] upgraded gst-plugins-base (1.26.2-2 -> 1.26.3-1) [2025-06-28T01:54:36-0500] [ALPM] upgraded gst-plugins-good (1.26.2-2 -> 1.26.3-1) [2025-06-28T01:54:36-0500] [ALPM] upgraded gum (0.16.1-1 -> 0.16.2-1) [2025-06-28T01:54:37-0500] [ALPM] upgraded intel-media-driver (25.2.3-1 -> 25.2.5-1) [2025-06-28T01:54:37-0500] [ALPM] upgraded libmpdclient (2.22-1 -> 2.23-1) [2025-06-28T01:54:37-0500] [ALPM] upgraded libunrar (1:7.1.7-1 -> 1:7.1.8-1) [2025-06-28T01:54:37-0500] [ALPM] upgraded libvoikko (4.3.2-4 -> 4.3.3-1) [2025-06-28T01:54:38-0500] [ALPM] upgraded linux (6.15.3.arch1-1 -> 6.15.4.arch2-1) [2025-06-28T01:54:45-0500] [ALPM] upgraded linux-docs (6.15.3.arch1-1 -> 6.15.4.arch2-1) [2025-06-28T01:54:45-0500] [ALPM] upgraded linux-firmware-whence (20250613.12fe085f-9 -> 20250627-1) [2025-06-28T01:54:46-0500] [ALPM] upgraded linux-firmware-amdgpu (20250613.12fe085f-9 -> 20250627-1) [2025-06-28T01:54:46-0500] [ALPM] upgraded linux-firmware-atheros (20250613.12fe085f-9 -> 20250627-1) [2025-06-28T01:54:47-0500] [ALPM] upgraded linux-firmware-broadcom (20250613.12fe085f-9 -> 20250627-1) [2025-06-28T01:54:47-0500] [ALPM] upgraded linux-firmware-cirrus (20250613.12fe085f-9 -> 20250627-1) [2025-06-28T01:54:47-0500] [ALPM] upgraded linux-firmware-intel (20250613.12fe085f-9 -> 20250627-1) [2025-06-28T01:54:47-0500] [ALPM] upgraded linux-firmware-mediatek (20250613.12fe085f-9 -> 20250627-1) [2025-06-28T01:54:49-0500] [ALPM] upgraded linux-firmware-nvidia (20250613.12fe085f-9 -> 20250627-1) [2025-06-28T01:54:49-0500] [ALPM] upgraded linux-firmware-other (20250613.12fe085f-9 -> 20250627-1) [2025-06-28T01:54:49-0500] [ALPM] upgraded linux-firmware-radeon (20250613.12fe085f-9 -> 20250627-1) [2025-06-28T01:54:49-0500] [ALPM] upgraded linux-firmware-realtek (20250613.12fe085f-9 -> 20250627-1) [2025-06-28T01:54:49-0500] [ALPM] upgraded linux-firmware (20250613.12fe085f-9 -> 20250627-1) [2025-06-28T01:54:53-0500] [ALPM] upgraded linux-headers (6.15.3.arch1-1 -> 6.15.4.arch2-1) [2025-06-28T01:54:53-0500] [ALPM] upgraded openresolv (3.16.5-1 -> 3.17.0-1) [2025-06-28T01:54:53-0500] [ALPM] upgraded pipewire (1:1.4.5-1 -> 1:1.4.6-1) [2025-06-28T01:54:53-0500] [ALPM] upgraded pkgconf (2.5.0-1 -> 2.5.1-1) [2025-06-28T01:54:53-0500] [ALPM] upgraded python-pygments (2.19.1-1 -> 2.19.2-1) [2025-06-28T01:54:54-0500] [ALPM] upgraded rust (1:1.87.0-2 -> 1:1.88.0-1) [2025-06-28T01:54:54-0500] [ALPM] upgraded systemd-sysvcompat (257.6-1 -> 257.7-1) [2025-06-28T01:54:54-0500] [ALPM] upgraded thin-provisioning-tools (1.1.0-1 -> 1.2.0-1) [2025-06-28T01:54:54-0500] [ALPM] upgraded unrar (1:7.1.7-1 -> 1:7.1.8-1) [2025-06-28T01:54:55-0500] [ALPM] upgraded yt-dlp (2025.06.09-1 -> 2025.06.25-1) I have got to find out why this failed to set the network device up and why dhcpd4 couldn't create /run/dhcpd4 for startup. Most of the time I admin this box remotely. Having it reboot and not reliably come back up isn't an option. Is there some issue with the new firmware and network devices? The network device uses the r8169 driver is that is an issue, it used the r1868 driver in the past. I haven't changed a thing related to it. 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 06) Subsystem: Gigabyte Technology Co., Ltd Onboard Ethernet Flags: bus master, fast devsel, latency 0, IRQ 16, NUMA node 0, IOMMU group 13 I/O ports at b000 [size=256] Memory at fe100000 (64-bit, non-prefetchable) [size=4K] Memory at dc100000 (64-bit, prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [70] Express Endpoint, IntMsgNum 1 Capabilities: [b0] MSI-X: Enable+ Count=4 Masked- Capabilities: [d0] Vital Product Data Capabilities: [100] Advanced Error Reporting Capabilities: [140] Virtual Channel Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00 Kernel driver in use: r8169 Kernel modules: r8169 What else to check to track down what caused the network not to come up enabled and why couldn't dhcpd4 create its pid file even after the link was brought up? This box is updated and rebooted with each new kernel and has always been bullet-proof. If it helps, the update on June 25 was minimal as well, updating: [2025-06-25T03:44:59-0500] [PACMAN] Running 'pacman -Syu linux-firmware' [2025-06-25T03:44:59-0500] [PACMAN] synchronizing package lists [2025-06-25T03:45:00-0500] [PACMAN] starting full system upgrade [2025-06-25T03:45:38-0500] [ALPM] running '60-mkinitcpio-remove.hook'... [2025-06-25T03:45:39-0500] [ALPM] running '71-dkms-remove.hook'... [2025-06-25T03:45:40-0500] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod nvidia/390.157 -k 6.15.2-arch1-1 [2025-06-25T03:45:41-0500] [ALPM-SCRIPTLET] ==> depmod 6.15.2-arch1-1 [2025-06-25T03:45:47-0500] [ALPM] transaction started [2025-06-25T03:45:47-0500] [ALPM] upgraded linux-firmware-whence (20250508.788aadc8-2 -> 20250613.12fe085f-9) [2025-06-25T03:45:47-0500] [ALPM] installed linux-firmware-amdgpu (20250613.12fe085f-9) [2025-06-25T03:45:47-0500] [ALPM] installed linux-firmware-atheros (20250613.12fe085f-9) [2025-06-25T03:45:47-0500] [ALPM] installed linux-firmware-broadcom (20250613.12fe085f-9) [2025-06-25T03:45:47-0500] [ALPM] installed linux-firmware-cirrus (20250613.12fe085f-9) [2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware-intel (20250613.12fe085f-9) [2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware-mediatek (20250613.12fe085f-9) [2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware-nvidia (20250613.12fe085f-9) [2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware-other (20250613.12fe085f-9) [2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware-radeon (20250613.12fe085f-9) [2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware-realtek (20250613.12fe085f-9) [2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware (20250613.12fe085f-9) [2025-06-25T03:45:48-0500] [ALPM] upgraded btrfs-progs (6.14-1 -> 6.15-1) [2025-06-25T03:45:48-0500] [ALPM] upgraded ca-certificates-mozilla (3.112-1 -> 3.113-1) [2025-06-25T03:45:49-0500] [ALPM] upgraded docker-compose (2.37.1-1 -> 2.37.3-1) [2025-06-25T03:45:49-0500] [ALPM] upgraded libxml2 (2.14.4-1 -> 2.14.4-2) [2025-06-25T03:45:49-0500] [ALPM] upgraded dvisvgm (3.5-1 -> 3.5-2) [2025-06-25T03:45:49-0500] [ALPM] upgraded enchant (2.8.9-1 -> 2.8.10-1) [2025-06-25T03:45:49-0500] [ALPM] upgraded nss (3.112-1 -> 3.113-1) [2025-06-25T03:45:49-0500] [ALPM] upgraded libssh (0.11.1-1 -> 0.11.2-1) [2025-06-25T03:45:49-0500] [ALPM] upgraded xorg-server-common (21.1.18-1 -> 21.1.18-2) [2025-06-25T03:45:49-0500] [ALPM] upgraded xorg-server (21.1.18-1 -> 21.1.18-2) [2025-06-25T03:45:49-0500] [ALPM] upgraded python (3.13.3-1 -> 3.13.5-1) [2025-06-25T03:45:50-0500] [ALPM] upgraded firefox (139.0.4-1 -> 140.0-1) [2025-06-25T03:45:51-0500] [ALPM] upgraded giac (2.0.0.4-1 -> 2.0.0.6-1) [2025-06-25T03:45:51-0500] [ALPM] upgraded gnupg (2.4.7-3 -> 2.4.8-1) [2025-06-25T03:45:52-0500] [ALPM] upgraded iniparser (4.2.6-1 -> 4.2.6-2) [2025-06-25T03:45:52-0500] [ALPM] upgraded liburcu (0.15.1-1 -> 0.15.3-1) [2025-06-25T03:45:53-0500] [ALPM] upgraded linux (6.15.2.arch1-1 -> 6.15.3.arch1-1) [2025-06-25T03:46:01-0500] [ALPM] upgraded linux-docs (6.15.2.arch1-1 -> 6.15.3.arch1-1) [2025-06-25T03:46:05-0500] [ALPM] upgraded linux-headers (6.15.2.arch1-1 -> 6.15.3.arch1-1) [2025-06-25T03:46:05-0500] [ALPM] upgraded parallel (20250522-1 -> 20250622-1) [2025-06-25T03:46:05-0500] [ALPM] upgraded pciutils (3.13.0-2 -> 3.14.0-1) [2025-06-25T03:46:05-0500] [ALPM] upgraded perl-image-exiftool (13.30-1 -> 13.30-2) [2025-06-25T03:46:05-0500] [ALPM] upgraded python-pypdf (5.6.0-1 -> 5.6.1-1) [2025-06-25T03:46:05-0500] [ALPM] upgraded python-scipy (1.15.3-1 -> 1.16.0-1) [2025-06-25T03:46:06-0500] [ALPM] upgraded qt6-base (6.9.1-1 -> 6.9.1-2) [2025-06-25T03:46:06-0500] [ALPM] upgraded seabios (1.16.3-1 -> 1.17.0-1) [2025-06-25T03:46:06-0500] [ALPM] upgraded sundials (7.3.0-1 -> 7.4.0-1) [2025-06-25T03:46:06-0500] [ALPM] upgraded virtiofsd (1.13.1-1 -> 1.13.2-1) [2025-06-25T03:46:06-0500] [ALPM] upgraded xorg-server-devel (21.1.18-1 -> 21.1.18-2) [2025-06-25T03:46:07-0500] [ALPM] transaction completed Absolutely no issue after that update. There have been no config changes to the server. Any idea what changed in the kernel, firmware or otherwise that may lead to these issues now after updates today (June 28)? -- David C. Rankin, J.D.,P.E.