All,
I have my Arch server that has been running for a decade, updates
weekly, never misses a beat. It has been so long since I've had an issue
with it, I thought Arch had finally reached Linux rolling release nirvana.
Then, for reasons I can't explain, the update tonight (an hour ago),
failed to start dhcpd4. I can't ping the box, I have to log in locally
and I find the network interface enp5s0 'down' - that's not happened
since Arch was installed.
So I bring the interface up:
# ip link set enp5s0 up
Presto, network is up, and well. So I try starting dhcpd4 and it
fails "cannot create /run/dhcpd4/dhcpd.pid". There is no /run/dhcpd4
directory. So I created it and manually started dhcpd4 without -q and
all starts fine and it is now happily handing out addresses and updating
the bind DNS zone for about 60 clients.
What happened? I did not install or uninstall anything new. I simply
did a "pacman -Syu" to update the system. The following packages were
upgraded:
[2025-06-28T01:54:00-0500] [PACMAN] Running 'pacman -Syu'
[2025-06-28T01:54:00-0500] [PACMAN] synchronizing package lists
[2025-06-28T01:54:01-0500] [PACMAN] starting full system upgrade
[2025-06-28T01:54:21-0500] [ALPM] running '60-mkinitcpio-remove.hook'...
[2025-06-28T01:54:21-0500] [ALPM] running '71-dkms-remove.hook'...
[2025-06-28T01:54:23-0500] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod
nvidia/390.157 -k 6.15.3-arch1-1
[2025-06-28T01:54:24-0500] [ALPM-SCRIPTLET] ==> depmod 6.15.3-arch1-1
[2025-06-28T01:54:31-0500] [ALPM] transaction started
[2025-06-28T01:54:31-0500] [ALPM] upgraded broot (1.46.5-3 -> 1.47.0-1)
[2025-06-28T01:54:31-0500] [ALPM] upgraded util-linux-libs (2.41-4 ->
2.41.1-1)
[2025-06-28T01:54:31-0500] [ALPM] upgraded systemd-libs (257.6-1 -> 257.7-1)
[2025-06-28T01:54:31-0500] [ALPM] upgraded cryptsetup (2.7.5-2 -> 2.8.0-1)
[2025-06-28T01:54:32-0500] [ALPM] upgraded docker (1:28.2.2-1 -> 1:28.3.0-1)
[2025-06-28T01:54:32-0500] [ALPM] upgraded libopenmpt (0.8.0-1 -> 0.8.1-1)
[2025-06-28T01:54:32-0500] [ALPM] upgraded libjpeg-turbo (3.1.0-1 ->
3.1.1-1)
[2025-06-28T01:54:32-0500] [ALPM] upgraded util-linux (2.41-4 -> 2.41.1-1)
[2025-06-28T01:54:32-0500] [ALPM] upgraded systemd (257.6-1 -> 257.7-1)
[2025-06-28T01:54:36-0500] [ALPM] upgraded firefox (140.0-1 -> 140.0.1-1)
[2025-06-28T01:54:36-0500] [ALPM] upgraded freetds (1.5.3-1 -> 1.5.4-1)
[2025-06-28T01:54:36-0500] [ALPM] upgraded gstreamer (1.26.2-2 -> 1.26.3-1)
[2025-06-28T01:54:36-0500] [ALPM] upgraded gst-plugins-base-libs
(1.26.2-2 -> 1.26.3-1)
[2025-06-28T01:54:36-0500] [ALPM] upgraded gst-libav (1.26.2-2 -> 1.26.3-1)
[2025-06-28T01:54:36-0500] [ALPM] upgraded gst-plugins-bad-libs
(1.26.2-2 -> 1.26.3-1)
[2025-06-28T01:54:36-0500] [ALPM] upgraded libpipewire (1:1.4.5-1 ->
1:1.4.6-1)
[2025-06-28T01:54:36-0500] [ALPM] upgraded gst-plugins-bad (1.26.2-2 ->
1.26.3-1)
[2025-06-28T01:54:36-0500] [ALPM] upgraded gst-plugins-base (1.26.2-2 ->
1.26.3-1)
[2025-06-28T01:54:36-0500] [ALPM] upgraded gst-plugins-good (1.26.2-2 ->
1.26.3-1)
[2025-06-28T01:54:36-0500] [ALPM] upgraded gum (0.16.1-1 -> 0.16.2-1)
[2025-06-28T01:54:37-0500] [ALPM] upgraded intel-media-driver (25.2.3-1
-> 25.2.5-1)
[2025-06-28T01:54:37-0500] [ALPM] upgraded libmpdclient (2.22-1 -> 2.23-1)
[2025-06-28T01:54:37-0500] [ALPM] upgraded libunrar (1:7.1.7-1 -> 1:7.1.8-1)
[2025-06-28T01:54:37-0500] [ALPM] upgraded libvoikko (4.3.2-4 -> 4.3.3-1)
[2025-06-28T01:54:38-0500] [ALPM] upgraded linux (6.15.3.arch1-1 ->
6.15.4.arch2-1)
[2025-06-28T01:54:45-0500] [ALPM] upgraded linux-docs (6.15.3.arch1-1 ->
6.15.4.arch2-1)
[2025-06-28T01:54:45-0500] [ALPM] upgraded linux-firmware-whence
(20250613.12fe085f-9 -> 20250627-1)
[2025-06-28T01:54:46-0500] [ALPM] upgraded linux-firmware-amdgpu
(20250613.12fe085f-9 -> 20250627-1)
[2025-06-28T01:54:46-0500] [ALPM] upgraded linux-firmware-atheros
(20250613.12fe085f-9 -> 20250627-1)
[2025-06-28T01:54:47-0500] [ALPM] upgraded linux-firmware-broadcom
(20250613.12fe085f-9 -> 20250627-1)
[2025-06-28T01:54:47-0500] [ALPM] upgraded linux-firmware-cirrus
(20250613.12fe085f-9 -> 20250627-1)
[2025-06-28T01:54:47-0500] [ALPM] upgraded linux-firmware-intel
(20250613.12fe085f-9 -> 20250627-1)
[2025-06-28T01:54:47-0500] [ALPM] upgraded linux-firmware-mediatek
(20250613.12fe085f-9 -> 20250627-1)
[2025-06-28T01:54:49-0500] [ALPM] upgraded linux-firmware-nvidia
(20250613.12fe085f-9 -> 20250627-1)
[2025-06-28T01:54:49-0500] [ALPM] upgraded linux-firmware-other
(20250613.12fe085f-9 -> 20250627-1)
[2025-06-28T01:54:49-0500] [ALPM] upgraded linux-firmware-radeon
(20250613.12fe085f-9 -> 20250627-1)
[2025-06-28T01:54:49-0500] [ALPM] upgraded linux-firmware-realtek
(20250613.12fe085f-9 -> 20250627-1)
[2025-06-28T01:54:49-0500] [ALPM] upgraded linux-firmware
(20250613.12fe085f-9 -> 20250627-1)
[2025-06-28T01:54:53-0500] [ALPM] upgraded linux-headers (6.15.3.arch1-1
-> 6.15.4.arch2-1)
[2025-06-28T01:54:53-0500] [ALPM] upgraded openresolv (3.16.5-1 -> 3.17.0-1)
[2025-06-28T01:54:53-0500] [ALPM] upgraded pipewire (1:1.4.5-1 -> 1:1.4.6-1)
[2025-06-28T01:54:53-0500] [ALPM] upgraded pkgconf (2.5.0-1 -> 2.5.1-1)
[2025-06-28T01:54:53-0500] [ALPM] upgraded python-pygments (2.19.1-1 ->
2.19.2-1)
[2025-06-28T01:54:54-0500] [ALPM] upgraded rust (1:1.87.0-2 -> 1:1.88.0-1)
[2025-06-28T01:54:54-0500] [ALPM] upgraded systemd-sysvcompat (257.6-1
-> 257.7-1)
[2025-06-28T01:54:54-0500] [ALPM] upgraded thin-provisioning-tools
(1.1.0-1 -> 1.2.0-1)
[2025-06-28T01:54:54-0500] [ALPM] upgraded unrar (1:7.1.7-1 -> 1:7.1.8-1)
[2025-06-28T01:54:55-0500] [ALPM] upgraded yt-dlp (2025.06.09-1 ->
2025.06.25-1)
I have got to find out why this failed to set the network device up
and why dhcpd4 couldn't create /run/dhcpd4 for startup. Most of the time
I admin this box remotely. Having it reboot and not reliably come back
up isn't an option.
Is there some issue with the new firmware and network devices? The
network device uses the r8169 driver is that is an issue, it used the
r1868 driver in the past. I haven't changed a thing related to it.
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 06)
Subsystem: Gigabyte Technology Co., Ltd Onboard Ethernet
Flags: bus master, fast devsel, latency 0, IRQ 16, NUMA node 0,
IOMMU group 13
I/O ports at b000 [size=256]
Memory at fe100000 (64-bit, non-prefetchable) [size=4K]
Memory at dc100000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, IntMsgNum 1
Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
Capabilities: [d0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
Kernel driver in use: r8169
Kernel modules: r8169
What else to check to track down what caused the network not to come
up enabled and why couldn't dhcpd4 create its pid file even after the
link was brought up? This box is updated and rebooted with each new
kernel and has always been bullet-proof.
If it helps, the update on June 25 was minimal as well, updating:
[2025-06-25T03:44:59-0500] [PACMAN] Running 'pacman -Syu linux-firmware'
[2025-06-25T03:44:59-0500] [PACMAN] synchronizing package lists
[2025-06-25T03:45:00-0500] [PACMAN] starting full system upgrade
[2025-06-25T03:45:38-0500] [ALPM] running '60-mkinitcpio-remove.hook'...
[2025-06-25T03:45:39-0500] [ALPM] running '71-dkms-remove.hook'...
[2025-06-25T03:45:40-0500] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod
nvidia/390.157 -k 6.15.2-arch1-1
[2025-06-25T03:45:41-0500] [ALPM-SCRIPTLET] ==> depmod 6.15.2-arch1-1
[2025-06-25T03:45:47-0500] [ALPM] transaction started
[2025-06-25T03:45:47-0500] [ALPM] upgraded linux-firmware-whence
(20250508.788aadc8-2 -> 20250613.12fe085f-9)
[2025-06-25T03:45:47-0500] [ALPM] installed linux-firmware-amdgpu
(20250613.12fe085f-9)
[2025-06-25T03:45:47-0500] [ALPM] installed linux-firmware-atheros
(20250613.12fe085f-9)
[2025-06-25T03:45:47-0500] [ALPM] installed linux-firmware-broadcom
(20250613.12fe085f-9)
[2025-06-25T03:45:47-0500] [ALPM] installed linux-firmware-cirrus
(20250613.12fe085f-9)
[2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware-intel
(20250613.12fe085f-9)
[2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware-mediatek
(20250613.12fe085f-9)
[2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware-nvidia
(20250613.12fe085f-9)
[2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware-other
(20250613.12fe085f-9)
[2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware-radeon
(20250613.12fe085f-9)
[2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware-realtek
(20250613.12fe085f-9)
[2025-06-25T03:45:48-0500] [ALPM] installed linux-firmware
(20250613.12fe085f-9)
[2025-06-25T03:45:48-0500] [ALPM] upgraded btrfs-progs (6.14-1 -> 6.15-1)
[2025-06-25T03:45:48-0500] [ALPM] upgraded ca-certificates-mozilla
(3.112-1 -> 3.113-1)
[2025-06-25T03:45:49-0500] [ALPM] upgraded docker-compose (2.37.1-1 ->
2.37.3-1)
[2025-06-25T03:45:49-0500] [ALPM] upgraded libxml2 (2.14.4-1 -> 2.14.4-2)
[2025-06-25T03:45:49-0500] [ALPM] upgraded dvisvgm (3.5-1 -> 3.5-2)
[2025-06-25T03:45:49-0500] [ALPM] upgraded enchant (2.8.9-1 -> 2.8.10-1)
[2025-06-25T03:45:49-0500] [ALPM] upgraded nss (3.112-1 -> 3.113-1)
[2025-06-25T03:45:49-0500] [ALPM] upgraded libssh (0.11.1-1 -> 0.11.2-1)
[2025-06-25T03:45:49-0500] [ALPM] upgraded xorg-server-common (21.1.18-1
-> 21.1.18-2)
[2025-06-25T03:45:49-0500] [ALPM] upgraded xorg-server (21.1.18-1 ->
21.1.18-2)
[2025-06-25T03:45:49-0500] [ALPM] upgraded python (3.13.3-1 -> 3.13.5-1)
[2025-06-25T03:45:50-0500] [ALPM] upgraded firefox (139.0.4-1 -> 140.0-1)
[2025-06-25T03:45:51-0500] [ALPM] upgraded giac (2.0.0.4-1 -> 2.0.0.6-1)
[2025-06-25T03:45:51-0500] [ALPM] upgraded gnupg (2.4.7-3 -> 2.4.8-1)
[2025-06-25T03:45:52-0500] [ALPM] upgraded iniparser (4.2.6-1 -> 4.2.6-2)
[2025-06-25T03:45:52-0500] [ALPM] upgraded liburcu (0.15.1-1 -> 0.15.3-1)
[2025-06-25T03:45:53-0500] [ALPM] upgraded linux (6.15.2.arch1-1 ->
6.15.3.arch1-1)
[2025-06-25T03:46:01-0500] [ALPM] upgraded linux-docs (6.15.2.arch1-1 ->
6.15.3.arch1-1)
[2025-06-25T03:46:05-0500] [ALPM] upgraded linux-headers (6.15.2.arch1-1
-> 6.15.3.arch1-1)
[2025-06-25T03:46:05-0500] [ALPM] upgraded parallel (20250522-1 ->
20250622-1)
[2025-06-25T03:46:05-0500] [ALPM] upgraded pciutils (3.13.0-2 -> 3.14.0-1)
[2025-06-25T03:46:05-0500] [ALPM] upgraded perl-image-exiftool (13.30-1
-> 13.30-2)
[2025-06-25T03:46:05-0500] [ALPM] upgraded python-pypdf (5.6.0-1 -> 5.6.1-1)
[2025-06-25T03:46:05-0500] [ALPM] upgraded python-scipy (1.15.3-1 ->
1.16.0-1)
[2025-06-25T03:46:06-0500] [ALPM] upgraded qt6-base (6.9.1-1 -> 6.9.1-2)
[2025-06-25T03:46:06-0500] [ALPM] upgraded seabios (1.16.3-1 -> 1.17.0-1)
[2025-06-25T03:46:06-0500] [ALPM] upgraded sundials (7.3.0-1 -> 7.4.0-1)
[2025-06-25T03:46:06-0500] [ALPM] upgraded virtiofsd (1.13.1-1 -> 1.13.2-1)
[2025-06-25T03:46:06-0500] [ALPM] upgraded xorg-server-devel (21.1.18-1
-> 21.1.18-2)
[2025-06-25T03:46:07-0500] [ALPM] transaction completed
Absolutely no issue after that update. There have been no config
changes to the server. Any idea what changed in the kernel, firmware or
otherwise that may lead to these issues now after updates today (June 28)?
--
David C. Rankin, J.D.,P.E.