Dev, This is a strange one. After updates on 11/23, httpd began segfaulting -- repeatedly:
l /var/log/httpd/error_log* -rw-r--r-- 1 root root 24306171 Nov 24 14:48 /var/log/httpd/error_log -rw-r--r-- 1 root root 625 Nov 24 00:00 /var/log/httpd/error_log.1 -rw-r--r-- 1 root root 1742 Nov 21 13:19 /var/log/httpd/error_log.2 -rw-r--r-- 1 root root 504 Nov 10 00:00 /var/log/httpd/error_log.3 -rw-r--r-- 1 root root 1754 Nov 3 00:00 /var/log/httpd/error_log.4
I have a second server that confirms this exact same behavior:
l /var/log/httpd/error_log* -rw-r--r-- 1 root root 25049142 Nov 24 14:48 /var/log/httpd/error_log -rw-r--r-- 1 root root 1137 Nov 24 00:00 /var/log/httpd/error_log.1 -rw-r--r-- 1 root root 1752 Nov 17 00:01 /var/log/httpd/error_log.2 -rw-r--r-- 1 root root 504 Nov 10 00:00 /var/log/httpd/error_log.3 -rw-r--r-- 1 root root 1754 Nov 3 00:01 /var/log/httpd/error_log.4
cat error_log.1 [Thu Nov 21 13:39:09.090480 2019] [suexec:notice] [pid 898] AH01232: suEXEC mechanism enabled (wrapper: /usr/bin/suexec) [Thu Nov 21 13:39:09.146764 2019] [lbmethod_heartbeat:notice] [pid 898] AH02282: No slotmem from mod_heartmonitor [Thu Nov 21 13:39:09.378344 2019] [mpm_prefork:notice] [pid 898] AH00163: Apache/2.4.41 (Unix) OpenSSL/1.1.1d PHP/5.6.40 configured -- resuming normal operations [Thu Nov 21 13:39:09.378406 2019] [core:notice] [pid 898] AH00094: Command
head error_log [Sun Nov 24 00:00:27.104644 2019] [lbmethod_heartbeat:notice] [pid 898] AH02282: No slotmem from mod_heartmonitor [Sun Nov 24 00:00:27.115890 2019] [mpm_prefork:notice] [pid 898] AH00163: Apache/2.4.41 (Unix) OpenSSL/1.1.1d PHP/5.6.40 configured -- resuming normal operations [Sun Nov 24 00:00:27.115901 2019] [core:notice] [pid 898] AH00094: Command
The log file size jumped from 625 bytes to 24306171 bytes (1137->25049142 on the other server) in just a matter of hours. The old log contains, in its entirety: line: '/usr/bin/httpd -D FOREGROUND' [Sun Nov 24 00:00:27.074145 2019] [mpm_prefork:notice] [pid 898] AH00171: Graceful restart requested, doing restart The new log began with: line: '/usr/bin/httpd -D FOREGROUND' [Sun Nov 24 00:00:28.120350 2019] [core:notice] [pid 898] AH00051: child pid 11154 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 00:00:28.120576 2019] [core:notice] [pid 898] AH00051: child pid 11156 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 00:00:28.120728 2019] [core:notice] [pid 898] AH00051: child pid 11158 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 00:00:28.120871 2019] [core:notice] [pid 898] AH00051: child pid 11160 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 00:00:28.121060 2019] [core:notice] [pid 898] AH00051: child pid 11162 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 00:00:30.126762 2019] [core:notice] [pid 898] AH00051: child pid 11253 exit signal Segmentation fault (11), possible coredump in /etc/httpd Updating again today (which is when I caught the problem), and restarting httpd has solved whatever problem there was and httpd is now happily running fine again. The only updates that occurred on the 23rd and 24th are as follows and nothing jumps out as being a problem: [2019-11-23T03:35:07-0600] [PACMAN] starting full system upgrade [2019-11-23T03:36:09-0600] [ALPM] transaction started [2019-11-23T03:36:09-0600] [ALPM] upgraded libusbmuxd (1.0.10+29+g9db5747-1 -> 2.0.0-2) [2019-11-23T03:36:09-0600] [ALPM] upgraded libimobiledevice (1.2.0+87+g92c5462-5 -> 1.2.0+87+g92c5462-6) [2019-11-23T03:36:10-0600] [ALPM] upgraded nodejs (13.1.0-2 -> 13.2.0-1) [2019-11-23T03:36:16-0600] [ALPM] upgraded python2-typing (3.6.6-1 -> 3.7.4.1-1) [2019-11-23T03:36:16-0600] [ALPM] upgraded qpdf (9.0.2-1 -> 9.1.0-1) [2019-11-23T03:36:17-0600] [ALPM] transaction completed [2019-11-23T03:36:17-0600] [ALPM] running '30-systemd-update.hook'... [2019-11-24T14:39:58-0600] [PACMAN] Running 'pacman -Syu' [2019-11-24T14:39:58-0600] [PACMAN] synchronizing package lists [2019-11-24T14:40:00-0600] [PACMAN] starting full system upgrade [2019-11-24T14:40:31-0600] [ALPM] transaction started [2019-11-24T14:40:31-0600] [ALPM] upgraded blas (3.8.0-2 -> 3.9.0-2) [2019-11-24T14:40:31-0600] [ALPM] upgraded cblas (3.8.0-2 -> 3.9.0-2) [2019-11-24T14:40:33-0600] [ALPM] upgraded mesa (19.2.4-1 -> 19.2.6-1) [2019-11-24T14:40:33-0600] [ALPM] upgraded xorg-server-common (1.20.5-4 -> 1.20.6-1) [2019-11-24T14:40:33-0600] [ALPM] upgraded libutil-linux (2.34-6 -> 2.34-8) [2019-11-24T14:40:33-0600] [ALPM] upgraded util-linux (2.34-6 -> 2.34-8) [2019-11-24T14:40:33-0600] [ALPM] upgraded xorg-server (1.20.5-4 -> 1.20.6-1) [2019-11-24T14:40:39-0600] [ALPM] upgraded nvidia-utils (440.31-1 -> 440.36-1) [2019-11-24T14:40:39-0600] [ALPM-SCRIPTLET] If you run into trouble with CUDA not being available, run nvidia-modprobe first. [2019-11-24T14:40:39-0600] [ALPM] upgraded eglexternalplatform (1.0+3+g7c8f8e2-1 -> 1.1-1) [2019-11-24T14:40:39-0600] [ALPM] upgraded egl-wayland (1.1.4-1 -> 1.1.4-2) [2019-11-24T14:40:39-0600] [ALPM] upgraded faad2 (2.9.1-1 -> 2.9.1-2) [2019-11-24T14:40:39-0600] [ALPM] upgraded graphviz (2.42.2-6 -> 2.42.3-1) [2019-11-24T14:40:40-0600] [ALPM] upgraded lapack (3.8.0-2 -> 3.9.0-2) [2019-11-24T14:40:41-0600] [ALPM] upgraded libmagick6 (6.9.10.71-1 -> 6.9.10.74-1) [2019-11-24T14:40:41-0600] [ALPM] upgraded libwpe (1.4.0-1 -> 1.4.0.1-1) [2019-11-24T14:40:41-0600] [ALPM] upgraded man-pages (5.03-2 -> 5.04-1) [2019-11-24T14:40:41-0600] [ALPM] upgraded perl-libwww (6.41-1 -> 6.42-1) [2019-11-24T14:40:41-0600] [ALPM] upgraded python-pyasn1 (0.4.7-3 -> 0.4.8-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded python-setuptools (1:41.4.0-1 -> 1:41.6.0-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded python2-pyasn1 (0.4.7-3 -> 0.4.8-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded python2-setuptools (1:41.4.0-1 -> 1:41.6.0-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded qhull (2019.1-1 -> 2019.1-2) [2019-11-24T14:40:42-0600] [ALPM] upgraded tcl (8.6.9-2 -> 8.6.10-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded tk (8.6.9.1-2 -> 8.6.10-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded vte-common (0.58.2-1 -> 0.58.3-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded vte3 (0.58.2-1 -> 0.58.3-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded wpebackend-fdo (1.4.0-1 -> 1.4.0-2) [2019-11-24T14:40:42-0600] [ALPM] upgraded xorg-server-devel (1.20.5-4 -> 1.20.6-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded yelp-xsl (3.34.0-1 -> 3.34.2-1) [2019-11-24T14:40:43-0600] [ALPM] transaction completed [2019-11-24T14:40:43-0600] [ALPM] running '20-systemd-sysusers.hook'... [2019-11-24T14:40:43-0600] [ALPM] running '30-systemd-daemon-reload.hook'... [2019-11-24T14:40:43-0600] [ALPM] running '30-systemd-udev-reload.hook'... [2019-11-24T14:40:43-0600] [ALPM] running '30-systemd-update.hook'... [2019-11-24T14:40:43-0600] [ALPM] running 'detect-old-perl-modules.hook'... Nothing really looks like it should directly implicate httpd. But after the first graceful restart of the server after the 5 packages were updated on 11/23 cause httpd to go haywire. Just thought I would pass it along. -- David C. Rankin, J.D.,P.E.