[arch-general] 11/23 update caused httpd to coredump?

Sun Nov 24 21:57:50 UTC 2019

Dev,

  This is a strange one. After updates on 11/23, httpd began segfaulting --
repeatedly:

> l /var/log/httpd/error_log*
-rw-r--r-- 1 root root 24306171 Nov 24 14:48 /var/log/httpd/error_log
-rw-r--r-- 1 root root      625 Nov 24 00:00 /var/log/httpd/error_log.1
-rw-r--r-- 1 root root     1742 Nov 21 13:19 /var/log/httpd/error_log.2
-rw-r--r-- 1 root root      504 Nov 10 00:00 /var/log/httpd/error_log.3
-rw-r--r-- 1 root root     1754 Nov  3 00:00 /var/log/httpd/error_log.4

I have a second server that confirms this exact same behavior:

> l /var/log/httpd/error_log*
-rw-r--r-- 1 root root 25049142 Nov 24 14:48 /var/log/httpd/error_log
-rw-r--r-- 1 root root     1137 Nov 24 00:00 /var/log/httpd/error_log.1
-rw-r--r-- 1 root root     1752 Nov 17 00:01 /var/log/httpd/error_log.2
-rw-r--r-- 1 root root      504 Nov 10 00:00 /var/log/httpd/error_log.3
-rw-r--r-- 1 root root     1754 Nov  3 00:01 /var/log/httpd/error_log.4

The log file size jumped from 625 bytes to 24306171 bytes (1137->25049142 on
the other server) in just a matter of hours. The old log contains, in its
entirety:

> cat error_log.1
[Thu Nov 21 13:39:09.090480 2019] [suexec:notice] [pid 898] AH01232: suEXEC
mechanism enabled (wrapper: /usr/bin/suexec)
[Thu Nov 21 13:39:09.146764 2019] [lbmethod_heartbeat:notice] [pid 898]
AH02282: No slotmem from mod_heartmonitor
[Thu Nov 21 13:39:09.378344 2019] [mpm_prefork:notice] [pid 898] AH00163:
Apache/2.4.41 (Unix) OpenSSL/1.1.1d PHP/5.6.40 configured -- resuming normal
operations
[Thu Nov 21 13:39:09.378406 2019] [core:notice] [pid 898] AH00094: Command
line: '/usr/bin/httpd -D FOREGROUND'
[Sun Nov 24 00:00:27.074145 2019] [mpm_prefork:notice] [pid 898] AH00171:
Graceful restart requested, doing restart

The new log began with:

> head error_log
[Sun Nov 24 00:00:27.104644 2019] [lbmethod_heartbeat:notice] [pid 898]
AH02282: No slotmem from mod_heartmonitor
[Sun Nov 24 00:00:27.115890 2019] [mpm_prefork:notice] [pid 898] AH00163:
Apache/2.4.41 (Unix) OpenSSL/1.1.1d PHP/5.6.40 configured -- resuming normal
operations
[Sun Nov 24 00:00:27.115901 2019] [core:notice] [pid 898] AH00094: Command
line: '/usr/bin/httpd -D FOREGROUND'
[Sun Nov 24 00:00:28.120350 2019] [core:notice] [pid 898] AH00051: child pid
11154 exit signal Segmentation fault (11), possible coredump in /etc/httpd
[Sun Nov 24 00:00:28.120576 2019] [core:notice] [pid 898] AH00051: child pid
11156 exit signal Segmentation fault (11), possible coredump in /etc/httpd
[Sun Nov 24 00:00:28.120728 2019] [core:notice] [pid 898] AH00051: child pid
11158 exit signal Segmentation fault (11), possible coredump in /etc/httpd
[Sun Nov 24 00:00:28.120871 2019] [core:notice] [pid 898] AH00051: child pid
11160 exit signal Segmentation fault (11), possible coredump in /etc/httpd
[Sun Nov 24 00:00:28.121060 2019] [core:notice] [pid 898] AH00051: child pid
11162 exit signal Segmentation fault (11), possible coredump in /etc/httpd
[Sun Nov 24 00:00:30.126762 2019] [core:notice] [pid 898] AH00051: child pid
11253 exit signal Segmentation fault (11), possible coredump in /etc/httpd

Updating again today (which is when I caught the problem), and restarting
httpd has solved whatever problem there was and httpd is now happily running
fine again. The only updates that occurred on the 23rd and 24th are as follows
and nothing jumps out as being a problem:

[2019-11-23T03:35:07-0600] [PACMAN] starting full system upgrade
[2019-11-23T03:36:09-0600] [ALPM] transaction started
[2019-11-23T03:36:09-0600] [ALPM] upgraded libusbmuxd (1.0.10+29+g9db5747-1 ->
2.0.0-2)
[2019-11-23T03:36:09-0600] [ALPM] upgraded libimobiledevice
(1.2.0+87+g92c5462-5 -> 1.2.0+87+g92c5462-6)
[2019-11-23T03:36:10-0600] [ALPM] upgraded nodejs (13.1.0-2 -> 13.2.0-1)
[2019-11-23T03:36:16-0600] [ALPM] upgraded python2-typing (3.6.6-1 -> 3.7.4.1-1)
[2019-11-23T03:36:16-0600] [ALPM] upgraded qpdf (9.0.2-1 -> 9.1.0-1)
[2019-11-23T03:36:17-0600] [ALPM] transaction completed
[2019-11-23T03:36:17-0600] [ALPM] running '30-systemd-update.hook'...
[2019-11-24T14:39:58-0600] [PACMAN] Running 'pacman -Syu'
[2019-11-24T14:39:58-0600] [PACMAN] synchronizing package lists
[2019-11-24T14:40:00-0600] [PACMAN] starting full system upgrade
[2019-11-24T14:40:31-0600] [ALPM] transaction started
[2019-11-24T14:40:31-0600] [ALPM] upgraded blas (3.8.0-2 -> 3.9.0-2)
[2019-11-24T14:40:31-0600] [ALPM] upgraded cblas (3.8.0-2 -> 3.9.0-2)
[2019-11-24T14:40:33-0600] [ALPM] upgraded mesa (19.2.4-1 -> 19.2.6-1)
[2019-11-24T14:40:33-0600] [ALPM] upgraded xorg-server-common (1.20.5-4 ->
1.20.6-1)
[2019-11-24T14:40:33-0600] [ALPM] upgraded libutil-linux (2.34-6 -> 2.34-8)
[2019-11-24T14:40:33-0600] [ALPM] upgraded util-linux (2.34-6 -> 2.34-8)
[2019-11-24T14:40:33-0600] [ALPM] upgraded xorg-server (1.20.5-4 -> 1.20.6-1)
[2019-11-24T14:40:39-0600] [ALPM] upgraded nvidia-utils (440.31-1 -> 440.36-1)
[2019-11-24T14:40:39-0600] [ALPM-SCRIPTLET] If you run into trouble with CUDA
not being available, run nvidia-modprobe first.
[2019-11-24T14:40:39-0600] [ALPM] upgraded eglexternalplatform
(1.0+3+g7c8f8e2-1 -> 1.1-1)
[2019-11-24T14:40:39-0600] [ALPM] upgraded egl-wayland (1.1.4-1 -> 1.1.4-2)
[2019-11-24T14:40:39-0600] [ALPM] upgraded faad2 (2.9.1-1 -> 2.9.1-2)
[2019-11-24T14:40:39-0600] [ALPM] upgraded graphviz (2.42.2-6 -> 2.42.3-1)
[2019-11-24T14:40:40-0600] [ALPM] upgraded lapack (3.8.0-2 -> 3.9.0-2)
[2019-11-24T14:40:41-0600] [ALPM] upgraded libmagick6 (6.9.10.71-1 -> 6.9.10.74-1)
[2019-11-24T14:40:41-0600] [ALPM] upgraded libwpe (1.4.0-1 -> 1.4.0.1-1)
[2019-11-24T14:40:41-0600] [ALPM] upgraded man-pages (5.03-2 -> 5.04-1)
[2019-11-24T14:40:41-0600] [ALPM] upgraded perl-libwww (6.41-1 -> 6.42-1)
[2019-11-24T14:40:41-0600] [ALPM] upgraded python-pyasn1 (0.4.7-3 -> 0.4.8-1)
[2019-11-24T14:40:42-0600] [ALPM] upgraded python-setuptools (1:41.4.0-1 ->
1:41.6.0-1)
[2019-11-24T14:40:42-0600] [ALPM] upgraded python2-pyasn1 (0.4.7-3 -> 0.4.8-1)
[2019-11-24T14:40:42-0600] [ALPM] upgraded python2-setuptools (1:41.4.0-1 ->
1:41.6.0-1)
[2019-11-24T14:40:42-0600] [ALPM] upgraded qhull (2019.1-1 -> 2019.1-2)
[2019-11-24T14:40:42-0600] [ALPM] upgraded tcl (8.6.9-2 -> 8.6.10-1)
[2019-11-24T14:40:42-0600] [ALPM] upgraded tk (8.6.9.1-2 -> 8.6.10-1)
[2019-11-24T14:40:42-0600] [ALPM] upgraded vte-common (0.58.2-1 -> 0.58.3-1)
[2019-11-24T14:40:42-0600] [ALPM] upgraded vte3 (0.58.2-1 -> 0.58.3-1)
[2019-11-24T14:40:42-0600] [ALPM] upgraded wpebackend-fdo (1.4.0-1 -> 1.4.0-2)
[2019-11-24T14:40:42-0600] [ALPM] upgraded xorg-server-devel (1.20.5-4 ->
1.20.6-1)
[2019-11-24T14:40:42-0600] [ALPM] upgraded yelp-xsl (3.34.0-1 -> 3.34.2-1)
[2019-11-24T14:40:43-0600] [ALPM] transaction completed
[2019-11-24T14:40:43-0600] [ALPM] running '20-systemd-sysusers.hook'...
[2019-11-24T14:40:43-0600] [ALPM] running '30-systemd-daemon-reload.hook'...
[2019-11-24T14:40:43-0600] [ALPM] running '30-systemd-udev-reload.hook'...
[2019-11-24T14:40:43-0600] [ALPM] running '30-systemd-update.hook'...
[2019-11-24T14:40:43-0600] [ALPM] running 'detect-old-perl-modules.hook'...

Nothing really looks like it should directly implicate httpd. But after the
first graceful restart of the server after the 5 packages were updated on
11/23 cause httpd to go haywire. Just thought I would pass it along.

-- 
David C. Rankin, J.D.,P.E.