[arch-general] 11/23 update caused httpd to coredump?
Dev, This is a strange one. After updates on 11/23, httpd began segfaulting -- repeatedly:
l /var/log/httpd/error_log* -rw-r--r-- 1 root root 24306171 Nov 24 14:48 /var/log/httpd/error_log -rw-r--r-- 1 root root 625 Nov 24 00:00 /var/log/httpd/error_log.1 -rw-r--r-- 1 root root 1742 Nov 21 13:19 /var/log/httpd/error_log.2 -rw-r--r-- 1 root root 504 Nov 10 00:00 /var/log/httpd/error_log.3 -rw-r--r-- 1 root root 1754 Nov 3 00:00 /var/log/httpd/error_log.4
I have a second server that confirms this exact same behavior:
l /var/log/httpd/error_log* -rw-r--r-- 1 root root 25049142 Nov 24 14:48 /var/log/httpd/error_log -rw-r--r-- 1 root root 1137 Nov 24 00:00 /var/log/httpd/error_log.1 -rw-r--r-- 1 root root 1752 Nov 17 00:01 /var/log/httpd/error_log.2 -rw-r--r-- 1 root root 504 Nov 10 00:00 /var/log/httpd/error_log.3 -rw-r--r-- 1 root root 1754 Nov 3 00:01 /var/log/httpd/error_log.4
cat error_log.1 [Thu Nov 21 13:39:09.090480 2019] [suexec:notice] [pid 898] AH01232: suEXEC mechanism enabled (wrapper: /usr/bin/suexec) [Thu Nov 21 13:39:09.146764 2019] [lbmethod_heartbeat:notice] [pid 898] AH02282: No slotmem from mod_heartmonitor [Thu Nov 21 13:39:09.378344 2019] [mpm_prefork:notice] [pid 898] AH00163: Apache/2.4.41 (Unix) OpenSSL/1.1.1d PHP/5.6.40 configured -- resuming normal operations [Thu Nov 21 13:39:09.378406 2019] [core:notice] [pid 898] AH00094: Command
head error_log [Sun Nov 24 00:00:27.104644 2019] [lbmethod_heartbeat:notice] [pid 898] AH02282: No slotmem from mod_heartmonitor [Sun Nov 24 00:00:27.115890 2019] [mpm_prefork:notice] [pid 898] AH00163: Apache/2.4.41 (Unix) OpenSSL/1.1.1d PHP/5.6.40 configured -- resuming normal operations [Sun Nov 24 00:00:27.115901 2019] [core:notice] [pid 898] AH00094: Command
The log file size jumped from 625 bytes to 24306171 bytes (1137->25049142 on the other server) in just a matter of hours. The old log contains, in its entirety: line: '/usr/bin/httpd -D FOREGROUND' [Sun Nov 24 00:00:27.074145 2019] [mpm_prefork:notice] [pid 898] AH00171: Graceful restart requested, doing restart The new log began with: line: '/usr/bin/httpd -D FOREGROUND' [Sun Nov 24 00:00:28.120350 2019] [core:notice] [pid 898] AH00051: child pid 11154 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 00:00:28.120576 2019] [core:notice] [pid 898] AH00051: child pid 11156 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 00:00:28.120728 2019] [core:notice] [pid 898] AH00051: child pid 11158 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 00:00:28.120871 2019] [core:notice] [pid 898] AH00051: child pid 11160 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 00:00:28.121060 2019] [core:notice] [pid 898] AH00051: child pid 11162 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 00:00:30.126762 2019] [core:notice] [pid 898] AH00051: child pid 11253 exit signal Segmentation fault (11), possible coredump in /etc/httpd Updating again today (which is when I caught the problem), and restarting httpd has solved whatever problem there was and httpd is now happily running fine again. The only updates that occurred on the 23rd and 24th are as follows and nothing jumps out as being a problem: [2019-11-23T03:35:07-0600] [PACMAN] starting full system upgrade [2019-11-23T03:36:09-0600] [ALPM] transaction started [2019-11-23T03:36:09-0600] [ALPM] upgraded libusbmuxd (1.0.10+29+g9db5747-1 -> 2.0.0-2) [2019-11-23T03:36:09-0600] [ALPM] upgraded libimobiledevice (1.2.0+87+g92c5462-5 -> 1.2.0+87+g92c5462-6) [2019-11-23T03:36:10-0600] [ALPM] upgraded nodejs (13.1.0-2 -> 13.2.0-1) [2019-11-23T03:36:16-0600] [ALPM] upgraded python2-typing (3.6.6-1 -> 3.7.4.1-1) [2019-11-23T03:36:16-0600] [ALPM] upgraded qpdf (9.0.2-1 -> 9.1.0-1) [2019-11-23T03:36:17-0600] [ALPM] transaction completed [2019-11-23T03:36:17-0600] [ALPM] running '30-systemd-update.hook'... [2019-11-24T14:39:58-0600] [PACMAN] Running 'pacman -Syu' [2019-11-24T14:39:58-0600] [PACMAN] synchronizing package lists [2019-11-24T14:40:00-0600] [PACMAN] starting full system upgrade [2019-11-24T14:40:31-0600] [ALPM] transaction started [2019-11-24T14:40:31-0600] [ALPM] upgraded blas (3.8.0-2 -> 3.9.0-2) [2019-11-24T14:40:31-0600] [ALPM] upgraded cblas (3.8.0-2 -> 3.9.0-2) [2019-11-24T14:40:33-0600] [ALPM] upgraded mesa (19.2.4-1 -> 19.2.6-1) [2019-11-24T14:40:33-0600] [ALPM] upgraded xorg-server-common (1.20.5-4 -> 1.20.6-1) [2019-11-24T14:40:33-0600] [ALPM] upgraded libutil-linux (2.34-6 -> 2.34-8) [2019-11-24T14:40:33-0600] [ALPM] upgraded util-linux (2.34-6 -> 2.34-8) [2019-11-24T14:40:33-0600] [ALPM] upgraded xorg-server (1.20.5-4 -> 1.20.6-1) [2019-11-24T14:40:39-0600] [ALPM] upgraded nvidia-utils (440.31-1 -> 440.36-1) [2019-11-24T14:40:39-0600] [ALPM-SCRIPTLET] If you run into trouble with CUDA not being available, run nvidia-modprobe first. [2019-11-24T14:40:39-0600] [ALPM] upgraded eglexternalplatform (1.0+3+g7c8f8e2-1 -> 1.1-1) [2019-11-24T14:40:39-0600] [ALPM] upgraded egl-wayland (1.1.4-1 -> 1.1.4-2) [2019-11-24T14:40:39-0600] [ALPM] upgraded faad2 (2.9.1-1 -> 2.9.1-2) [2019-11-24T14:40:39-0600] [ALPM] upgraded graphviz (2.42.2-6 -> 2.42.3-1) [2019-11-24T14:40:40-0600] [ALPM] upgraded lapack (3.8.0-2 -> 3.9.0-2) [2019-11-24T14:40:41-0600] [ALPM] upgraded libmagick6 (6.9.10.71-1 -> 6.9.10.74-1) [2019-11-24T14:40:41-0600] [ALPM] upgraded libwpe (1.4.0-1 -> 1.4.0.1-1) [2019-11-24T14:40:41-0600] [ALPM] upgraded man-pages (5.03-2 -> 5.04-1) [2019-11-24T14:40:41-0600] [ALPM] upgraded perl-libwww (6.41-1 -> 6.42-1) [2019-11-24T14:40:41-0600] [ALPM] upgraded python-pyasn1 (0.4.7-3 -> 0.4.8-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded python-setuptools (1:41.4.0-1 -> 1:41.6.0-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded python2-pyasn1 (0.4.7-3 -> 0.4.8-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded python2-setuptools (1:41.4.0-1 -> 1:41.6.0-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded qhull (2019.1-1 -> 2019.1-2) [2019-11-24T14:40:42-0600] [ALPM] upgraded tcl (8.6.9-2 -> 8.6.10-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded tk (8.6.9.1-2 -> 8.6.10-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded vte-common (0.58.2-1 -> 0.58.3-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded vte3 (0.58.2-1 -> 0.58.3-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded wpebackend-fdo (1.4.0-1 -> 1.4.0-2) [2019-11-24T14:40:42-0600] [ALPM] upgraded xorg-server-devel (1.20.5-4 -> 1.20.6-1) [2019-11-24T14:40:42-0600] [ALPM] upgraded yelp-xsl (3.34.0-1 -> 3.34.2-1) [2019-11-24T14:40:43-0600] [ALPM] transaction completed [2019-11-24T14:40:43-0600] [ALPM] running '20-systemd-sysusers.hook'... [2019-11-24T14:40:43-0600] [ALPM] running '30-systemd-daemon-reload.hook'... [2019-11-24T14:40:43-0600] [ALPM] running '30-systemd-udev-reload.hook'... [2019-11-24T14:40:43-0600] [ALPM] running '30-systemd-update.hook'... [2019-11-24T14:40:43-0600] [ALPM] running 'detect-old-perl-modules.hook'... Nothing really looks like it should directly implicate httpd. But after the first graceful restart of the server after the 5 packages were updated on 11/23 cause httpd to go haywire. Just thought I would pass it along. -- David C. Rankin, J.D.,P.E.
On 11/24/2019 03:57 PM, David C. Rankin wrote:
Dev,
This is a strange one. After updates on 11/23, httpd began segfaulting -- repeatedly:
l /var/log/httpd/error_log* -rw-r--r-- 1 root root 24306171 Nov 24 14:48 /var/log/httpd/error_log -rw-r--r-- 1 root root 625 Nov 24 00:00 /var/log/httpd/error_log.1 -rw-r--r-- 1 root root 1742 Nov 21 13:19 /var/log/httpd/error_log.2 -rw-r--r-- 1 root root 504 Nov 10 00:00 /var/log/httpd/error_log.3 -rw-r--r-- 1 root root 1754 Nov 3 00:00 /var/log/httpd/error_log.4
I have a second server that confirms this exact same behavior:
l /var/log/httpd/error_log* -rw-r--r-- 1 root root 25049142 Nov 24 14:48 /var/log/httpd/error_log -rw-r--r-- 1 root root 1137 Nov 24 00:00 /var/log/httpd/error_log.1 -rw-r--r-- 1 root root 1752 Nov 17 00:01 /var/log/httpd/error_log.2 -rw-r--r-- 1 root root 504 Nov 10 00:00 /var/log/httpd/error_log.3 -rw-r--r-- 1 root root 1754 Nov 3 00:01 /var/log/httpd/error_log.4
The log file size jumped from 625 bytes to 24306171 bytes (1137->25049142 on the other server) in just a matter of hours. The old log contains, in its entirety:
<snip> Just a follow-up, all is back to normal since the 11/24 update: [Sun Nov 24 14:48:40.734427 2019] [core:notice] [pid 898] AH00051: child pid 1128862 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 14:48:40.734535 2019] [core:notice] [pid 898] AH00051: child pid 1128863 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 14:48:41.876097 2019] [mpm_prefork:notice] [pid 898] AH00170: caught SIGWINCH, shutting down gracefully [Sun Nov 24 14:48:41.978494 2019] [suexec:notice] [pid 1128921] AH01232: suEXEC mechanism enabled (wrapper: /usr/bin/suexec) [Sun Nov 24 14:48:41.996637 2019] [lbmethod_heartbeat:notice] [pid 1128921] AH02282: No slotmem from mod_heartmonitor [Sun Nov 24 14:48:42.011618 2019] [mpm_prefork:notice] [pid 1128921] AH00163: Apache/2.4.41 (Unix) OpenSSL/1.1.1d PHP/5.6.40 configured -- resuming normal operations [Sun Nov 24 14:48:42.011659 2019] [core:notice] [pid 1128921] AH00094: Command line: '/usr/bin/httpd -D FOREGROUND' [Mon Nov 25 12:11:23.505137 2019] [mpm_prefork:notice] [pid 1128921] AH00170: caught SIGWINCH, shutting down gracefully [Mon Nov 25 12:12:20.422676 2019] [suexec:notice] [pid 677] AH01232: suEXEC mechanism enabled (wrapper: /usr/bin/suexec) [Mon Nov 25 12:12:20.520933 2019] [lbmethod_heartbeat:notice] [pid 677] AH02282: No slotmem from mod_heartmonitor [Mon Nov 25 12:12:22.566075 2019] [mpm_prefork:notice] [pid 677] AH00163: Apache/2.4.41 (Unix) OpenSSL/1.1.1d PHP/5.6.40 configured -- resuming normal operations [Mon Nov 25 12:12:22.566164 2019] [core:notice] [pid 677] AH00094: Command line: '/usr/bin/httpd -D FOREGROUND' No clue what happened with the 11/23 updates that originally caused the issue. Gremlins... -- David C. Rankin, J.D.,P.E.
On 11/27/2019 06:45 PM, David C. Rankin wrote:
On 11/24/2019 03:57 PM, David C. Rankin wrote:
Dev,
This is a strange one. After updates on 11/23, httpd began segfaulting -- repeatedly:
l /var/log/httpd/error_log* -rw-r--r-- 1 root root 24306171 Nov 24 14:48 /var/log/httpd/error_log -rw-r--r-- 1 root root 625 Nov 24 00:00 /var/log/httpd/error_log.1 -rw-r--r-- 1 root root 1742 Nov 21 13:19 /var/log/httpd/error_log.2 -rw-r--r-- 1 root root 504 Nov 10 00:00 /var/log/httpd/error_log.3 -rw-r--r-- 1 root root 1754 Nov 3 00:00 /var/log/httpd/error_log.4
I have a second server that confirms this exact same behavior:
l /var/log/httpd/error_log* -rw-r--r-- 1 root root 25049142 Nov 24 14:48 /var/log/httpd/error_log -rw-r--r-- 1 root root 1137 Nov 24 00:00 /var/log/httpd/error_log.1 -rw-r--r-- 1 root root 1752 Nov 17 00:01 /var/log/httpd/error_log.2 -rw-r--r-- 1 root root 504 Nov 10 00:00 /var/log/httpd/error_log.3 -rw-r--r-- 1 root root 1754 Nov 3 00:01 /var/log/httpd/error_log.4
The log file size jumped from 625 bytes to 24306171 bytes (1137->25049142 on the other server) in just a matter of hours. The old log contains, in its entirety:
<snip>
Just a follow-up, all is back to normal since the 11/24 update:
[Sun Nov 24 14:48:40.734427 2019] [core:notice] [pid 898] AH00051: child pid 1128862 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 14:48:40.734535 2019] [core:notice] [pid 898] AH00051: child pid 1128863 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Sun Nov 24 14:48:41.876097 2019] [mpm_prefork:notice] [pid 898] AH00170: <snip> [Mon Nov 25 12:12:22.566164 2019] [core:notice] [pid 677] AH00094: Command line: '/usr/bin/httpd -D FOREGROUND'
No clue what happened with the 11/23 updates that originally caused the issue. Gremlins...
Damn, Whatever it is came back after updates on 12/1 -rw-r--r-- 1 root root 71509927 Dec 2 19:22 error_log -rw-r--r-- 1 root root 24306916 Dec 1 00:00 error_log.1 -rw-r--r-- 1 root root 625 Nov 24 00:00 error_log.2 -rw-r--r-- 1 root root 1742 Nov 21 13:19 error_log.3 -rw-r--r-- 1 root root 504 Nov 10 00:00 error_log.4 Same error both server and server is down until reloaded, e.g. [Mon Dec 02 19:22:36.265040 2019] [core:notice] [pid 677] AH00051: child pid 3290821 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Mon Dec 02 19:22:36.265147 2019] [core:notice] [pid 677] AH00051: child pid 3290823 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Mon Dec 02 19:22:36.265255 2019] [core:notice] [pid 677] AH00051: child pid 3290824 exit signal Segmentation fault (11), possible coredump in /etc/httpd Any idea on on what is triggering this? -- David C. Rankin, J.D.,P.E.
On Tue, 3 Dec 2019 at 01:27, David C. Rankin
[Mon Dec 02 19:22:36.265040 2019] [core:notice] [pid 677] AH00051: child pid 3290821 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Mon Dec 02 19:22:36.265147 2019] [core:notice] [pid 677] AH00051: child pid 3290823 exit signal Segmentation fault (11), possible coredump in /etc/httpd [Mon Dec 02 19:22:36.265255 2019] [core:notice] [pid 677] AH00051: child pid 3290824 exit signal Segmentation fault (11), possible coredump in /etc/httpd
Any idea on on what is triggering this?
Are you using wsgi module by chance? I have experienced this last year and when I recompiled both apache and wsgi with debugging symbols to find out what's what, it suddenly worked again and has since!
On 12/3/19 5:36 AM, Andy Pieters wrote:
I have experienced this last year and when I recompiled both apache and wsgi with debugging symbols to find out what's what, it suddenly worked again and has since!
That sounds like use of variable before set - as debugging initializes variables to 0 (in C, C++ ... ). Could be buggy code.
On 12/03/2019 11:28 AM, Genes Lists via arch-general wrote:
On 12/3/19 5:36 AM, Andy Pieters wrote:
I have experienced this last year and when I recompiled both apache and wsgi with debugging symbols to find out what's what, it suddenly worked again and has since!
That sounds like use of variable before set - as debugging initializes variables to 0 (in C, C++ ... ).
Could be buggy code.
Nope, now wsgi... Funny thing is for 10 years I though all updates I have never had any problems with httpd. The whatever changed in the 11/23 updates it was segfaulting/coredumping repeatedly. I thought the follow-up updates solved it, but then found both servers websites down again. Restarting httpd brings the server back up, but it was shocking to look at the logs and see on one server: -rw-r--r-- 1 root root 71509927 Dec 2 19:22 error_log -rw-r--r-- 1 root root 24306916 Dec 1 00:00 error_log.1 -rw-r--r-- 1 root root 625 Nov 24 00:00 error_log.2 -rw-r--r-- 1 root root 1742 Nov 21 13:19 error_log.3 -rw-r--r-- 1 root root 504 Nov 10 00:00 error_log.4 and then on the other: -rw-r--r-- 1 root root 73141520 Dec 2 19:15 error_log -rw-r--r-- 1 root root 25049886 Dec 1 00:00 error_log.1 -rw-r--r-- 1 root root 1137 Nov 24 00:00 error_log.2 -rw-r--r-- 1 root root 1752 Nov 17 00:01 error_log.3 -rw-r--r-- 1 root root 504 Nov 10 00:00 error_log.4 Yikes! -- David C. Rankin, J.D.,P.E.
participants (3)
-
Andy Pieters
-
David C. Rankin
-
Genes Lists