[arch-general] mayday - new updates killed httpd - make_sock: could not bind to address [::]:443
guys, I updated my test box and it checked out, then I updated my server and apache will not start. The error given is: [03:58 nirvana:/etc/httpd/conf] # apachectl start (98)Address already in use: make_sock: could not bind to address [::]:443 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:443 no listening sockets available, shutting down Unable to open logs The updates tonight included: [2010-08-09 00:41] upgraded consolekit (0.4.1-2 -> 0.4.1-3) [2010-08-09 00:41] upgraded dhcpcd (5.2.5-1 -> 5.2.7-1) [2010-08-09 00:41] upgraded dnsutils (9.6.1-3 -> 9.7.1.P2-1) [2010-08-09 00:41] upgraded libnice (0.0.12-1 -> 0.0.13-1) [2010-08-09 00:41] upgraded farsight2 (0.0.20-1 -> 0.0.21-1) [2010-08-09 00:41] upgraded libva (0.31.0_p13-2 -> 1.0.4-1) [2010-08-09 00:41] upgraded ffmpeg (24460-1 -> 24460-2) [2010-08-09 00:41] installed iana-etc (2.30-1) [2010-08-09 00:41] upgraded filesystem (2010.02-4 -> 2010.07-1) [2010-08-09 00:41] upgraded hwdetect (2010.07-1 -> 2010.08-1) [2010-08-09 00:42] upgraded kernel26 (2.6.34.2-1 -> 2.6.34.2-2) [2010-08-09 00:42] upgraded kernel26-headers (2.6.34.2-1 -> 2.6.34.2-2) [2010-08-09 00:44] upgraded kernel26-lts (2.6.32.16-1 -> 2.6.32.17-2) [2010-08-09 00:44] upgraded kernel26-lts-headers (2.6.32.16-1 -> 2.6.32.17-2) [2010-08-09 00:44] upgraded networkmanager (0.8-1 -> 0.8.1-1) [2010-08-09 00:44] warning: /etc/pacman.d/mirrorlist installed as /etc/pacman.d/mirrorlist.pacnew [2010-08-09 00:44] upgraded pacman-mirrorlist (20100621-1 -> 20100808-1) [2010-08-09 00:44] upgraded python-telepathy (0.15.17-1 -> 0.15.18-1) [2010-08-09 00:44] upgraded rpcbind (0.2.0-1 -> 0.2.0-2) [2010-08-09 00:44] upgraded sqlite3 (3.7.0-1 -> 3.7.0.1-1) [2010-08-09 00:44] upgraded telepathy-haze (0.3.6-1 -> 0.4.0-1) [2010-08-09 00:44] upgraded thunderbird (3.1.1-1 -> 3.1.2-1) [2010-08-09 00:44] upgraded util-linux-ng (2.18-2 -> 2.18-3) [2010-08-09 00:44] upgraded vdpau-video (0.6.9-2 -> 0.6.10-2) [2010-08-09 00:44] upgraded vlc (1.1.2-1 -> 1.1.2-2) I have switched to a backup server so I'm not dead, but to have apache just not start was a bit disconcerting to say the least. This has got to be an issue with the kernel26 (2.6.34.2-1 -> 2.6.34.2-2) upgrade. My primary and backup servers are both Arch 64bit boxes with identical software installs. The box with the problem is an Opteron(tm) Processor 180 running on Tyan Tomcat S2865, the backup is a Phenom(tm) 9850 on an MSI K9N2 SLI Platinum. The fact that the software is identical and the primary can't make/bind to sockets anymore --- tells me what? Anyway, I need help. Any thoughts on what do check? I'll drop back to LTS and see how that goes and report back. Thanks. -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com
On Mon, 2010-08-09 at 04:19 -0500, David C. Rankin wrote:
(98)Address already in use: make_sock: could not bind to address [::]:443 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:443
Maybe you should read the error messages and check out with tools like lsof what process is using that port.
On 08/09/2010 04:31 AM, Jan de Groot wrote:
Maybe you should read the error messages and check out with tools like lsof what process is using that port.
OK, This is where I need to add some tools to my toolbox. lsof always just gives me reams of data that doesn't make much sense to me. lsof what here? -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com
lsof -i TCP | grep LISTEN Otherwise the man page of lsof On 9 August 2010 11:35, David C. Rankin <drankinatty@suddenlinkmail.com>wrote:
On 08/09/2010 04:31 AM, Jan de Groot wrote:
Maybe you should read the error messages and check out with tools like lsof what process is using that port.
OK,
This is where I need to add some tools to my toolbox. lsof always just gives me reams of data that doesn't make much sense to me. lsof what here?
-- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com
On 08/09/2010 04:31 AM, Jan de Groot wrote:
On Mon, 2010-08-09 at 04:19 -0500, David C. Rankin wrote:
(98)Address already in use: make_sock: could not bind to address [::]:443 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:443
Maybe you should read the error messages and check out with tools like lsof what process is using that port.
OK Googling help: [04:38 nirvana:/home/david] # lsof -i tcp:443 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME cupsd 1896 root 8u IPv4 7293 0t0 TCP *:https (LISTEN) cupsd 1896 root 9u IPv6 7294 0t0 TCP *:https (LISTEN) cups?? Now the question is what do I do to fix it? Currently, cups is started as one of the last things: DAEMONS=(syslog-ng network hal named @netfs @ntpd @sshd @dhcpd @crond @avahi-daemon @postfix @mysqld @dovecot @httpd @hylafax @samba @cups @sensors @upsd kdm) I guess I'll try taking them out of background processing. The over 'Arching' point (no pun intended) is how can we make sure that updates, that do not change any packages by adding/deleting, do not kill a primary part of a box? And, why this box and not the backup? -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com
On Mon, 2010-08-09 at 04:45 -0500, David C. Rankin wrote:
The over 'Arching' point (no pun intended) is how can we make sure that updates, that do not change any packages by adding/deleting, do not kill a primary part of a box? And, why this box and not the backup?
I think I've seen this said to you before, but the responsibility for your box falls on you. JGC has answered your specific question, but I'd comment on the broader assumption you make that the package maintainer is responsible for your system in a specific manner. He is not. Archers need to know their systems. You seem to, to some extent. Therefore I don't really understand the request for (as I see it) a more fool-proof system. You're not a fool.
On 08/09/2010 06:50 AM, Ng Oon-Ee wrote:
I think I've seen this said to you before, but the responsibility for your box falls on you. JGC has answered your specific question, but I'd comment on the broader assumption you make that the package maintainer is responsible for your system in a specific manner. He is not.
Archers need to know their systems. You seem to, to some extent. Therefore I don't really understand the request for (as I see it) a more fool-proof system. You're not a fool.
I agree with everything you said. The backgrounding issue was a bit surprising having 2 identical boxes behave completely opposite due to a self-imposed bit of cups Russian-roulette. The request follows logically from what you have expressed. Archer need to know their systems. When faced with unexpected behavior, I want to do my part and understand what happened. Thus the question -- "What Changed?" More specifically I guess would be "What changed in the updated packages such that cups went from losing the race to grab port 443 for the past 18 months to suddenly 'winning' the race for port 443 despite being backgrounded later in the DAEMONS line. Logically something did it for cups to go from never getting the port to always getting the port. That's what I was trying to learn and understand. I suspect the only person that can answer that is the one of a few that know what influences the port race and what change they made that would affect it. Regardless, learning has occurred. An obscure cups setting has been changed, lsof is better understood and my system is more robust as a result. A good start to the day :p -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com
On 08/09/2010 04:19 AM, David C. Rankin wrote:
Anyway, I need help. Any thoughts on what do check? I'll drop back to LTS and see how that goes and report back. Thanks.
Jan, Thank you for your prodding in the right direction. It was fricking cups that was the problem and the work around was to stop cups and start apache and restart cups: [04:42 nirvana:/home/david] # lsof -i tcp:443 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME cupsd 1896 root 8u IPv4 7293 0t0 TCP *:https (LISTEN) cupsd 1896 root 9u IPv6 7294 0t0 TCP *:https (LISTEN) [04:45 nirvana:/home/david] # rccups stop /etc/rc.d/functions: line 303: .: /etc/rc.d/functions.d/functions.d: is a directory :: Stopping CUPS Daemon [DONE] [04:45 nirvana:/home/david] # lsof -i tcp:443 [04:45 nirvana:/home/david] # rchttpd start /etc/rc.d/functions: line 303: .: /etc/rc.d/functions.d/functions.d: is a directory :: Starting Apache Web Server [DONE] [04:45 nirvana:/home/david] # rccups start /etc/rc.d/functions: line 303: .: /etc/rc.d/functions.d/functions.d: is a directory :: Starting CUPS Daemon [DONE] Now how do I fix this in rc.conf to make sure cups doesn't screw me over again? Take http out of the background? I've booted this box 15 times or so in the past 1.5 years and this has never been a problem until the updates tonight. What changed? How many others are going to get caught? -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com
On Mon, 2010-08-09 at 04:49 -0500, David C. Rankin wrote:
[04:45 nirvana:/home/david] # rccups start /etc/rc.d/functions: line 303: .: /etc/rc.d/functions.d/functions.d: is a directory :: Starting CUPS Daemon [DONE]
Now how do I fix this in rc.conf to make sure cups doesn't screw me over again? Take http out of the background?
I've booted this box 15 times or so in the past 1.5 years and this has never been a problem until the updates tonight. What changed? How many others are going to get caught?
First of all, why don't you fix that annoying error with /etc/rc.d/functions.d that pops up on every rc script invocation? As for cups: the default cupsd.conf contains "Listen localhost:631" and "Listen /var/run/cups/cups.sock". I think your cupsd.conf contains an entry for 443 also there. The problem with your setup is that your setup relies on a failure: apache binds to https before cups can, so cups doesn't bind to 443 anymore. Because you're using backgrounding, any update to your system can interfere with timing and cause issues like these if your configuration is not right.
On 08/09/2010 05:00 AM, Jan de Groot wrote:
First of all, why don't you fix that annoying error with /etc/rc.d/functions.d that pops up on every rc script invocation?
OK - you're on, don't keep me in suspense, just go ahead and tell me what notice I missed that says do X or Y to fix the very annoying /etc/rc.d/functions.d that pops up with every init... I thought that was a side effect of the bashification of the init scrips and that an update would fix it? wrong?
As for cups: the default cupsd.conf contains "Listen localhost:631" and "Listen /var/run/cups/cups.sock". I think your cupsd.conf contains an entry for 443 also there.
Damn, your clairvoyant: # # "$Id: cupsd.conf.in 8805 2009-08-31 16:34:06Z mike $" # # Sample configuration file for the CUPS scheduler. See "man cupsd.conf" for a # complete description of this file. # # Log general information in error_log - change "warn" to "debug" # for troubleshooting... ServerName nirvana.3111skyline.com ServerAdmin david@3111skyline.com ServerAlias nirvana ServerAlias www.3111skyline.com ServerAlias localhost # Allow remote access Port 631 SSLPort 443 <snip> now #SSLPort 443 I have no recognition of making that change and don't know why I would have done that. Could that be a host:631 cups auto setting in response to some other selection? The problem with your setup is that your setup
relies on a failure: apache binds to https before cups can, so cups doesn't bind to 443 anymore.
(I see that) Because you're using backgrounding, any
update to your system can interfere with timing and cause issues like these if your configuration is not right.
Yes, I'm using backgrounding and my config hasn't changed since I first started using Arch and was done in response to a post here pointing out that it was recommended to bring up hal in the DAEMONS line and then background everything thereafter up to your display manager. Was that advise incorrect? I have been running with the following: DAEMONS=(syslog-ng network hal named @netfs @ntpd @sshd @dhcpd @crond @avahi-daemon @postfix @mysqld @dovecot @httpd @hylafax @samba @cups @sensors @upsd kdm) What needs to be changed? Or -- where is an Arch reference that provides the pros & cons of backgrouding the various processes? I guess I have been lucky because Arch has been bullet-proof since April '09 as my primary server. Last night was the first DAEMONS issue I had run into except for the little 'display manager respawning too fast' nit. What say the experts on the backgrounding issue? -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com
On 08/09/2010 01:20 PM, David C. Rankin wrote:
On 08/09/2010 05:00 AM, Jan de Groot wrote:
First of all, why don't you fix that annoying error with /etc/rc.d/functions.d that pops up on every rc script invocation?
OK - you're on, don't keep me in suspense, just go ahead and tell me what notice I missed that says do X or Y to fix the very annoying /etc/rc.d/functions.d that pops up with every init...
I thought that was a side effect of the bashification of the init scrips and that an update would fix it? wrong?
As for cups: the default cupsd.conf contains "Listen localhost:631" and "Listen /var/run/cups/cups.sock". I think your cupsd.conf contains an entry for 443 also there.
Damn, your clairvoyant:
# # "$Id: cupsd.conf.in 8805 2009-08-31 16:34:06Z mike $" # # Sample configuration file for the CUPS scheduler. See "man cupsd.conf" for a # complete description of this file. #
# Log general information in error_log - change "warn" to "debug" # for troubleshooting... ServerName nirvana.3111skyline.com ServerAdmin david@3111skyline.com ServerAlias nirvana ServerAlias www.3111skyline.com ServerAlias localhost # Allow remote access Port 631 SSLPort 443 <snip>
now #SSLPort 443
I have no recognition of making that change and don't know why I would have done that. Could that be a host:631 cups auto setting in response to some other selection?
The problem with your setup is that your setup
relies on a failure: apache binds to https before cups can, so cups doesn't bind to 443 anymore.
(I see that)
Because you're using backgrounding, any
update to your system can interfere with timing and cause issues like these if your configuration is not right.
Yes, I'm using backgrounding and my config hasn't changed since I first started using Arch and was done in response to a post here pointing out that it was recommended to bring up hal in the DAEMONS line and then background everything thereafter up to your display manager. Was that advise incorrect?
I have been running with the following:
DAEMONS=(syslog-ng network hal named @netfs @ntpd @sshd @dhcpd @crond @avahi-daemon @postfix @mysqld @dovecot @httpd @hylafax @samba @cups @sensors @upsd kdm)
What needs to be changed? Or -- where is an Arch reference that provides the pros & cons of backgrouding the various processes?
I guess I have been lucky because Arch has been bullet-proof since April '09 as my primary server. Last night was the first DAEMONS issue I had run into except for the little 'display manager respawning too fast' nit.
What say the experts on the backgrounding issue?
I'm not an expert but as I see it you should have seen it coming, backgrounding the daemons is ok as long as the daemons you background are independent. On the listening port issue, you did that to yourself, you must have explicitly configured cups to listen on port 443 and you configured something else to listen on the same port, only two outcomes are possible, either the applications fail gracefully and move on the best wait they can or they will explode and refuse to work (which is always lots of fun). Besides you should have noticed before that something wasn't right, you did test your system to check if all the functionalities you wanted to be working were actually working, right? I guess now you know you can't have two programs listening on the same port(s) and it's the sysadmin responsibility to assure that doesn't happen or deal with it when the sticky smelly stuff hits the fan. -- Mauro Santos
participants (5)
-
ALAUX, Guillaume
-
David C. Rankin
-
Jan de Groot
-
Mauro Santos
-
Ng Oon-Ee