[arch-general] Linux server crash causing router switch to stop working

[David Rosenstrauch]

Tnx much for the reply!  Responses inline below.


On 1/21/22 2:29 PM, Genes Lists via arch-general wrote:
>      When you say server crashes - I assume kernel crashed and is not 
> functioning (as opposed to kernel is up and running but network is not 
> doing what I want)? Does server respond to any keyboard or mouse 
> movement on console?

It's hard to know for sure exactly what's happening, as the server is 
headless.  It does have a KVM-over-IP hooked up to it, though.  But when 
I try use the KVM when the server gets in this state, it doesn't show 
anything on the screen, or respond to any keyboard or mouse input.  (At 
least not that I can see.)  Eventually I wind up having to just power 
off the machine.


>       What do kernel logs show leading up to the crash? If the kernel 
> oopsed it is often logged. Prior log messages may be helpful in tracking 
> down that problem.

When the crash happened this time I went back and looked at the logs and 
system journal.  Last log message was at 2.59:01am.  Timing is a little 
suspicious, as I do have some cron jobs that kick off at 3am.  However, 
the first time the crash happened it was during the day, so I don't 
think 3am cron jobs could be solely responsible.


> Is server running desktop or no graphics?

It's mostly a server.  But I do have a desktop manager running (using 
the CPU's onboard graphics) and connected to the KVM in case I need to 
log in.  (Which I rarely do.  I mostly SSH in.)


>      Your network topology - Is the Netgear routing to internet or is it 
> your new server or a separate firewall?  Is your internet ipv4 or ipv6 
> or both? Same question - is internet firewall routing (ipv6) or NAT'ing 
> (ipv4)?

Netgear router is home LAN's gateway to the internet.  It uses a 
combination of ipv4 and v6.


>      Is there a periodicity to the problem? Like say dhcp lease 
> expiration length or something?

Not that I can tell.  Again, the issue only happened twice - once at 
around 10am, and the second time at (it looks like) 3am.  And there was 
probably a good week or so between the 2 events.


>      Can we assume that Netgear has been working fine with same configs 
> before new server deployed?

Yep.  Router had been pretty solid up until this point.  Plus the fact 
that unplugging the server's network cable makes the issue go away leads 
me to believe the problem isn't the router.


> Assuming the netgear is your firewall / 
> router to internet - When things are "broken" can internal clients see 
> each other (say ping from one client to another) or is all internal 
> traffic hung up as well as internet traffic?

Anything using an ethernet port on the router pretty much goes dead. 
That includes the new server itself, a network printer, the upstream 
modem that the router is connected to, a POE wifi extender in the other 
room, etc.  IIRC, it might have still been possible to connect to the 
router using wifi during the outage.  (I.e., it would respond and assign 
an IP address with DHCP.)  But with the upstream modem unreachable, 
having a wifi connection wasn't of much use.


>      And in that vein, sorry for obvious, but I'll ask anyway - can I 
> assume only 1 server (or kea with hot-standby) is providing dhcp service?

Yep, only 1 machine on the network handing out IP addresses - the router 
itself.


>       Also I notice that latest stable dd-wrt on website is r44715 and 
> your build seems to be beta from last July - I note there are newer 
> builds of the beta - I have no view on the firmware just making an 
> observation.

Yes, I am definitely a bit behind on dd-wrt updates.  But the version 
I'm running has been quite stable up till now, so I didn't see any 
urgency to update.


One thing I did notice after doing a bit more digging on this issue: 
although the r8169 network module does seem to work with the mobo's 
onboard network chip (RTL8125B), that's not technically the right driver 
for it.  There's an r8125 module that's not part of the kernel, which is 
available on AUR.  (https://aur.archlinux.org/packages/r8125-dkms/) 
I've switched over to start using that (and blacklisted r8169).  I've 
also upgraded to the most recent kernel (5.16.2).  So I'm watching to 
see if either/both of those changes eliminate the issue.


I was mostly posting to the list really to ask if anyone had heard of 
such a thing as a crashed server somehow either sending screwed up 
network packets or flooding the network in such a way that it could 
render a router/switch inoperable.  From the limited amount I know of 
networking I think that might be possible.  But I don't know exactly how 
one would remedy something like that.  I guess fixing the underlying 
issue that is crashing the server would be the way to do that, but I 
haven't been able to pin down the cause yet.

Anyway, thanks again for the response, and I appreciate the debugging 
tips.  If any other ideas come to mind, please LMK


Thanks,

DR