Network Bridge - Do not pass back to router
Hello, So my current setup on my server is a network bridge (br0) with a physical interface as a slave to the bridge. I deploy lxc containers using the bridge, it is not a NAT bridge, and thus DHCP is used to lease an address from the physical router in which the server is connected to. My issue is, each LXC container can communicate with one another over the bridge, this means as I have an Arch mirror in a container, I can pull packages close to the SATA limit (just under 6gbps). However when the host wants to pull from the Arch mirror, it passes back to the router and then back to the server then to the bridge to the LXC container. I am not very experienced with virtual interfaces, but I would like the server to stop wasting router resources passing packets back to the router just to be passed back into itself. Any advice would be useful. Thanks, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
El mié, 25 ene 2023 a las 23:36, Polarian (
Hello,
[...]
Any advice would be useful.
From what you say initially both the bridge and your main network (the one that connects your client machines with the bridge) are in the same network, let's say 192.168.1.0/24. However the behavior you describe is as if the bridge is on a different network (say 192.168.10.0/24) and therefore needs to go to the router (which knows both networks and can reach them) in order to communicate. The truth is that I would need a small diagram of your network to be able to figure out what might be happening, since with the data you provide it is a bit complicated to know. Thanks in advance. -- Óscar García Amor | ogarcia at moire.org | http://ogarcia.me
Hello, I have attempted to make a network diagram below: +----------------------+ | | | Router | | [192.168.2.1] | | | | | +----------------------+ | | | V +-----------------------+ | | | Server | | [Physical NIC] | | | +-----------------------+ | | | V +-------------------------+ | | | Bridge (br0) | | [192.168.2.2] | | | +-------------------------+ | | +------------------------+ | | | | V V +--------------+ +----------------+ |LXC Container | | LXC Container | |[192.168.2.11]| | [192.168.2.10] | |(Arch Mirror) | | | +--------------+ +----------------+ Sending the nmcli log might help too so here: br0: connected to bridge-br0 "br0" bridge, E2:1B:81:AC:C7:A1, sw, mtu 1500 ip4 default inet4 192.168.2.2/24 route4 192.168.2.0/24 metric 0 route4 default via 192.168.2.1 metric 0 inet6 fe80::d474:abac:4022:cecd/64 route6 fe80::/64 metric 1024 enp1s0: connected to bridge-slave-enp1s0 "Realtek RTL8111/8168/8411" ethernet (r8169), D0:27:88:94:DB:07, hw, mtu 1500 master br0 veth3d143e61: unmanaged "veth3d143e61" ethernet (veth), 02:C5:A3:61:D0:61, sw, mtu 1500 veth4cd972f9: unmanaged "veth4cd972f9" ethernet (veth), 0A:19:91:57:A8:B5, sw, mtu 1500 veth696dc3aa: unmanaged "veth696dc3aa" ethernet (veth), BE:4E:23:0A:02:DD, sw, mtu 1500 veth734ec5cc: unmanaged "veth734ec5cc" ethernet (veth), 5E:C9:71:5C:34:EC, sw, mtu 1500 vethc171d051: unmanaged "vethc171d051" ethernet (veth), 02:47:E1:3C:BC:8E, sw, mtu 1500 lo: unmanaged "lo" loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536 All the veths are LXC containers running, the are bridged using the br0 interface, they are assigned IP addresses from DHCP server running on the router. When 192.168.2.10 pulls from 192.168.2.11 (The arch mirror) the bridge passes the traffic from one veth to the other. However when the Physical server tries to pull from the Arch mirror, it does the following: 192.168.2.2 (server) --> 192.168.2.1 (router, default gateway) --> 192.168.2.11 (pass back to the server NIC but passes it into the lXC container instead). I want to eliminate the hop to the router, I thought the bridge slave would do this, but obviously not. I know my diagram sucks but I hope it and the nmcli output will help :) Thanks, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
Note.. the diagram has messed up in transit... sorry about this :/ -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
Hello,
Hi! First of all, sorry for the delay, too much work this week.
I have attempted to make a network diagram below:
[..]
I know my diagram sucks but I hope it and the nmcli output will help :)
The truth is that it is a very strange behavior. Especially considering that the 192.168.2.0/24 network is directly connected and therefore would not have to go to the router at all. Out of curiosity, what happens when you physically disconnect the server from the router, does it communicate with the LXC containers? If you do an `ip ro` it looks like this? 192.168.2.0/24 dev br0 proto kernel scope link src 192.168.2.2 Greetings. -- Óscar García Amor | ogarcia at moire.org | http://ogarcia.me
Hello, Sorry for the late reply, I was busy over the weekend. The output of ip ro is: default via 192.168.2.1 dev br0 proto dhcp src 192.168.2.2 metric 425 192.168.2.0/24 dev br0 proto kernel scope link src 192.168.2.2 metric 425 Nothing seems wrong here, unless there is a priority system? As the default is to pass back to the router, and then it is to the bridge to handle it, so maybe it is a priority related issue within the routing table?
The truth is that it is a very strange behavior. Especially considering that the 192.168.2.0/24 network is directly connected and therefore would not have to go to the router at all.
Out of curiosity, what happens when you physically disconnect the server from the router, does it communicate with the LXC containers?
If I unplugged the Ethernet cable, yes the lxc containers would lose network access, their only route is via a single cable. Thanks for the help, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
Hello, This issue is a lot worse than I initially thought, and I got no clue what is causing it. After further investigation into the issue, and observation of the interfaces the packets are passing through, the following path has been determined: server (host interface) --> router (local interface) --> router (WAN interface) --> exchange --> router (WAN interface) --> router (LAN interface) --> server (host interface) --> server (bridge interface, br0) --> LXC container (vethernet) This is seriously bad, internal packets are being leaked, and it is not a problem with a router, the router rules are set correctly. I am going to tcpdump every hop I can control and see what is going on, if anyone here knows anything about this issue please let me know ASAP! Thanks, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
Scrap that, I have figured out that the reason it was leaving the LAN was because of a misconfiguration. I have fixed it, this thread can be ignored now! Thanks for all the suggestions, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
Could you share what the misconfiguration and fix was, so that if someone runs into the same problem in the future they could have it as reference?
On 1/31/23 3:51 PM, Polarian
Scrap that,
I have figured out that the reason it was leaving the LAN was because of a misconfiguration.
I have fixed it, this thread can be ignored now!
Thanks for all the suggestions,
Its embarrassing, Lets just say make sure to check which IP address you are sending packets to before worrying it is not bouncing in the LAN, I had accidentally set the wrong destination. And also highly recommend tcpdump for finding these issues, took me 2 minutes after tcpdumping the traffic to realise the packets had the wrong destination address. -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
participants (4)
-
nihilistzsche＠gmail.com
-
ogarcia＠moire.org
-
Polarian
-
Óscar García Amor