Any though of having archlinux-keyring-wkd-sync check for iptables and recommend rule?
Archdevs, Depending on how restrictive the iptables rules, if the IP for archlinux-keyring-wkd-sync falls into a blocked range, the logs quickly fill. An idea is to have the service insert a temporary rule to either (1) allow the IP for the sync check, or (2) allow established, related connections while the service runs. It may also be worth updating the wiki to provide model rules for iptables/nftables to allow archlinux-keyring-wkd-sync to run successfully. Just food for thought. -- David C. Rankin, J.D.,P.E.
On 9/20/23 04:36, David C. Rankin wrote:
Archdevs,
Depending on how restrictive the iptables rules, if the IP for archlinux-keyring-wkd-sync falls into a blocked range, the logs quickly fill. An idea is to have the service insert a temporary rule to either (1) allow the IP for the sync check, or (2) allow established, related connections while the service runs.
It may also be worth updating the wiki to provide model rules for iptables/nftables to allow archlinux-keyring-wkd-sync to run successfully.
Just food for thought.
You brought this up in Feb [1] and then as now, I don't understand what actual problem you're facing. Inbound 'blocked ranges' (SYN packets) should have no effect. Nothing is 'inbound' to your machines other than replies initiated from your machine - i.e. ESTABLISHED,RELATED. So if you're not able to get replies from arch web servers back to your own machines, then likely your firewall rules are incorrect. As per the earlier thread, WKD is simply a "web key directory" service - so all the application is doing is pulling from a web server. Unless you're blocking outbound packets to such web servers everything should just work provided you allow arch to reply when you go their web servers. You should not need any nftables (or legacy iptables) rules provided you allow client machine to have access to the web servers. One caveat would be if using nftables instead of iptables. nftables supports 2 kinds of blocks - 'netdev' and 'inet'. 'inet' are the normal blocks. 'netdev' blocks very early - and any IPs blocked at that level would not allow inbound or even replies. For anything you want to get replies for you should use 'inet' blocks not netdev. iptables didn't have netdev blocks available last I used it several years ago. What problem do you actually have? best gene [1] https://lists.archlinux.org/archives/list/arch-general@lists.archlinux.org/t...
On 9/20/23 07:10, Genes Lists wrote:
You brought this up in Feb [1] and then as now, I don't understand what actual problem you're facing. Inbound 'blocked ranges' (SYN packets) should have no effect. Nothing is 'inbound' to your machines other than replies initiated from your machine - i.e. ESTABLISHED,RELATED. So if you're not able to get replies from arch web servers back to your own machines, then likely your firewall rules are incorrect.
As per the earlier thread, WKD is simply a "web key directory" service - so all the application is doing is pulling from a web server. Unless you're blocking outbound packets to such web servers everything should just work provided you allow arch to reply when you go their web servers.
You should not need any nftables (or legacy iptables) rules provided you allow client machine to have access to the web servers.
One caveat would be if using nftables instead of iptables. nftables supports 2 kinds of blocks - 'netdev' and 'inet'. 'inet' are the normal blocks. 'netdev' blocks very early - and any IPs blocked at that level would not allow inbound or even replies. For anything you want to get replies for you should use 'inet' blocks not netdev. iptables didn't have netdev blocks available last I used it several years ago.
What problem do you actually have?
Genes, You helped greatly with your model nftables.conf! Thank you! Things worked great. I used that to find out how to do the same thing in iptables. The problem is Archlinux now uses Hetzner as the cloud provider with most of the Arch IPs falling in the 95.216.xx.xx range. Unfortunately, there is a lot of miscreants that attempt intrusions from that range so it is dropped in my iptables setup. I had tried to whitelist (ACCEPT) specific Arch IPs, but the archlinux-keyring-wkd-sync was illusive to get a stable IP for. This causes the journal fill with hundreds of thousands of lines of cruft. It makes 'audit' look like a nice journal user. This is compounded by the service file including: Restart=on-failure RestartSec=5minutes Your solution of enabling related, established connections worked, but for those who are not steeped in iptables lore, it wasn't immediately apparent that provided the solution. archlinux-keyring-wkd-sync flies under the RADAR on install as a dependent package whose service file is automatically. Rather than the user having to do something more to ensure archlinux-keyring-wkd-sync can run, I would propose: (1) having archlinux-keyring-wkd-sync check connectivity and for a firewall and add a temporary rule whitelisting the IP it will use so the burden isn't on the user; or (2) on connectivity failure, not restart in perpetuity only to fail again, and again, and again until it can establish a connection successfully. Surely we can at least break the sync loop and disable restart when a connectivity failure occurs and not continue to loop over every signature? It's fine to let is try and run again on schedule a week later and see if the connectivity has gone away. I would also propose adding the model ntfables and iptables rules to the wiki, so anyone else faced with the problem of having archlinux-keyring-wkd-sync fail filling the logs due to a firewall issue have that information at their disposal. This was the circumstance and thoughts I had on the issue so I thought I would pass them along. And thank you again Genes for originally having provided me with the solution. -- David C. Rankin, J.D.,P.E.
On 21-09-2023 01:52, David C. Rankin wrote:
archlinux-keyring-wkd-sync flies under the RADAR on install as a dependent package whose service file is automatically. Rather than the user having to do something more to ensure archlinux-keyring-wkd-sync can run,
archlinux-keyring-wkd-sync uses a systemd timer and those can only be masked / unmasked . Did you have keyring issues before the archlinux-keyring package was created/added ? If not , you can disable the timer - like I have done - by executing (as root) sysctl mask archlinux-keyring-wkd-sync.timer LW
On 9/21/23 02:38, Lone_Wolf wrote:
Did you have keyring issues before the archlinux-keyring package was created/added ?
If not , you can disable the timer - like I have done - by executing (as root)
sysctl mask archlinux-keyring-wkd-sync.timer
No, I didn't have any issues with the keyring or keys. The only reason it got my attention was checking the journal looking for a mail message, dhcp handout or named zone transfer and finding thousands and thousands of lines of archlinux-keyring-wkd-sync timeouts. Genes fix for related, established connections solve the problem, but that just got me thinking there has to be a better way than having the user stumble across the issue if they have it and then fumble around looking for a fix. Having archlinux-keyring-wkd-sync check connectivity before looping over each key seemed reasonable. I want the sync to run. I think it is a good idea. It just saves having to do pacman -Sy --needed archlinux-keyring when a key issue surfaces. If it is worth adding a check for connectivity, it shouldn't be that hard to do, if not, well at least the smart folks thought about it and decided it wasn't needed. Adding a note to the wiki would be a good fix as well. At least a note that addresses the event of a connection failure due to firewall config and suggests adding the related, established rule to whatever firewall you are running with a short example for nftables and iptables could be done in about 6 lines. I'll drop the idea in the talk page. -- David C. Rankin, J.D.,P.E.
On 2023-09-21 09:52 AM, David C. Rankin wrote:
You helped greatly with your model nftables.conf! Thank you! Things worked great. I used that to find out how to do the same thing in iptables.
The problem is Archlinux now uses Hetzner as the cloud provider with most of the Arch IPs falling in the 95.216.xx.xx range. Unfortunately, there is a lot of miscreants that attempt intrusions from that range so it is dropped in my iptables setup. I had tried to whitelist (ACCEPT) specific Arch IPs, but the archlinux-keyring-wkd-sync was illusive to get a stable IP for.
I still don't understand your issue. If you're actively blocking outbound to Hetzner ranges then that is a YOU problem. Forgive my bluntness, but I think you have a fundamental misunderstanding of how a firewall works.
This causes the journal fill with hundreds of thousands of lines of cruft. It makes 'audit' look like a nice journal user. This is compounded by the service file including:
Again, this is because of what you seem to be doing and blocking outbound traffic because you think the other baddies on Hetzner ranges can get in if you don't blanket block all the IPs.
Your solution of enabling related, established connections worked, but for those who are not steeped in iptables lore, it wasn't immediately apparent that provided the solution.
"Steeped in iptables lore" is a bit of a stretch, the Arch wiki covers the simplest iptables setup that is elegant and effective. If you're locking down all traffic and only allowing certain outbound things then, again, it's a you thing to deal with. https://wiki.archlinux.org/title/Simple_stateful_firewall
archlinux-keyring-wkd-sync flies under the RADAR on install as a dependent package whose service file is automatically. Rather than the user having to do something more to ensure archlinux-keyring-wkd-sync can run, I would propose:
(1) having archlinux-keyring-wkd-sync check connectivity and for a firewall and add a temporary rule whitelisting the IP it will use so the burden isn't on the user; or
(2) on connectivity failure, not restart in perpetuity only to fail again, and again, and again until it can establish a connection successfully.
Surely we can at least break the sync loop and disable restart when a connectivity failure occurs and not continue to loop over every signature? It's fine to let is try and run again on schedule a week later and see if the connectivity has gone away.
I would also propose adding the model ntfables and iptables rules to the wiki, so anyone else faced with the problem of having archlinux-keyring-wkd-sync fail filling the logs due to a firewall issue have that information at their disposal.
You propose all of this to workaround something you're actively doing / blocking. Please don't do any of this Arch, it'd be creating a mess for the rest of us who adhere to sane defaults. -- Simon Perry (aka Pezz)
On 9/21/23 03:12, Simon Perry wrote:
If you're actively blocking outbound to Hetzner ranges then that is a YOU problem.
No, outbound was fine, it was the INPUT chain block from the 95.216 ranges that got me. I have a top-100 script that dumps the address ranges with the top 100 blocked invalid ssh, smtp, etc.. attempts and those ranges from RIPE, APNIC, and AFRINIC are added to my INPUT chain to drop - that is what also blocked the Arch IPs. Yes, a bit heavy-handed, but I have no overseas business and this eliminated 99% of the annoying server intrusion attempts. -- David C. Rankin, J.D.,P.E.
Hi David,
No, outbound was fine, it was the INPUT chain block from the 95.216 ranges that got me. […]
I might be wrong but this thread reads like there is a misunderstanding of what the difference is between "inbound" and "INPUT". The two phrases are not the same. At the risk of mansplaining the difference, I hope this clears up some confusion. First let us define "inbound" and "outbound". "inbound" usually refers to packets to your computer/machine (e.g. internet). "outbound" usually refers to packets from your computer/machine to somewhere else (internet, NAS, etc.). "INPUT" refers to an iptables chain [1]. Both inbound and outbound packets will go through this chain (for nftables this is very similar but the name of the chain might be different). This means that if you block everything related to a certain IP address in the INPUT chain, you will also block outbound traffic. If you would drop all default rules from the net filter that make your firewall stateful [2]. You could end up with the following situation: You have something like `iptables -A INPUT -s $bad_source_ip -j DROP` dropping all packets from $bad_source_ip. 1. You try to establish a connection to the $bad_source_ip 2. You don't have a rule that makes your firewall stateful and accepts the reply packet before every other rule can take effect 3. The rule above takes effect and the reply packet from the $bad_source_ip will be dropped 4. (Due to TCP's behaviour there will be retries that will end up with the same result) So my conclusion here would be that you did one of two things. 1. Make you firewall un-stateful by removing the default ruleset 2. You blocked the "bad ips" not just as source but also as destination TL;DR: INPUT chain also takes effect for outbound traffic. So you need to allow related and established packets to get accepted. If you want to check if you removed your statefulness from your firewall, you can do a `iptables -S |grep -e RELATED -e ESTABLISHED`. I would expect some lines to come up either in the INPUT chain or a chain that is very early jumped to in the INPUT chain. If you do not want to get involved with iptables/nftables "wizardry", I would recommend something like ufw[3] which has some defaults and can easily be configured via the GUI with gufw. Best, Chistian [1]: https://en.wikipedia.org/wiki/Iptables#/media/File:Netfilter-packet-flow.svg [2]: https://wiki.archlinux.org/title/Simple_stateful_firewall [3]: https://wiki.archlinux.org/title/Uncomplicated_Firewall
Christian <syphdias+archlinuxml@gmail.com> wrote:
Hi David,
No, outbound was fine, it was the INPUT chain block from the 95.216 ranges that got me. […]
I might be wrong but this thread reads like there is a misunderstanding of what the difference is between "inbound" and "INPUT". The two phrases are not the same. At the risk of mansplaining the difference, I hope this clears up some confusion.
First let us define "inbound" and "outbound". "inbound" usually refers to packets to your computer/machine (e.g. internet). "outbound" usually refers to packets from your computer/machine to somewhere else (internet, NAS, etc.).
"INPUT" refers to an iptables chain [1]. Both inbound and outbound packets will go through this chain (for nftables this is very similar but the name of the chain might be different).
Referring to the figure at https://netfilter.org/documentation/HOWTO/packet-filtering-HOWTO-6.html , are you saying A program running on the box can send network packets. These packets pass through the OUTPUT chain only if the INPUT chain allows it ? If you do, note my understanding of statement 4 at buttom of the link is different. Am I wrong? -- u34
This means that if you block everything related to a certain IP address in the INPUT chain, you will also block outbound traffic.
If you would drop all default rules from the net filter that make your firewall stateful [2]. You could end up with the following situation: You have something like `iptables -A INPUT -s $bad_source_ip -j DROP` dropping all packets from $bad_source_ip. 1. You try to establish a connection to the $bad_source_ip 2. You don't have a rule that makes your firewall stateful and accepts the reply packet before every other rule can take effect 3. The rule above takes effect and the reply packet from the $bad_source_ip will be dropped 4. (Due to TCP's behaviour there will be retries that will end up with the same result)
So my conclusion here would be that you did one of two things. 1. Make you firewall un-stateful by removing the default ruleset 2. You blocked the "bad ips" not just as source but also as destination
TL;DR: INPUT chain also takes effect for outbound traffic. So you need to allow related and established packets to get accepted.
If you want to check if you removed your statefulness from your firewall, you can do a `iptables -S |grep -e RELATED -e ESTABLISHED`. I would expect some lines to come up either in the INPUT chain or a chain that is very early jumped to in the INPUT chain.
If you do not want to get involved with iptables/nftables "wizardry", I would recommend something like ufw[3] which has some defaults and can easily be configured via the GUI with gufw.
Best, Chistian
[1]: https://en.wikipedia.org/wiki/Iptables#/media/File:Netfilter-packet-flow.svg [2]: https://wiki.archlinux.org/title/Simple_stateful_firewall [3]: https://wiki.archlinux.org/title/Uncomplicated_Firewall
https://netfilter.org/documentation/HOWTO/packet-filtering-HOWTO-6.html , are you saying
A program running on the box can send network packets. These packets pass through the OUTPUT chain only if the INPUT chain allows it
?
If you do, note my understanding of statement 4 at buttom of the link is different. Am I wrong?
You are correct. I was wrong. You can even see it in the flow diagram I linked [1]. Thank you for pointing that out! If it was on a separate router/firewall machine the reasoning would hold, I think. Please correct me if I am wrong! I guess, it is back to not understanding why blocking inbound connections would be a problem for outbound connections. Best, Christian [1]: https://en.wikipedia.org/wiki/Iptables#/media/File:Netfilter-packet-flow.svg
Christian <syphdias+archlinuxml@gmail.com> wrote:
https://netfilter.org/documentation/HOWTO/packet-filtering-HOWTO-6.html , are you saying
A program running on the box can send network packets. These packets pass through the OUTPUT chain only if the INPUT chain allows it
?
If you do, note my understanding of statement 4 at buttom of the link is different. Am I wrong?
You are correct. I was wrong. You can even see it in the flow diagram I linked [1]. Thank you for pointing that out!
If it was on a separate router/firewall machine the reasoning would hold, I think. Please correct me if I am wrong!
Quting https://netfilter.org/documentation/HOWTO/packet-filtering-HOWTO-6.html a program running on the box can send network packets. These packets pass through the OUTPUT chain immediately As it says. No matter if this is a desktop, laptop, router, firewall. A local process sending packets passes its packets to the OUTPUT chain immediately. As a side note, the title of the link is How Packets Traverse The Filters. It could be you are confused because there is an indirect interaction between packets in the OUTPUT and INPUT chains. If a process is sending packets to www.archlinux.org in the OUTPUT chain, but blocks www.archlinux.org in the INPUT chain, it could have thought its packets didn't went out. Which is not the case. The fact that the process blocks the replies doesn't necessarily prove www.archlinux.org didn't reply. Furthere more, the order of the rules matter. With pseudo rules, www.archlinux.org drop www.archlinux.org {ESTABLISHED,RELATED} accept has different semantics then www.archlinux.org {ESTABLISHED,RELATED} accept www.archlinux.org drop And the drop rule in the later example is not necessarily with no effect. It is meaningless at the former example. -- u34
I guess, it is back to not understanding why blocking inbound connections would be a problem for outbound connections.
Best, Christian
[1]: https://en.wikipedia.org/wiki/Iptables#/media/File:Netfilter-packet-flow.svg
u34@net9.cf wrote:
Christian <syphdias+archlinuxml@gmail.com> wrote:
https://netfilter.org/documentation/HOWTO/packet-filtering-HOWTO-6.html , are you saying
A program running on the box can send network packets. These packets pass through the OUTPUT chain only if the INPUT chain allows it
?
If you do, note my understanding of statement 4 at buttom of the link is different. Am I wrong?
You are correct. I was wrong. You can even see it in the flow diagram I linked [1]. Thank you for pointing that out!
If it was on a separate router/firewall machine the reasoning would hold, I think. Please correct me if I am wrong!
Quting https://netfilter.org/documentation/HOWTO/packet-filtering-HOWTO-6.html
a program running on the box can send network packets. These packets pass through the OUTPUT chain immediately
As it says. No matter if this is a desktop, laptop, router, firewall. A local process sending packets passes its packets to the OUTPUT chain immediately. As a side note, the title of the link is How Packets Traverse The Filters. It could be you are confused because there is an indirect interaction between packets in the OUTPUT and INPUT chains. If a process is sending packets to www.archlinux.org in the OUTPUT chain, but blocks www.archlinux.org in the INPUT chain, it could have thought its packets didn't went out. Which is not the case. The fact that the process blocks the replies doesn't necessarily prove www.archlinux.org didn't reply. Furthere more, the order of the rules matter. With pseudo rules,
www.archlinux.org drop www.archlinux.org {ESTABLISHED,RELATED} accept
has different semantics then
www.archlinux.org {ESTABLISHED,RELATED} accept www.archlinux.org drop
And the drop rule in the later example is not necessarily with no effect. It is meaningless at the former example.
Soory, What is meaningless at the former example is the accept rule. Not the drop rule. -- u34
-- u34
I guess, it is back to not understanding why blocking inbound connections would be a problem for outbound connections.
Best, Christian
[1]: https://en.wikipedia.org/wiki/Iptables#/media/File:Netfilter-packet-flow.svg
On 9/23/23 13:51, Christian wrote: ... In case of interest, the nft rules that I shared with David previously are available here [1]. This is a sample nftables ruleset for a laptop or workstation. It allows established / related packets to come back. These packets are returned after a connection is initiated from the local machine. e.g. going to a website, or sending an icmp ping. It supports local services running on same machine where nftables rules are installewd (these are services which are available to the internet): - DNS server - SSH server - WEB server for http and https including http/2 and http/3. Uncomment to turn on. It also allows blocking from a list of CIDR addresses. This prevents any IP from the blocked list any access to the above services offered on the machine. N.B. Replies from these "blocked" IPs, are still permitted to come back in if they are related/established. e.g. if you go to website hosted at a blocked IP everything should work normally. The reason this works, is that these 'blocks' are done in the 'inet' table. If you wanted to block inbound SYN and in addition block established/related - then add similar blocks for ingress hook in the 'netdev' table. Adding ingress blocks prevents any packets from those IPs from getting in - regardless if related/established. It is very early in the packet flow - see [2] for how the packets flow in nftables. ingress hook is not available in (legacy) iptables last I checked. gene [1] https://github.com/gene-git/blog/tree/master/nftables [2] https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks
On 9/23/23 12:51, Christian wrote:
I might be wrong but this thread reads like there is a misunderstanding of what the difference is between "inbound" and "INPUT". The two phrases are not the same. At the risk of mansplaining the difference, I hope this clears up some confusion.
Hey, There is nothing wrong with "mansplaning". I can read the documents, but that is no guarantee I don't come away with a incorrect understanding of the finer points of interplay between the chains. In fact I did. I was under the impression that INPUT and OUTPUT chains function was separate and entries in the INPUT chain would not prevent outbound traffic on a blocked IP or range in the INPUT chain. The subsequent messages in the thread show it's all to easy to miss a point or two. Thank you Christian, u34 and especially Genes for advancing my understanding of how the chains work together. I will redo my rule-set in a hopefully more stateful manner. It has been cobbled together to address the main ranges where intrusions attempts have come from since I installed Arch on the server in 2015 building on the default iptables.rules provided at that time -- it no-doubt has a bit of age-rot in the rule-set. Thanks again for the help. Now if we could just get Redis to play-nice with Nextcloud, life would be great :) -- David C. Rankin, J.D.,P.E.
On 9/24/23 02:52, David C. Rankin wrote:
On 9/23/23 12:51, Christian wrote:
In addition to the workstation (single interface) nftables example, I have just uploaded an example of nftables firewall rules. i.e. for a router with 2 interfaces that sits between the internet and internal network. This supports services provided by firewall itself (DNS or ssh etc) as well as forwarded services to servers on internal network (web server, ssh, vpn etc). It has blocks and whitelist - and includes both inet and netdev blocks. I hand edited a fully working firewall for this example and hope it's useful. After edits, before trying please confirm no typos etc by running check: nft -c nftables.conf The nftables rules and sample files containing sets of CIDR blocks for whitelist or blocks are included. Obviously these will need editing. The set files are designed to be easily generated from a script - after any changes to the sets, reload the rules to pick up the new set data. It's available in my gh blog area in the nftables/firewall directory: https://github.com/gene-git/blog/tree/master/nftables Hope you find this helpful. And if you find typos or boo boos please let me know! thanks gene
On 9/24/23 07:22, Genes Lists wrote:
nft -c nftables.conf
typo - should be: nft -c -f nftables.conf gene
participants (6)
-
Christian
-
David C. Rankin
-
Genes Lists
-
Lone_Wolf
-
Simon Perry
-
u34＠net9.cf