[arch-general] HTTP spam from China - CIDR compacting tool

[Juha Kankare]

On 26/02/2019 23:01, brent s. wrote:
> On 2/26/19 1:20 PM, Genes Lists via arch-general wrote:
>> On 2/26/19 1:13 PM, Juha Kankare via arch-general wrote:
>>> On 26/02/2019 20:11, Genes Lists via arch-general wrote:
>> ...
>>> My current script is just pulling cn.zone from ipdeny.com. This looks
>>> super useful, I'm saving it. Thank you dude!
>>>
>> You're welcome.
>>
>> I just ran it on cn.zone and it reduces the number of lines from 8,337
>> to 5,120. It can make a significant difference.
>>
>> best,
>>
>> gene
>>
> Just to +1 what Gene has said, I've taken similar approaches to
> compacting into CIDRs and it really does make a significant difference.
>
> For clarification on his ipset[0] point, I also have to strongly
> recommend it. It not only *greatly* simplifies your ruleset, but it can
> be dynamically altered without needing to reload your firewall rules.
>
> e.g. assuming you have an IP set named "china_ips",
>
> -A INPUT -m set --match-set china_ips src -p tcp -m tcp --dport 80 -j DROP
>
> will drop traffic for all those entries. You've then simplified many
> (MANY) rules to one. :)
>
> You can (Gene, you may find this particularly useful since you feed to
> ipset) use the pyroute2.IPSet() function to actually manage the live
> kernel's ipsets as well. Make sure your running kernel and latest
> installed kernel match, otherwise you'll need to reboot so the ipset
> kernel module can be loaded.
>
> Untested, but should be pretty darn close if not functional:
>
> #####
> import subprocess
> import pyroute2
>
>
> # (...)
> ipset = pyroute2.IPSet()
> setsfile = '/etc/ipset.conf'
> setname = 'china_ips'
> tmpset = '{0}_TMP'.format(setname)
>
> set_exists = False
> try:
>      # Check to see if the list exists.
>      ipset.headers(setname)
>      # list is done here as a quick-and-dirty sanity/exception check,
>      # which is why it's in both the try and exception.
>      setlist = ipset.list(name = setname)
>      set_exists = True
> except pyroute2.ipset._IPSetError:
>      ipset.create(setname, stype = 'hash:net')
>      setlist = ipset.list(name = setname)
>
> # We use a temporary set so we don't affect any current iptables
> # processing. Most likely unnecessary, but better safe than sorry.
> try:
>      ipset.destroy(name = tmpset)
> except pyroute2.ipset._IPSetError:
>      # It doesn't exist (yet), which is what we want.
>      pass
>
> # Create the temporary set
> ipset.create(tmpset, stype = 'hash:net')
> for n in set1.iter_cidrs():  # "set1" is from Gene's script
>      ipset.add(tmpset, n)
>
> # Make the temporary set live
> ipset.swap(setname, tmpset)
>
> # And cleanup the now-unnecessary tmpset
> ipset.destroy(name = tmpset)
>
> # Save them to the persistent file so it's applied on a reboot.
> # Remember to "systemctl enable ipset.service".
> # Unfortunately, there isn't a built-in save function.
> # You could easily write your own iterator/generator, though,
> # if you want to avoid a subprocess call.
> # The syntax is pretty simple.
> with open(setsfile, 'w') as f:
>      ipset_cfg = subprocess.run(['/usr/bin/ipset',
>                                  'save'],
>                                 stdout = f)
> # DONE.
> #####
>
>
>
>
>
> [0] https://wiki.archlinux.org/index.php/Ipset

Yes, I wrote a portable shellscript to do ipset already. It literally 
just blocks china, should run on research unix from the 70's as long as 
it had ipset, iptables, and wget, and in general it has beautiful 
looking output with info in case the user really chooses to do some 
unrecommended things. You too, should check it out: 
https://bbs.archlinux.org/viewtopic.php?pid=1833895#p1833895, it's 
pretty neat.

-- 
Regards, Juha Kankare