[arch-devops] fail2ban deployed for the wiki

[Jelle van der Waa]

Hi All,

Today and earlier we suffered a huge influx of bots from china crawling
the wiki. We have taken a few measures to reduce the impact:

- fastcgi_cache for /load.php which caches the load.php page for 10
  minutes which contains assets such as css/js. This should offload
  php-fpm which was overloaded.
- Enabled a plugin to disable viewing wiki page revisions for anonymous
  users this reduces some heavy requests and hopefully the impact of
  bots

The last step was enabling fail2ban for HTTP/HTTPS requests, only for
the wiki now. It blocks every ip doing more then 300 requests in 30
minutes. This might be a bit too aggressive but for now it dropped our
load from ~ 20 -> ~ 2/3 and blocks 85 ips. This can be tweaked later,
maybe it should be 400/500?

To view the blocked ips execute:

fail2ban-client status wiki-nginx-dos

To unban a valid IP:

fail2ban-client unban $ip


fail2ban does use a lot of CPU which we should look into tuning, but
maybe it will get better over time when the log files are smaller due to
less bots coming through :-)


The fail2ban role is in ansible, but not suited yet to be re-used on
other hosts.

Greetings,

Jelle