Today and earlier we suffered a huge influx of bots from china crawling the wiki. We have taken a few measures to reduce the impact:
- fastcgi_cache for /load.php which caches the load.php page for 10 minutes which contains assets such as css/js. This should offload php-fpm which was overloaded. - Enabled a plugin to disable viewing wiki page revisions for anonymous users this reduces some heavy requests and hopefully the impact of bots
The last step was enabling fail2ban for HTTP/HTTPS requests, only for the wiki now. It blocks every ip doing more then 300 requests in 30 minutes. This might be a bit too aggressive but for now it dropped our load from ~ 20 -> ~ 2/3 and blocks 85 ips. This can be tweaked later, maybe it should be 400/500?
To view the blocked ips execute:
fail2ban-client status wiki-nginx-dos
To unban a valid IP:
fail2ban-client unban $ip
fail2ban does use a lot of CPU which we should look into tuning, but maybe it will get better over time when the log files are smaller due to less bots coming through :-)
The fail2ban role is in ansible, but not suited yet to be re-used on other hosts.