I've finally started setting up a monitoring solution (zabbix). Thanks to Bartłomiej for getting zabbix packages into our repos!
If anyone from the devops team wants accounts on zabbix, please send me a mail.
Right now it doesn't monitor too much (items and hosts), but I plan to increase coverage over time. I'll likely also set up grafana later for pretty graphs.