[arch-devops] Centralized log monitoring and alerting?

[Bartłomiej Piotrowski]

On 10/09/2018 02.53, Andrew Crerar wrote:
> Hi all,
> 
> On 2018-07-28 there were some discussions in #archlinux-devops around setting up
> some sort of centralized logging/monitoring/alerting solution for the various
> services on Apollo (and maybe other?) server(s). I had mentioned possibly using
> the ELK[1] stack for this task. There was some back and forth about it
> potentially being a bit heavy handed for what was needed and how we would most
> likely need to repurpose/dedicate something like nymeria to handle the stack.
> There was also the suggestion of possibly using something like tenshi[2] if
> we're aiming for a low overhead solution, however, there would be much writing
> of the regexes.
> 
> With that being said, the purpose of this email is to have a more formal
> discussion around what we're trying to capture from the logs, the actions we
> want to have taken with what ends up being captured, and possibly come to a
> consensus on what tool(s) we could leverage.
> 
> Thoughts?
> 
> Regards,
> 
> Andrew
> 
> 
> [1] https://www.elastic.co/de/elk-stack
> [2]https://github.com/inversepath/tenshi
> 

We already have centralized alerting with Zabbix. What are you trying to
solve exactly? I don't recall anyone from devops team complaining about
having to ssh to run journalctl somewhere. Each new gear added to infra
means more time spent on maintaining it, while our goal is quite the
opposite thing.

Bartłomiej