[arch-devops] Centralized log monitoring and alerting?

[Giancarlo Razzolini]

Em setembro 10, 2018 6:06 Florian Pritz via arch-devops escreveu:
> 
> Personally I use tenshi on my servers and while it's not ideal, it works
> fairly well. It does require regular maintenance though since sometimes
> log messages change with updates. One problem I worry about is that I
> might filter interesting messages by accident, especially if the formats
> change and an old regex now matches some new message that I actually
> didn't want it match. Also, after a while, the regex list can become quite
> long. You can split it in files and group them nicely, but I rarely
> actually bothered to review them and remove old stuff so it mostly just
> grows. Also to put things in perspective, my current personal rules are
> between 3 to 10 for most small things, 10-20 for "normal" services and
> around 40-70 for dovecot, amavis and postfix.
>

Never used tenshi, but I guess that, after we write the rules for one webapp,
the others will be similar and adaptable.

> Another issue I have with using tenshi for us is that I'm conflicted
> about publishing the config we use. I'm worried that an attacker might
> look at the config and try to stay under the radar and within any
> alerting limits we set. Then again, there are probably easier ways to
> attack us. Any opinions here are welcome.

This is a perfect example of security through obscurity that might actually
make the life of the attacker slightly harder. I personally wouldn't loose
sleep about this, but we can put on a vault, like Thore suggested.

> 
> I haven't used other solutions so far so I welcome a discussion about
> this. In general I think log monitoring could help us reduce future work
> load and make things more predictable, but yeah, it requires some
> investment at the beginning and some maintenance.
> 

Since log monitoring has an intersection with security, perhaps we should include
someone from the security team to weigh in on this as well? I don't think this is
something we want to keep only on arch-devops. Having said that, my personal opinion
is that log monitoring is orders of magnitude more important for security than it is
for detecting actual problems on the applications. Because those problems will most
likely trigger 50x errors and/or be reported by users. Obviously, some errors might
give hints of what is to come, but if we don't act on them, the result is the same.

So, in essence, log monitoring will give us security insights, and might help us have
a more proactive instance to problems that zabbix currently don't/can't detect. I think
it's worth taking some time to invest in this. But I still want to hear what the security
guys have to say.

Regards,
Giancarlo Razzolini

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 870 bytes
Desc: not available
URL: <https://lists.archlinux.org/pipermail/arch-devops/attachments/20180911/645f5c34/attachment.sig>