On Thursday 13 Feb 2014 13:35:59 Thomas Bächler wrote:

Am 13.02.2014 13:04, schrieb Paul Gideon Dann:

...

Does anyone know of any standard system for receiving notifications from systemd for unit state changes? I currently use Monit for the monitoring of many processes, and it'll e-mail me when things happen (e.g. a process was restarted). Since switching to systemd, it's felt a bit silly that for several processes, I'm having Monit monitor them simply because systemd is unable to tell me it restarted a unit. Monit isn't actually required to keep those processes alive as it once was, because systemd can do that.

I'd place a bet on the systemd dbus API: IIRC, it exports the state of each unit as a property and then emits the standard org.freedesktop.DBus.Properties.PropertiesChanged signal when the state changes.

So, your task would be to subscribe to that signal and act on it. This could be nicely done in python (and maybe someone has done it already).

Agreed, but I'm baffled as to why there isn't already a well-known tool. To be honest, I'd have expected it to be important enough to be produced along-side the systemd project, probably with several backends for different notification systems. Paul

Am 13.02.2014 16:05, schrieb Rodrigo Rivas:

Ok... I'll take the chance to practice my DBus abilities... It is a bit long, but it kind of works. Just replace the print() call with your favourite sendmail function and you'll get a notification every time any of the units specified in the command line changes status.

This isn't too long and it actually seems to work fine. I guess what Paul actually wants is a system where you would subscribe to all services, not just some of them. This should be possible as well with the API.

On Thursday 13 Feb 2014 16:11:56 Thomas Bächler wrote:

Am 13.02.2014 16:05, schrieb Rodrigo Rivas:

...

Ok... I'll take the chance to practice my DBus abilities... It is a bit long, but it kind of works. Just replace the print() call with your favourite sendmail function and you'll get a notification every time any of the units specified in the command line changes status.

This isn't too long and it actually seems to work fine.

I guess what Paul actually wants is a system where you would subscribe to all services, not just some of them. This should be possible as well with the API.

Yeah, though actually I'm just really surprised that, given the incredible administrative benefits of systemd, there isn't currently anything that leverages it for actual process monitoring and reporting. As far as I can tell, systemd is also not yet able to automatically restart bloated or stale services (e.g. worker instances that may go haywire). Hopefully these things will come along now that "systemd --user" is maturing. I'm actually pretty excited about user-systemd. I'm implementing monitoring for long- running Ruby processes for a Rails app, and although the go-to tool for this in the Ruby world is "god", I've found it really icky and brittle, and really wish I could replace it with systemd, but for now I will make my peace. Paul

On Thursday 13 Feb 2014 13:36:35 Denis A. Altoé Falqueto wrote:

On Thu, Feb 13, 2014 at 1:29 PM, Paul Gideon Dann wrote:

...

Yeah, though actually I'm just really surprised that, given the incredible administrative benefits of systemd, there isn't currently anything that leverages it for actual process monitoring and reporting. As far as I can tell, systemd is also not yet able to automatically restart bloated or stale services (e.g. worker instances that may go haywire). Hopefully these things will come along now that "systemd --user" is maturing. I think you can rely on software or hardware watchdogs, which are supported by systemd.

Yeah, I think it's possible to get systemd to poll a script, or there's always cron (or a timer unit) that should allow us to manually inspect a process and restart it if necessary. But it would be cooler if there were shortcuts to features that we see in Monit and other similar systems; something like this in a unit file: MaxMemoryThreshold=100M MaxMemoryCheckInterval=30 MaxMemoryIntervalThrehold=2 The memory is then checked every 30 seconds. When the unit exceeds this amount of RAM for 2 successive intervals, the unit is restarted. Paul

On Thursday 13 Feb 2014 17:58:05 Damjan Georgievski wrote:

...

Yeah, I think it's possible to get systemd to poll a script, or there's always cron (or a timer unit) that should allow us to manually inspect a process and restart it if necessary. But it would be cooler if there were shortcuts to features that we see in Monit and other similar systems; something like this in a unit file:

MaxMemoryThreshold=100M MaxMemoryCheckInterval=30 MaxMemoryIntervalThrehold=2

The memory is then checked every 30 seconds. When the unit exceeds this amount of RAM for 2 successive intervals, the unit is restarted.

there is setting of ulimits in systemd, it's much more brutal though, will not wait for 30 seconds

man systemd.exec

Thank you; I didn't know about that, and it's very interesting. The limits are indeed a bit strict for server process monitoring. I imagine it would be really handy as a failsafe in embedded environments, though. The great benefit of them is that (as far as I can tell), it's the kernel that enforces the limits, so there's no polling involved. I suppose that the gold standard for monitoring would be a combination of the two: a userspace component (e.g. systemd or associated monitoring tool) registering to receive events from the kernel on certain resource thresholds for a process, so that e.g. time spent above RAM threshold could be monitored accurately in an event-based way, without polling. That would be awesome. Paul

On Thursday 13 Feb 2014 16:05:05 Rodrigo Rivas wrote:

Ok... I'll take the chance to practice my DBus abilities... It is a bit long, but it kind of works. Just replace the print() call with your favourite sendmail function and you'll get a notification every time any of the units specified in the command line changes status.

HTH.

#!/usr/bin/python from gi.repository import GObject import sys import dbus from dbus.mainloop.glib import DBusGMainLoop

DBusGMainLoop(set_as_default=True)

bus = dbus.SystemBus() systemd = bus.get_object('org.freedesktop.systemd1', '/org/freedesktop/systemd1') manager = dbus.Interface(systemd, 'org.freedesktop.systemd1.Manager')

def GetPropChanged(obj, iface, propName, changes, invalids): if propName in invalids: return obj.Get(iface, propName, dbus_interface=dbus.PROPERTIES_IFACE) elif propName in changes: return changes[propName] else: return None

def OnPropChanged(unit, name, iface, changes, invalids): if iface != 'org.freedesktop.systemd1.Unit': return state = GetPropChanged(unit, iface, 'ActiveState', changes, invalids) substate = GetPropChanged(unit, iface, 'SubState', changes, invalids) if state or substate: print('Status changed', name, state, substate)

for unitName in sys.argv[1:]: unit = bus.get_object('org.freedesktop.systemd1', manager.GetUnit(unitName)) unit.connect_to_signal('PropertiesChanged', lambda a,b,c : OnPropChanged(unit, unitName, a, b, c), dbus_interface=dbus.PROPERTIES_IFACE)

loop = GObject.MainLoop() loop.run()

Thanks Rodrigo; great work. I may come back to this in future if no "official" solution ever seems forthcoming. Paul

On Thursday 13 Feb 2014 14:21:36 ushi wrote:

Am 13.02.2014 13:04, schrieb Paul Gideon Dann:

...

Hello all,

Does anyone know of any standard system for receiving notifications from systemd for unit state changes? I currently use Monit for the monitoring of many processes, and it'll e-mail me when things happen (e.g. a process was restarted). Since switching to systemd, it's felt a bit silly that for several processes, I'm having Monit monitor them simply because systemd is unable to tell me it restarted a unit. Monit isn't actually required to keep those processes alive as it once was, because systemd can do that.

Paul

Hey Paul,

Check out OnFailure=

[Unit] Description="HTTP Service" OnFailure=mail-root@http.service ...

The mail-root@.service is a generic oneshot service that mails you some stuff.

[Unit] Description="Mailer"

[Service] Type=oneshot ExecStart=/path/to/my/mail-script %i

[Install] WantedBy=multi-user.target

And your mail-script is somthing like...

#!/usr/bin/env sh

systemctl status "${1}.service" | \ mail -s "Failed Service: ${1}" admin@example.com

See: http://www.freedesktop.org/software/systemd/man/systemd.unit.html#OnFailure=

Thanks ushi, that's certainly something. It looks promising for custom unit files, but not so great for catching unexpected unit failures. I'll definitely keep that one in mind, though. Paul

On Thursday 13 Feb 2014 13:41:21 Damien Churchill wrote:

You could use unit overrides[0] to add the OnFailure to provided units. So should be able to set it up for any service you are interested in receiving notifications for, I would assume.

Yes, absolutely, but this only works for a subset of units: it wouldn't be practical to override every unit on the system in order to monit for failures. Currently, I have a cron job that runs "systemctl --failed" and mails me the output if it's not an empty list. It would be nice to have a more elegant solution than that to monitor for unexpected failures. Paul