[This is a comment to arch-dev-public thread "Use systemd timers instead of /etc/cron.{hourly, daily, weekly, monthly}?". As I was unable to post to that list, I am following instructions provided by the list server by sending this message to arch-general and the author of the original thread. I'd like to mention that I also tried and was not able to file a bug report on this issue, due to bug tracker server failing to send a registration code. The original thread can be viewed here: http://comments.gmane.org/gmane.linux.arch.devel/20644 ]
I am sorry to say that the decision to replace cron.daily tasks with systemd timers is causing problems. After a routine update I noticed that my machines now perform daily maintenance tasks exactly at midnight. Not only is this time not optimal (too early), but all machines perform their maintenance at the same moment, which is far from ideal, especially for servers. Previously I had each server to perform daily cron jobs at various times, spread between 3 and 6am. On my machines this affects updatedb, man-db, logrotate and shadow, updatedb generating a lot of I/O and thus being the most problematic. I was not able to find any way to configure systemd to fire daily calendar timers at a different time of the day (please correct me if I missed something). So far I found two workarounds:
1. Override timer files and set OnCalendar with a specific time, rather than "daily". This has to be done separately for each timer. 2. Restore cron.daily scripts and mask relevant systemd services and timers (i.e. revert the configuration to what it was before the update). The resulting config is simpler to manage than the first workaround (no separate time settings for each task), so I went with this one.
I am not sure what to suggest as a general fix for this issue, other than a feature request to systemd upstream maintainers. I guess it is too late to roll back the changes in affected Arch packages. I am posting this for other users looking for a solution to this problem.
Thanks Maciej Puzio
Am 24.04.2014 17:20, schrieb Maciej Puzio:
I am sorry to say that the decision to replace cron.daily tasks with systemd timers is causing problems. After a routine update I noticed that my machines now perform daily maintenance tasks exactly at midnight. Not only is this time not optimal (too early), but all machines perform their maintenance at the same moment, which is far from ideal, especially for servers. Previously I had each server to perform daily cron jobs at various times, spread between 3 and 6am. On my machines this affects updatedb, man-db, logrotate and shadow, updatedb generating a lot of I/O and thus being the most problematic. I was not able to find any way to configure systemd to fire daily calendar timers at a different time of the day (please correct me if I missed something). So far I found two workarounds:
This is supposed to be handled by the AccuracySec=, which is supposed to schedule timers at random times within the specified interval (in our case, 12h). I have no idea why this doesn't work, it's definitely worth looking into.
- Override timer files and set OnCalendar with a specific time,
rather than "daily". This has to be done separately for each timer.
Yes, bad solution. As I said above, we should try to figure out why AccurarySec= doesn't do anything first.
- Restore cron.daily scripts and mask relevant systemd services and
timers (i.e. revert the configuration to what it was before the update). The resulting config is simpler to manage than the first workaround (no separate time settings for each task), so I went with this one.
This shouldn't be a permanent solution.
This is supposed to be handled by the AccuracySec=, which is supposed to schedule timers at random times within the specified interval (in our case, 12h). I have no idea why this doesn't work, it's definitely worth looking into.
So far, I have seen all timers elapsing around midnight. But once I saw that, I quickly reverted to cron setup. One machine hasn't been updated yet, I will do it next week and perform more testing.
But anyway, even if timers were spread between midnight and noon, this is hardly ideal either. Servers running maintenance tasks at unpredictable times, including work hours, is not really what I want to see. I can live with randomness or without it, but I would like more control over when things are happening, as had been possible with cron.
I submitted request for enhancement to systemd bug tracker. The link is: https://bugs.freedesktop.org/show_bug.cgi?id=77938
However, now I started to worry about AccuracySec=12h. Even if systemd maintainers implement a way to customize timer elapse time (as I proposed in RFE), the long period in AccuracySec will then become a problem.
Thanks Maciej Puzio
It seems to me that, compared to cron/anacron, systemd timers currently offer reduced functionality and no clear benefit (I don't consider the act of uninstalling cron to have significant value in itself). Losing the ability to control when jobs run is giving up a lot; without that, the more use is made of systemd timers, the more unpredictable the results will be.
Perhaps I'm missing something, because I don't fully understand all the points Thomas raised last month in his proposal on arch-dev-public:
Pros:
- enabled by default (in contrast to cronie)
- systems without need for crontabs can disable/uninstall cron
- service will be simpler than the rather long dropin scripts
Cons:
- services are run in parallel instead of sequentially (is this even a con? timer start will be randomized, and we can increase accuracy to an hour to randomize even more)
- no holdoff time after boot as it seems
pro#3 is a mystery to me, and missing from the "con" list is the point Maciej raised about the effect random timer jobs can have on busy servers.
If this ship is already sailing, though, the goal should be to add functionality to systemd timers so that it becomes a real replacement for cron, e.g. allow execution windows to be defined, jobs placed within these windows, and scheduling parameters like acceptable degrees of parallelism to be specified. Something like that could be more powerful and flexible than cron, and potentially easier to manage than a set of crontabs. Does anyone know if the systemd timers roadmap includes anything like this? Carl
On Fri, 25 Apr 2014 17:19:04 -0400 Carl Schaefer schaefer@trilug.org wrote:
It seems to me that, compared to cron/anacron, systemd timers currently offer reduced functionality and no clear benefit (I don't consider the act of uninstalling cron to have significant value in itself).
But then again, you don't maintain the cron package...
Losing the ability to control when jobs run is giving up a lot; without that, the more use is made of systemd timers, the more unpredictable the results will be.
See systemd.timer(5) manpage. The timer configuration _should be_ one-to-one equivalent to a crontab, except for a different syntax...
Perhaps I'm missing something, because I don't fully understand all the points Thomas raised last month in his proposal on arch-dev-public:
Pros:
- enabled by default (in contrast to cronie)
- systems without need for crontabs can disable/uninstall cron
- service will be simpler than the rather long dropin scripts
Cons:
- services are run in parallel instead of sequentially (is this even a con? timer start will be randomized, and we can increase accuracy to an hour to randomize even more)
- no holdoff time after boot as it seems
pro#3 is a mystery to me,
Just compare {man-db,logrotate,updatedb}.service files with the corresponding scripts in cron.daily...
and missing from the "con" list is the point Maciej raised about the effect random timer jobs can have on busy servers.
So far, I have seen only (minor) inconveniences associated with switching to the timers, which essentially have to do with system(d) startup time. Notice that these issues will not be visible on servers because there systemd's parallellization of the boot process is irrelevant.
The timer jobs are niced to 19 so if the server is busy they should wait, no?
If this ship is already sailing, though, the goal should be to add functionality to systemd timers so that it becomes a real replacement for cron, e.g. allow execution windows to be defined, jobs placed within these windows, and scheduling parameters like acceptable degrees of parallelism to be specified. Something like that could be more powerful and flexible than cron, and potentially easier to manage than a set of crontabs. Does anyone know if the systemd timers roadmap includes anything like this?
Ask this on [systemd-devel].
Cheers, L.
Losing the ability to control when jobs run is giving up a lot; without that, the more use is made of systemd timers, the more unpredictable the results will be.
See systemd.timer(5) manpage. The timer configuration _should be_ one-to-one equivalent to a crontab, except for a different syntax...
thanks, the description of the OnCalendar syntax is actually in systemd.time(7), and I'd missed it. It does look like an improvement over the crontab file syntax in readability, at least.
- service will be simpler than the rather long dropin scripts
Just compare {man-db,logrotate,updatedb}.service files with the corresponding scripts in cron.daily...
if there's a complex script in cron.daily, I'd expect the service file would have to invoke a similar script to do the same work.
and missing from the "con" list is the point Maciej raised about the effect random timer jobs can have on busy servers.
The timer jobs are niced to 19 so if the server is busy they should wait, no?
that sounds like a reasonable assumption, but based on the problem report it may not be true.
thanks! Carl
I have been testing the issue for a week. Daily timers are fired between 0:00 and 0:01 without exception - all timers at the same time, all machines at the same time, every day at the same time. The largest variation I have seen was 30 seconds. So yes, there is definitely an issue with AccuracySec=12h not being honored.
However, whether timer accuracy is 30 seconds or 12 hours, this makes little difference to me, as both are unacceptable without the possibility to customize timer elapse time. I have reverted all my Arch machines to previous cron-based config and intend to keep it this way. Perhaps it is not "cool", but at least it works.
Which brings me to what I consider the most important problem here. Having read the original thread on arch-dev-public I was left with an impression that perceived "coolness" of the new setup took precedence over consideration of its impact. As a result, modification has been released without sufficient testing, as we can clearly see now. It did not help that discussion was carried on a mailing list that does not allow user comments. Perhaps a larger audience would have allowed a consideration of alternative solutions. An example of such a solution would be hourly/daily/monthly/weekly timers that execute scripts from relevant /etc/cron.* directories. That would allow for removal of cronie while sidestepping timer elapse problems that we are discussing here. It would also have a benefit of handling all cron tasks in addition to logrotate/updatedb/man-db/shadow.
Thanks Maciej Puzio
On 05-05-14 15:05, Maciej Puzio wrote:
Perhaps a larger audience would have allowed a consideration of alternative solutions. An example of such a solution would be hourly/daily/monthly/weekly timers that execute scripts from relevant /etc/cron.* directories. That would allow for removal of cronie while sidestepping timer elapse problems that we are discussing here. It would also have a benefit of handling all cron tasks in addition to logrotate/updatedb/man-db/shadow.
Thanks Maciej Puzio
That sounds like a good idea. Instead of removing the entire cron system, this would allow systemd-timers to function as an alternative to cron. it would also reduce the increasing dependency archlinux has on systemd as init system.
Lone_Wolf
Am 05.05.2014 15:05, schrieb Maciej Puzio:
I have been testing the issue for a week. Daily timers are fired between 0:00 and 0:01 without exception - all timers at the same time, all machines at the same time, every day at the same time. The largest variation I have seen was 30 seconds. So yes, there is definitely an issue with AccuracySec=12h not being honored.
However, whether timer accuracy is 30 seconds or 12 hours, this makes little difference to me, as both are unacceptable without the possibility to customize timer elapse time.
Of course you can configure the elapse time, by adjusting the associated .timer unit. With anacron, there was no way to adjust the time, so this is a step forward.
Hi,
On Mon, 5 May 2014 08:05:08 -0500 Maciej Puzio mx34567@gmail.com wrote:
I have been testing the issue for a week. Daily timers are fired between 0:00 and 0:01 without exception - all timers at the same time, all machines at the same time, every day at the same time. The largest variation I have seen was 30 seconds. So yes, there is definitely an issue with AccuracySec=12h not being honored.
AFAIU systemd is supposed to start timers "randomly" in the time interval [1d, 1d + 12h]; different timers are started in parallel. Are you arguing that the starting time is not "random" enough?
However, whether timer accuracy is 30 seconds or 12 hours, this makes little difference to me, as both are unacceptable without the possibility to customize timer elapse time. I have reverted all my Arch machines to previous cron-based config and intend to keep it this way. Perhaps it is not "cool", but at least it works.
You misunderstood the point here.
Systemd timers (at least in the current form) are _not_ cron replacement. However, they are adequate for daily maintainance jobs that are shipped with packages. If you had custom, carefully scheduled cron jobs, you should continue using cronie. What I don't understand is why do you care when man-db/updatedb runs?
An example of such a solution would be hourly/daily/monthly/weekly timers that execute scripts from relevant /etc/cron.* directories. That would allow for removal of cronie while sidestepping timer elapse problems that we are discussing here. It would also have a benefit of handling all cron tasks in addition to logrotate/updatedb/man-db/shadow.
The scripts mainly set up the environment which is now done by systemd. You issue is with scheduling, and it will _not_ go away because scripts are still executed by systemd (as opposed to cronie).
Cheers,
Thomas Bächler wrote:
Of course you can configure the elapse time, by adjusting the associated .timer unit. With anacron, there was no way to adjust the time, so this is a step forward.
As I wrote before, I can edit every timer file and set the elapse time. What I can't do is to change one setting which says when daily maintenance tasks are run. This was possible with cron, but is no longer possible now. What's the problem to edit four files? Well, this is multiplied by the number of machines that are under my care. Again, here is relevant systemd RFE link: https://bugs.freedesktop.org/show_bug.cgi?id=77938
On Mon, May 5, 2014 at 12:21 PM, Leonid Isaev lisaev@umail.iu.edu wrote:
AFAIU systemd is supposed to start timers "randomly" in the time interval [1d, 1d + 12h]; different timers are started in parallel. Are you arguing that the starting time is not "random" enough?
I am not arguing on what systemd is supposed to do, but describing what it is doing in practice. Testing shows that there is a clear discrepancy between the former and the latter, possibly a result of a bug. From what I have seen, there is no randomness in timer elapse time at all.
Systemd timers (at least in the current form) are _not_ cron replacement. However, they are adequate for daily maintainance jobs that are shipped with packages. If you had custom, carefully scheduled cron jobs, you should continue using cronie. What I don't understand is why do you care when man-db/updatedb runs?
Updatedb is I/O intensive and should run at time when servers are least used. I am not really the first person to bump into this issue; maintenance tasks were by default scheduled to run between 4 and 7am since the dawn of Linux. That is - until now. In Arch they now run at midnight.
The scripts mainly set up the environment which is now done by systemd. You issue is with scheduling, and it will _not_ go away because scripts are still executed by systemd (as opposed to cronie).
No, my main issue is system maintainability. I find it unacceptable to separately schedule every single maintenance script on every machine. With cron I could just schedule when daily jobs were run - that was one setting.
On Tue, 6 May 2014 13:23:26 -0500 Maciej Puzio mx34567@gmail.com wrote:
As I wrote before, I can edit every timer file and set the elapse time. What I can't do is to change one setting which says when daily maintenance tasks are run. This was possible with cron, but is no longer possible now. What's the problem to edit four files? Well, this is multiplied by the number of machines that are under my care. Again, here is relevant systemd RFE link: https://bugs.freedesktop.org/show_bug.cgi?id=77938
This is bogus.
See "man 7 systemd.time"; what you want is something like "OnCalendar=*-*-* 04:10:20" for a job that runs every day at 4:10:20 AM.
On Mon, May 5, 2014 at 12:21 PM, Leonid Isaev lisaev@umail.iu.edu wrote:
Systemd timers (at least in the current form) are _not_ cron replacement. However, they are adequate for daily maintainance jobs that are shipped with packages. If you had custom, carefully scheduled cron jobs, you should continue using cronie. What I don't understand is why do you care when man-db/updatedb runs?
After re-reading the documentation I have to take this back, systemd timers seem to implement all features provided by cronie.
Cheers,
On 06/05/14 04:13 PM, Leonid Isaev wrote:
After re-reading the documentation I have to take this back, systemd timers seem to implement all features provided by cronie.
AFAIK, the only notable missing feature is the ability for non-root users to run jobs when they're not logged in. This is intentionally left out, and it's not hard for an administrator to replicate the feature with a timer unit for a specific user allowed to run jobs.
(of course, this isn't really the topic at hand :P)
(Tue, May 06, 2014 at 04:28:48PM -0400) Daniel Micay :
On 06/05/14 04:13 PM, Leonid Isaev wrote:
After re-reading the documentation I have to take this back, systemd timers seem to implement all features provided by cronie.
AFAIK, the only notable missing feature is the ability for non-root users to run jobs when they're not logged in. This is intentionally left out, and it's not hard for an administrator to replicate the feature with a timer unit for a specific user allowed to run jobs.
Note that loginctl enable-linger <user> launches the systemd of <user> even when he is not there
On Tue, May 6, 2014 at 3:13 PM, Leonid Isaev lisaev@umail.iu.edu wrote:
On Tue, 6 May 2014 13:23:26 -0500 Maciej Puzio mx34567@gmail.com wrote:
As I wrote before, I can edit every timer file and set the elapse time. What I can't do is to change one setting which says when daily maintenance tasks are run. This was possible with cron, but is no longer possible now. What's the problem to edit four files? Well, this is multiplied by the number of machines that are under my care. Again, here is relevant systemd RFE link: https://bugs.freedesktop.org/show_bug.cgi?id=77938
This is bogus.
Leonid, please reread the above paragraph and my previous posts, with comprehension. With all due respect, your replies are not related to the issue discussed.
Hi,
On Wed, 7 May 2014 13:34:29 -0500 Maciej Puzio mx34567@gmail.com wrote:
On Tue, May 6, 2014 at 3:13 PM, Leonid Isaev lisaev@umail.iu.edu wrote:
On Tue, 6 May 2014 13:23:26 -0500 Maciej Puzio mx34567@gmail.com wrote:
As I wrote before, I can edit every timer file and set the elapse time. What I can't do is to change one setting which says when daily maintenance tasks are run. This was possible with cron, but is no
You mean, you could change /etc/anacrontab?
longer possible now. What's the problem to edit four files? Well, this is multiplied by the number of machines that are under my care.
Why can't you put additional configs in /etc/systemd/system/xxx.timer.d/ dir?
Again, here is relevant systemd RFE link: https://bugs.freedesktop.org/show_bug.cgi?id=77938
This is bogus.
Leonid, please reread the above paragraph and my previous posts, with comprehension. With all due respect, your replies are not related to the issue discussed.
It is completely unclear what you want to be fixed and the title is misleading...
If you have machines m_1 ... m_N and want to spread the anacron jobs, you need to edit N anacrontabs, right? Now you need to do k*N changes, k -- the number of jobs used to be started by anacron. Are you asking for a centralized control affecting all "daily" timers like the per-machine anacrontab? If true, this is unlikely to be implemented (at least I wouldn't do it).
Perhaps a proper approach is to create a special target for your maintainance jobs which would pull all relevant services and would itself be triggered by an OnCalendar timer. Although I am not sure whether a timer can directly trigger a target yet...
Cheers,
On Thu, May 8, 2014 at 6:23 AM, Leonid Isaev lisaev@umail.iu.edu wrote:
Perhaps a proper approach is to create a special target for your maintainance jobs which would pull all relevant services and would itself be triggered by an OnCalendar timer. Although I am not sure whether a timer can directly trigger a target yet...
They can, do refer to https://wiki.archlinux.org/index.php/Systemd/cron_functionality where you can find some examples of the same.
On Thu, 8 May 2014 14:24:45 +0900 Savyasachee Jha savya.jha91@gmail.com wrote:
On Thu, May 8, 2014 at 6:23 AM, Leonid Isaev lisaev@umail.iu.edu wrote:
Perhaps a proper approach is to create a special target for your maintainance jobs which would pull all relevant services and would itself be triggered by an OnCalendar timer. Although I am not sure whether a timer can directly trigger a target yet...
They can, do refer to https://wiki.archlinux.org/index.php/Systemd/cron_functionality where you can find some examples of the same.
Cool. I didn't know about that wikipage.
Thanks,
On Thu, May 8, 2014 at 12:24 AM, Savyasachee Jha savya.jha91@gmail.com wrote:
They can, do refer to https://wiki.archlinux.org/index.php/Systemd/cron_functionality where you can find some examples of the same.
Thank you for the link, it is quite interesting. A rather verbose solution (in sheer number of files), but it does what I had in mind. Hopefully, as more distros start to migrate away from cron, systemd maintainers will simplify and standardize the configuration needed for cron replacement.
Maciej Puzio
On 05/05/14 09:05 AM, Maciej Puzio wrote:
I have been testing the issue for a week. Daily timers are fired between 0:00 and 0:01 without exception - all timers at the same time, all machines at the same time, every day at the same time. The largest variation I have seen was 30 seconds. So yes, there is definitely an issue with AccuracySec=12h not being honored.
However, whether timer accuracy is 30 seconds or 12 hours, this makes little difference to me, as both are unacceptable without the possibility to customize timer elapse time. I have reverted all my Arch machines to previous cron-based config and intend to keep it this way. Perhaps it is not "cool", but at least it works.
Which brings me to what I consider the most important problem here. Having read the original thread on arch-dev-public I was left with an impression that perceived "coolness" of the new setup took precedence over consideration of its impact. As a result, modification has been released without sufficient testing, as we can clearly see now. It did not help that discussion was carried on a mailing list that does not allow user comments. Perhaps a larger audience would have allowed a consideration of alternative solutions. An example of such a solution would be hourly/daily/monthly/weekly timers that execute scripts from relevant /etc/cron.* directories. That would allow for removal of cronie while sidestepping timer elapse problems that we are discussing here. It would also have a benefit of handling all cron tasks in addition to logrotate/updatedb/man-db/shadow.
Thanks Maciej Puzio
cronie was not enabled by default before and these systemd timers are, so there has been an improvement in the default configuration.
The issue of systemd not spreading out the timers enough can be fixed, because if true it's a bug.
arch-general@lists.archlinux.org