Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 69777 - sys-process/cronbase: /etc/crontab deletion of lastrun files causes run-crons to execute more often than it should
Summary: sys-process/cronbase: /etc/crontab deletion of lastrun files causes run-crons...
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: High normal with 2 votes (vote)
Assignee: Anthony Basile
URL:
Whiteboard:
Keywords:
: 110963 197758 605574 (view as bug list)
Depends on:
Blocks: 169449
  Show dependency tree
 
Reported: 2004-11-01 15:26 UTC by Scott Beck
Modified: 2022-11-01 00:45 UTC (History)
16 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
proposed fix for run-crons. (run-crons,3.31 KB, text/plain)
2007-01-08 20:41 UTC, Peter Volkov (RETIRED)
Details
propsed fix for run-crons. (run-crons.diff,2.30 KB, text/plain)
2007-01-09 09:55 UTC, Peter Volkov (RETIRED)
Details
Add 1h to cleanup timeout for daily, weekly and monthly (cronbase-0.3.7-run-crons-dst.patch,1.24 KB, patch)
2017-10-29 11:32 UTC, Remy Blank
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Scott Beck 2004-11-01 15:26:03 UTC
Last nigh DST happened and clocks went back. This caused all cron that happen around 3am to happen twice. I'm assuming it has to do with this line:
  find ${LOCKDIR} -name cron.$BASE $TIME -exec rm {} \;

where TIME is a -cmin based on wether we are cron.hourly, cron.daily etc. Fun thing is, on our old systems (slackware) this wasn't an issue. They use cron it's self to decide when a cron should be ran. Their cronjob looks like this:

# Run hourly cron jobs at 47 minutes after the hour:
47 * * * * /usr/bin/run-parts /etc/cron.hourly 1> /dev/null
#
# Run daily cron jobs at 2:40 every day:
40 2 * * * /usr/bin/run-parts /etc/cron.daily 1> /dev/null
#
# Run weekly cron jobs at 2:30 on the first day of the week:
30 2 * * 0 /usr/bin/run-parts /etc/cron.weekly 1> /dev/null
#
# Run monthly cron jobs at 2:20 on the first day of the month:
20 2 1 * * /usr/bin/run-parts /etc/cron.monthly 1> /dev/null

I guess cron handles DST where this shell script does not

Reproducible: Always
Steps to Reproduce:
1. Wait a year
2. Have a cron at 3am
3. Watch it happen twice
Comment 1 Jason Rhinelander 2004-11-01 18:25:57 UTC
The problem is in /usr/sbin/run-crons, here:

    if [ -e ${LOCKDIR}/cron.$BASE ]
    then
        case $BASE in
            hourly)
                #>= 1 hour, 5 min -=> +65 min
                TIME="-cmin +65" ;;
            daily)
                #>= 1 day, 5 min -=> +1445 min
                TIME="-cmin +1445"  ;;
            weekly)
                #>= 1 week, 5 min -=> +10085 min
                TIME="-cmin +10085"  ;;
            monthly)
                #>= 31 days, 5 min -=> +44645 min
                TIME="-cmin +44645" ;;
        esac
        find ${LOCKDIR} -name cron.$BASE $TIME -exec rm {} \;
    fi


For the moment, just keep in mind the "daily" part:

    >= 1 day, 5 min -=> +1445 min

Now, the standard /etc/crontab has this to delete cron.daily:

1  3  * * *	root	rm -f /var/spool/cron/lastrun/cron.daily

Running at 3:01 isn't a problem as DST changes from 2AM -> 1AM, or 2AM -> 3AM - 3:01 will
always occur after the DST change.  The problem, however, comes in from the -cmin find
option:

    -cmin +1445

Normally, a "daily" cron runs at 3:10 - the cron.daily is deleted as 3:01, and 3:10 is
run-crons' first run after 3:01.

In this case, when run-crons ran at 2:10 (the *second* 2:10, 24 hours after the previous
day's 3:10), the file was older than 1d5m - it would have been 1d9m old - so it deleted
cron.daily and, since cron.daily didn't exist, ran the daily cron job.

Now, at 3:01, 25 hours after the previous day's 3:01, the /etc/crontab line to delete
cron.daily runs, and at 3:10 the cron job runs again, only an hour after the previous
run.

The exact same problem exists for weekly and monthly - the monthly would have run twice
this morning, and the weekly will run twice on this coming Saturday (in both cases
assuming the times in /etc/crontab haven't been changed).

I'm not certain on the best fix for the problem - one possibly solution is to change the
cases in run-crons from 1d5m to 1d1h5m, 1w5m to 1w1h5m, and 31d5m to 31d1h5m.

Comment 2 Aaron Walker (RETIRED) gentoo-dev 2004-11-02 03:11:13 UTC
Took me a little while to figure out why this was supposedly assigned to us, yet I wasnt getting any mail about it... cron@ is, as you can imagine an actual system account for the cron daemon :) 
Comment 3 Jakub Moc (RETIRED) gentoo-dev 2005-11-05 02:33:25 UTC
*** Bug 110963 has been marked as a duplicate of this bug. ***
Comment 4 Anton Koinov 2005-11-05 11:02:21 UTC
Jason, good catch. I was really wandering where in run-crons is the problem, and
had the same suspicion as Scott had about find ${LOCKDIR} -name cron.$BASE $TIME
-exec rm {} \;.

I was going to revert my clock back to test it, but you nailed it before that
saving me a lot of trouble :)

One note here, the first cron run happened at 2:20 standard time ("the new
time"), then we have 2:20 ST - 3:10 DT (a day earlier) > 1d5m.

I think the solution with making the check for 1d1h5m is fine. The other option
would be to make the remove statements in cron 

1  3  * * *     root    rm -f /var/spool/cron/lastrun/cron.daily
15 4  * * 6     root    rm -f /var/spool/cron/lastrun/cron.weekly
30 5  1 * *     root    rm -f /var/spool/cron/lastrun/cron.monthly

into a script that would delete the lock only if it is older than 1h5m (or even
maybe 2h) -- but should always touch it to reset the time of the lock file.

As to the weekly and monthly, these will run twice too. My weekly script ran
twice today, and the monthly one will run twice next month. Would be nice to
have this bug fixed before next month :)
Comment 5 Anton Koinov 2005-11-05 11:17:11 UTC
Correction, monthly did run twice already this month...
Comment 6 Peter Volkov (RETIRED) gentoo-dev 2007-01-08 20:41:11 UTC
Created attachment 106105 [details]
proposed fix for run-crons.

Attached script is modified version of run-crons which uses seconds since the epoch (date +%s), which are not affected by daylight saving time change, instead of time stamps. This should help to avoid running crons twice. I have not tested it much yet but seems that it works here. Any feedback highly appreciated. :)
Comment 7 SpanKY gentoo-dev 2007-01-09 00:41:54 UTC
(1) you should be posting diffs, not full files
(2) when dealing with relative dates, the preferred method is doing everything relative to UTC
Comment 8 Anton Koinov 2007-01-09 04:45:17 UTC
Since an epoch second defines a time instant, timezone doesn't matter.

But, I still think this doesn't solve the problem. It doesn't seem to be any different than the find -cmin variant -- in both cases time difference between when the cron.* file was created and the run-crons execution is correct. The cause is that the cron.* files are deleted "too early" on daylight time change by these statements, which causes the double execution:

1  3  * * *     root    rm -f /var/spool/cron/lastrun/cron.daily
15 4  * * 6     root    rm -f /var/spool/cron/lastrun/cron.weekly
30 5  1 * *     root    rm -f /var/spool/cron/lastrun/cron.monthly

So far, Jason's suggestion seems to be the best option.

Given that this issue has been open for more than two years, can we get some kind of fix implemented please?
Comment 9 Peter Volkov (RETIRED) gentoo-dev 2007-01-09 09:55:17 UTC
Created attachment 106173 [details]
propsed fix for run-crons.

(In reply to comment #7)
> (1) you should be posting diffs, not full files

Ok. Here it is, but with small modification.

> (2) when dealing with relative dates, the preferred method is doing everything
> relative to UTC 

And number of seconds since the epoch is the number the number of seconds since 1970-01-01 00:00:00 UTC. Thus relative to UTC.

(In reply to comment #8)
> But, I still think this doesn't solve the problem.
> 1  3  * * *     root    rm -f /var/spool/cron/lastrun/cron.daily
> 15 4  * * 6     root    rm -f /var/spool/cron/lastrun/cron.weekly
> 30 5  1 * *     root    rm -f /var/spool/cron/lastrun/cron.monthly

Well. Personally I could not imaging case when this entries are necessary as run-crons itself tests the time of previous execution of cron jobs. Can anybody explain why do we need them? Just start run-crons once 10 minutes and it will clean /var/spool/cron/lastrun as required.

The only problem I currently found with the previous run-crons script is that it checks exactly for 1 hour or exactly for 1 day or etc... Thus if we execute run-crons with period T that is divisible on 1 hour (thus divisible on 1 day, 1 week, 1 month) this could cause shift on T (that is scripts inside hourly executed every 1 hour + T). This is fixed and now script checks for 1 day - 15 seconds.

Such script also fixes the same problem for cron which does not take into account daylight-saving. But I think we do not have such crons int our tree. :)

This implementation is backward compatible. But to allow fluent upgrade function which converts file modification time into seconds since the epoch is required. Can anybody suggest something better then parsing ls output and feeding it to date?

To summarise. Two questions left:
1. Why do we need rm -f /var/spool/cron/lastrun/cron.* entries in crontab?
2. How in shell to convert file modification time into seconds since epoch?
Comment 10 Thilo Bangert (RETIRED) (RETIRED) gentoo-dev 2007-02-28 19:04:20 UTC
isn't this just trying to work around a bug in the respective cron implementation?

i know bcron handles DST changes correctly - for vixie-cron a patch has been available for a long time, and AFAIK we use it. fcron handles it correctly as well...

anybody know what the status for dcron is?
Comment 11 Peter Volkov (RETIRED) gentoo-dev 2007-03-11 12:25:45 UTC
Thilo, I do not think this is "workaround". In any way the sript itself checks for time it ran cron jobs last time. But script that is currently in portage has this check broken.
Comment 12 Jakub Moc (RETIRED) gentoo-dev 2007-11-01 11:54:51 UTC
*** Bug 197758 has been marked as a duplicate of this bug. ***
Comment 13 Alexander Zubkov 2007-11-01 12:16:42 UTC
> 1. Why do we need rm -f /var/spool/cron/lastrun/cron.* entries in crontab?

The idea is:
cron.hourly runs every hour at 0 minute, i.e. 0:00, 1:00, 2:00, ..., 23:00
cron.daily runs every day after 3:01 (3:10 because run-crons runs every 10 minutes this case)
cron.weekly runs every week at Saturday after 4:15 (4:20 this case)
cron.monthly runs every month first day after 5:30
And this should be so. Your solution (as I think) will make periodic crons run with hour, day, week or month interval, but not at the preferred time.
This things with rm's and run-cron's are need only to run skipped scripts when cron was not running.

PS. This bug is almost 3 years old. Will somebody fix it?
Comment 14 Allen Brooker (AllenJB) 2009-10-27 23:47:24 UTC
OK. It's attack of the 5 year old bug.

Is there a reason for the time management (which is broken) being kept in run-crons itself? Wouldn't getting rid of that and just keeping the rm entries in the crontab (which aren't affected by DST) actually fix this issue?

In fact, I don't think even those should be needed. Strip all the time management out of run-crons and just run it as frequently as desired, passing the directory to use as a parameter. For my solution, see:
http://allenjb.me.uk/?p=167

Does anybody see any issues with this solution?
Comment 15 Leho Kraav (:macmaN @lkraav) 2010-01-13 23:02:52 UTC
i'm also trying to find out why my cron.daily is running twice. is this why?
Comment 16 Florian 2010-08-06 12:32:34 UTC
Hi,

(In reply to comment #14)
> OK. It's attack of the 5 year old bug.
> 
> Is there a reason for the time management (which is broken) being kept in
> run-crons itself? Wouldn't getting rid of that and just keeping the rm entries
> in the crontab (which aren't affected by DST) actually fix this issue?
> 
> In fact, I don't think even those should be needed. Strip all the time
> management out of run-crons and just run it as frequently as desired, passing
> the directory to use as a parameter. For my solution, see:
> http://allenjb.me.uk/?p=167
> 
> Does anybody see any issues with this solution?


The original cron (daemon, crontab file and /etc/cron.d) is meant to run tasks at scheduled dates/times.
The cronbase "framework" (run-crons script and /etc/cron.{hourly,daily,weekly,monthly}) is meant to run tasks after a given amount of time has elapsed (periodic tasks).

To my understanding, your solution neglect the difference between a scheduled task and a periodic task, and it correctly relies on cron itself for all time management (since cron handle DST, it solves the current bug).

Apart from the DST bug discussed here, an important question is "what to do if computer was powered down at the time a task should have run ?"

The answer may (or not) be different for scheduled and periodic tasks. The actual cronbase has the ability to run periodic tasks ASAP on each boot. And some cron implementations can do the same for scheduled tasks (even on a task basis).

But your solution, if used with vixie-cron or similar crons, reverts to the previous behavior of not running the tasks if power was off. And that's precisely one of the issues the actual run-crons script was meant to resolve.

Of course, there is no reason to have any time management in a run-cron script, but "simply getting rid of it" is a feature regression : it breaks the actual behavior of periodic tasks (which are ran on boot if power was off).

Since DST is correctly handled by cron itself, your solution needs for example anacron, or fcron's bootrun option to preserve this behavior.

Finally, since the defect comes from the lack of "periodic tasks" feature in vixie-cron (and others), there is no issue in your solution.

A good question would be : how the portage tree may handle different "run-crons" scripts depending on the installed cron without becoming a mess to manage ?
Or : Should portage drop vixie-cron and move to another cron to solve the 6 year old bug without adding new bugs ?

Korbian
Comment 17 Michael Orlitzky gentoo-dev 2012-11-06 13:48:11 UTC
(In reply to comment #16)
> The original cron (daemon, crontab file and /etc/cron.d) is meant to run
> tasks at scheduled dates/times.
>
> The cronbase "framework" (run-crons script and
> /etc/cron.{hourly,daily,weekly,monthly}) is meant to run tasks after a given
> amount of time has elapsed (periodic tasks).

Do we really need this? I'll choose e.g. fcron if I want this behavior, or e.g. vixie-cron if I don't.

It breaks with expectations and is (apparently) bug-prone.

I'm with Allen: drop the time management from run-crons, leave the rest as it is. The DST issue will be fixed. If you need your missed jobs to run after a reboot, you're either already using fcron (why would you expect it to work otherwise?) or you can start.
Comment 18 SpanKY gentoo-dev 2015-07-22 10:21:46 UTC
(In reply to Peter Volkov from comment #9)

run-crons hasn't run on exact 1hr/1day/etc... intervals since 2002.  you're right though that there's a bit of jitter in the system, but our default crontab runs every 10 min, and we use a offset of 5 min.  i think that's fine.

however, the rm of lastrun files by /etc/crontab forces run-crons to normalize on the declared times (the opposite of time slewing).  DST shifts are not the only scenario where this bites -- if your system has been off for a long time (let's say a few days), when it comes back up (let's say at 17:30), run-crons should kick off quickly and execute hourly & daily.  but then the crontab entries will delete the hourly one at 17:59 (so hourly will be run again at 18:10) and the daily at 03:09 (so daily will be run again at 03:10).  in all cases, once the duplicate run happens, the timings stabilize at what is declared in crontab.

Alexander Zubkov in comment #13 partially described this, but i don't think we've ever made the guarantee that cron.daily runs in the middle of the nite, just that it runs "once a day".  run-crons is like a cheap anacron, and anacron does not include that guarantee (explicitly by design -- that's the entire point of anacron actually).

the history also shows we did not make guarantees.  the crontab cleanups were more of a contingency.  you can see that in the original version that was in baselayout:
https://sources.gentoo.org/sys-apps/baselayout/files/crontab

and the later unification into cronbase:
https://archives.gentoo.org/gentoo-dev/message/7fd7351ab73bfb664da7c344a7c00f40
https://sources.gentoo.org/sys-apps/dcron/files/crontab?revision=1.3

it was later changed from midnite to times that match other distros in bug 3296 and committed here:
https://sources.gentoo.org/sys-apps/dcron/files/crontab?r1=1.4&r2=1.5

which brings us to where we are today.  making run-crons work for DST jumps (once a year) ignores the more common bugs i describe above (every time you suspend/resume or cold boot).  if we delete the crontab lines and make the run-crons time checks more stable, it'd fix the double run, but it would change the periodic start time for each class, and make it non-deterministic (it'd be "sticky" to whenever it resumed).

the way out of this tunnel imo is:
(1) delete the /etc/crontab entries that delete the lastrun entries and accept the slight jitter and non-determinism
(2) investigate timing slewing where you'd specify a preferred time-of-day, and we'd slowly adjust the delay window until we hit that (e.g. we'd run cron.daily every 23.5 or 24.5 hours until we got back to 03:00)

i would highlight that the default behavior in other distros is basically to run anacron w/no time-of-day guarantees (Ubuntu/Debian/Fedora/RedHat).

(In reply to Michael Orlitzky from comment #17)

run-parts is not an option -- see bug 98189 for details there

also, no one is forcing you to use run-crons.  if you don't like the behavior, or want to use run-parts, or something else (like fcron's % and bootrun options), you're free to manage your cron and /etc/crontab however you want.  the whole point of run-crons is to provide /etc/cron.xxx behavior in the default Gentoo configs for all cron daemons with anacron-like guarantees.
Comment 19 Richard Freeman gentoo-dev 2015-07-22 11:08:23 UTC
(In reply to SpanKY from comment #18)
> also, no one is forcing you to use run-crons.  if you don't like the
> behavior, or want to use run-parts, or something else (like fcron's % and
> bootrun options), you're free to manage your cron and /etc/crontab however
> you want.  the whole point of run-crons is to provide /etc/cron.xxx behavior
> in the default Gentoo configs for all cron daemons with anacron-like
> guarantees.

++

I see the role of the base install as to provide some place to dump all those daily configs so that package maintainers don't have to worry about specific implementations, and the default crontab is a good generic starting point.  You should adapt it to your situation.

We don't want to be re-implementing anacron or similar in a shell script.  

No matter how we tune this you're going to run into similar arguments in any case.  If your system is down for a week and you boot it at 1AM and the scripts are set to run at 3AM should it run them twice, or just run them at 3AM?  What if you power on at 10PM, or 8PM, or 12PM, or 9AM, and so on?  Unless your system is on 24x7 you're always going to deal with the issue of extra runs or missed runs.
Comment 20 Michael Orlitzky gentoo-dev 2015-07-22 14:47:16 UTC
(In reply to Richard Freeman from comment #19)
> 
> We don't want to be re-implementing anacron or similar in a shell script.  
> 
> No matter how we tune this you're going to run into similar arguments in any
> case.

I don't remember what I was talking about back in 2012, but I agree here. This is the solution I've settled on (for a custom interval):

# Every fifteen minutes.
*/15  *  * * *  root	find -L /etc/cron.quarterhourly -type f -executable -execdir '{}' \;

I don't think it even solves this bug, but if it's going to be buggy, it's at least a whole hell of a lot simpler and more predictable. If you want anything more fancy, anacron or fcron are there.
Comment 21 SpanKY gentoo-dev 2015-07-23 03:08:08 UTC
(In reply to Michael Orlitzky from comment #20)

that cronjob will execute files that start with a . and end with a ~.  maybe that's fine for your setup, but you should be aware.  fwiw, if you don't mind the naming restrictions imposed by debianutils' run-parts, this might be better for your needs:

17 * * * * cd / && run-parts --report /etc/cron.hourly
Comment 22 Michael Orlitzky gentoo-dev 2015-07-25 05:55:16 UTC
(In reply to SpanKY from comment #21)
> (In reply to Michael Orlitzky from comment #20)
> 
> that cronjob will execute files that start with a . and end with a ~.  maybe
> that's fine for your setup, but you should be aware.  fwiw, if you don't
> mind the naming restrictions imposed by debianutils' run-parts, this might
> be better for your needs:
> 
> 17 * * * * cd / && run-parts --report /etc/cron.hourly

Yeah, thanks. More egregiously, mine will run anything in a subdirectory.

But it's nice and simple: "put something in cron.daily and it will be run daily." The fewer footnotes and caveats to that mental model, the better.
Comment 23 SpanKY gentoo-dev 2015-07-27 03:25:10 UTC
(In reply to Michael Orlitzky from comment #22)

i certainly laud the simplistic angle, but your statement hides the real world nuances and why we have `run-crons`.  here's what it should actually read:
"put something in cron.daily and it will be run daily at 03:00 iff your computer happens to be powered up at that time, otherwise none of them will run daily, and it might be weeks or months before your 'daily' jobs run (e.g. laptops)"
Comment 24 Michael Orlitzky gentoo-dev 2015-07-27 18:53:51 UTC
(In reply to SpanKY from comment #23)
> (In reply to Michael Orlitzky from comment #22)
> 
> i certainly laud the simplistic angle, but your statement hides the real
> world nuances and why we have `run-crons`.  here's what it should actually
> read:
> "put something in cron.daily and it will be run daily at 03:00 iff your
> computer happens to be powered up at that time, otherwise none of them will
> run daily, and it might be weeks or months before your 'daily' jobs run
> (e.g. laptops)"

Well whatever promises you try to make to me, I can turn you into a liar by hitting the power button on the glowing box. The question is how big of a "but..." do we want to attach to that case.

First, we could do nothing. "If you wanted your computer to do something, you shouldn't have turned it off." This is nice and simple, and even more desirable in some cases. For example, I run a log analyzer overnight that temporarily bogs down our web server. Since our customers are local, it's fine to run it at midnight. But if it were to kick off at 8am, it would cripple the websites of everyone trying to log in as they get to work. If it doesn't run at midnight, I don't want it to run at all. The find ... -exec recipe basically does that, and that's how cron historically worked.

The second thing you might want to do is make a best effort to run missed jobs. There are plenty of reasons to do that too. Maybe you want to glsa-check roughly once a day -- if the machine is off, who cares. But, it makes things way more complicated (the problem sounds easier than it is). Plus, there are already cron "alternatives" that have this feature. Anacron, fcron, and now cronie all fit the bill. I don't think it's too awful to tell people to use a sexier cron if they need this feature.

As far as this bug is concerned, it seems like a result of biting off more than we could chew. If we had punted on the "your computer was off" issue, I think everyone needing that feature would understand and use an alternative cron. But users get upset if you try to do something and then mess it up. During the install, the user has to choose his or her cron implementation. Pretending for a moment that this is an informed decision, cronbase shouldn't quietly subvert it and give you anacron behavior with e.g. vixie-cron.
Comment 25 SpanKY gentoo-dev 2015-08-10 08:14:42 UTC
(In reply to Michael Orlitzky from comment #24)

not really.  if you don't turn the system back on, then obviously software won't run, and then it's a moot issue.

the point is to have things "just work" which is what run-crons does.  we don't guarantee specific timings (never have), just that things are eventually run within the reasonable time frames.  if you have a system with more critical constraints, then you can figure out how to make things work best for you in your environment.  but that is not the normal scenario which means it too is irrelevant to this discussion.
Comment 26 Michael Orlitzky gentoo-dev 2017-01-14 05:25:31 UTC
*** Bug 605574 has been marked as a duplicate of this bug. ***
Comment 27 c.cboldt 2017-01-14 08:45:30 UTC
I added one hour to each of the ctime/age tests in run-crons.  It resolves my 24/7 system problem (and some of the cron jobs I strongly prefer to not run twice a day), and does not offend the laptop user's needs either, because the lastrun files age even while the laptop is off, so his cron jobs will run tomorrow when he turns the computer on again.
Comment 28 Remy Blank 2017-10-29 11:30:39 UTC
I was redirected here from bug #605574, which I believe is only a small subset of this issue, but an annoying one. I understand that time management is hard, even more so with systems that aren't powered 24/7.

However, the run-crons script does have a bug that I believe is not desirable for any use case. It cleans up stale /var/spool/cron/lastrun/cron.* files too early. On a permanently-running system, that code section should never do anything. On an intermittently-running system, it ensures that missed cron jobs are run soon after powering up.

Currently, the daily lastrun file is cleaned up after 1d5m. The daily removal is scheduled at 03:09. On a switch from DST to non-DST, the clock goes back from 03:00 DST to 02:00 non-DST. The stale cleanup triggers at 02:14 non-DST, so the daily cron jobs run at 02:20. Then, at 03:09 non-DST, the file is removed again, and the daily cron jobs run again at 03:10.

This can be fixed very easily by increasing the stale cleanup timeout by one hour to 1d1h5m for daily, 1w1h5m for weekly, and 31d1h5m for monthly, as described in comment 1 and comment 27. Then the cleanup will trigger at 03:14 non-DST, after the normal run, and therefore not have any effect. This will avoid the double run, and will not affect systems that aren't powered continuously.

Attaching a patch that adjusts the cleanup intervals. I know it's not a perfect fix, but it's arguably better than the current state. I would appreciate if it could be applied.
Comment 29 Remy Blank 2017-10-29 11:32:28 UTC
Created attachment 500786 [details, diff]
Add 1h to cleanup timeout for daily, weekly and monthly
Comment 30 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2017-11-23 18:22:40 UTC
(In reply to Remy Blank from comment #29)
> Created attachment 500786 [details, diff] [details, diff]
> Add 1h to cleanup timeout for daily, weekly and monthly

A quick read of this shows me a problem:
In the non-DST-transition case, the deletion period slows wraps around.

This script was originally taken from SuSE, circa 2002. Can somebody track down how they dealt with this problem?
Comment 31 ixuz 2022-10-30 09:53:36 UTC
Why is this even a thing after 18 years? :-)

Anyways I don't know if my system suffers from this "bug" or if I did some misconfiguration while the migration to cronie couple years ago.

My gentoo installation is from 2010 and I migrated from vixie-cron to cronie in 2018.

Tonight my daily cron job run twice because of the DST change (clock went 1 hour back).

My config:

/etc/crontab

# check scripts in cron.hourly, cron.daily, cron.weekly and cron.monthly
59  *  * * *    root    rm -f /var/spool/cron/lastrun/cron.hourly
9  3  * * *     root    rm -f /var/spool/cron/lastrun/cron.daily
19 4  * * 6     root    rm -f /var/spool/cron/lastrun/cron.weekly
29 5  1 * *     root    rm -f /var/spool/cron/lastrun/cron.monthly
*/10  *  * * *  root    test -x /usr/sbin/run-crons && /usr/sbin/run-crons

Do I even need these "rm -f lastrun" entries? Cause I see that "run-crons" does delete these with an time offset!?
Comment 32 Michael Orlitzky gentoo-dev 2022-10-30 13:13:32 UTC
(In reply to ixuz from comment #31)
> 
> My gentoo installation is from 2010 and I migrated from vixie-cron to cronie
> in 2018.
> 
...
> 
> Do I even need these "rm -f lastrun" entries? Cause I see that "run-crons"
> does delete these with an time offset!?

You probably don't need that stuff.

The default cronie config basically does what I was arguing for, and what comment #21 suggested. Unlike vixie-cron IIRC, cronie respects /etc/cron.d, and comes with an "empty" crontab. But in /etc/cron.d I have

  $ cat /etc/cron.d/dailyjobs 
  ...

  9 5 * * * root [ ! -f /etc/cron.hourly/0anacron ] && run-parts /etc/cron.daily
  22 4 * * 0 root [ ! -f /etc/cron.hourly/0anacron ] && run-parts /etc/cron.weekly
  42 4 1 * * root [ ! -f /etc/cron.hourly/0anacron ] && run-parts /etc/cron.monthly

which is nice and simple, but may not be exactly what you want, either (see the subsequent comments). If you need something more complicated, you probably want a more complicated cron daemon.
Comment 33 ixuz 2022-10-30 16:00:05 UTC
Thanks, what I want is easy (I thought): The cronjobs should not executed twice once a year due to DST change.

IIRC while migrating to cronie I had the problem that all cronjobs were running twice. This was related to anacron, so I disabled this and the corresponding (new anacron) cron jobs. So basically I use cronie with cron jobs like it was before with vixie cron.

Since only the daily job seems to be running twice after DST change, I just changed it to start 1 hour and 10 min. later (from 3:09 to 4:19 AM). This is outside the "changing time range" (2 AM <--> 3 AM). But I am not completely sure if this works as the daily cron job should start at 3:09 AM while on the other hand the clock jumped back at 3:00 AM to 2:00 AM. So it happens 9 minutes before that job even started!? Or wait this is because the run-crons scripts counts static minutes from the lastrun file creation?
Comment 34 ixuz 2022-10-30 17:26:22 UTC
NVM I just switched to "run-parts" for hourly, daily, weekly, monthly and removed the "rm lastruns" and "run-crons" jobs as this system runs 24/7.

Just need to keep in mind to not schedule a cronjob between 2-3 AM ;-).

My humble opinion on this whole "bug": Why would you IMPLICITLY introduce a feature (via "run-crons") which catch up missed cronjobs while the system was shutdown which - due to this - breaks any system who "suffers" on DST change at least once a year. It would be better to rip off this "feature" and if you are one who needs this, you can just search EXPLICITLY for a solution like Anacron. It is better to have not at all a feature than to have a feature which could possibly destroy things when it runs twice were it is supposed to be to run only once.
Comment 35 Michael Orlitzky gentoo-dev 2022-11-01 00:45:15 UTC
(In reply to ixuz from comment #34)
> 
> My humble opinion on this whole "bug": Why would you IMPLICITLY introduce a
> feature (via "run-crons") which catch up missed cronjobs while the system
> was shutdown which - due to this - breaks any system who "suffers" on DST
> change at least once a year. It would be better to rip off this "feature"
> and if you are one who needs this, you can just search EXPLICITLY for a
> solution like Anacron. It is better to have not at all a feature than to
> have a feature which could possibly destroy things when it runs twice were
> it is supposed to be to run only once.

It would be nice to see bcron and fcron copy what cronie does so that we can finally throw run-crons in the recycle bin. The other option, dcron, is unmaintained in Gentoo with its last release in 2011. It can probably be last-rited.