The problem is, for /etc/cron.{hourly,daily,weekly,monthly} we have the following requirements:
1) Scripts should ideally be run at a specified time.
2) Ideally the time at which the scripts run should be slightly different for each of the groupings to avoid unnecessarily loading the system and reduce potential conflicts in related scripts in different groups.
3) Scripts should never run twice within the same time period
4) If the scripts are not run for any reason in their given time period (eg. the system is offline), we want to run them at the next convenient opportunity
5) Even if the scripts are not run at their normal time for any reason, we want to run them at the normal time in the next time period
6) Because this is something very useful, we want to do this with as few resources or dependencies as possible on as many platforms as possible

So the solution the world came up with is run-crons. Which works. Most of the time.

The real headache comes because of daylight savings. Even if we don’t schedule to run in daylight savings, it leaves its mark in the form of an extra or less hour in the day that’s unexpected. This causes a problem because run-crons compares the timestamp on its “last run” spool file with the current time. Once this gets more than 24 hours old, we want to run the daily scripts again… except on daylight savings changes, when this figure is 23 or 25 hours. And we’re trying to do this in a shell script, using the most generic tools possible.

So we could use “3am” as the time to compare to – but then run-crons has to know this time. It also still brings up the issue of if the system was down for 20 hours, and comes back up at 11pm, do you want to run the cron.daily scripts for that day still? Then again at 3am the following day?

Brain damage ensues.

Frankly I’m beginning to think the world would be a better place if we did away with cron.{hourly,daily,weekly,monthly} altogether and just used cron.d. But even that still doesn’t resolve all the issues.

Turns out this isn’t quite the case. There’s a 5 year old bug in Gentoo’s Bugzilla about this known issue and it appears to be down to the way run-crons manages its lock / last run files. I’m already running a run-crons patched for recursive directory scanning on some of my systems, so it wasn’t that big a step to basically do away with Gentoo’s supplied script altogether.

Now I’m running “run-crons-in” – a much simplified script that simply relies on the cron daemon itself for timing (which is how it should be, in my opinion). Here’s my script:

#!/bin/bash
# 2009-10-27 Complete rewrite of run-crons
# This version is designed to be called using @hourly @daily @weekly or @monthly
# With, for example: run-crons-in daily
#
# This version assumes:
# - We don't care if a previous script is still running
# - The cron daemon handles all timing in a sensible manner
# - Therefore no locking is necessary

AJB_DEBUG=0
AJB_TIMES=1
AJB_PTIMES=1
AJB_TIMEFORMAT='%Y-%m-%d %T'
BASE=$1

AJB_STARTTIME="`date +\"${AJB_TIMEFORMAT}\"`"
[ $AJB_TIMES -eq 1 ] && echo "Start time: ${AJB_STARTTIME}"

run_recursive() {
        CRONDIR="$1"
        [ $AJB_DEBUG -eq 1 ] && echo "Executing cron scripts in directory: $CRONDIR"
        for SCRIPT in $CRONDIR/* ; do
                if [[ -x $SCRIPT && ! -d $SCRIPT ]]; then
                        [ $AJB_DEBUG -eq 1 ] && echo "Executing cron script: $SCRIPT"
                        AJB_PSTARTTIME="`date +\"${AJB_TIMEFORMAT}\"`"
                        [ $AJB_PTIMES -eq 1 ] && echo "Script start time: ${AJB_PSTARTTIME}"
                        $SCRIPT
                        AJB_PSTOPTIME="`date +\"${AJB_TIMEFORMAT}\"`"
                        [ $AJB_PTIMES -eq 1 ] && echo "Script stop time: ${AJB_PSTOPTIME}"
                fi

                if [[ -d $SCRIPT ]]; then
                        run_recursive $SCRIPT
                fi
        done
}

CRONDIR="/etc/cron.${BASE}"

test -d "$CRONDIR" || echo "Directory does not exist or is not a directory: ${CRONDIR}"
set +e
test -d "$CRONDIR" && run_recursive "$CRONDIR"

AJB_STOPTIME="`date +\"${AJB_TIMEFORMAT}\"`"
[ $AJB_TIMES -eq 1 ] && echo "End time: ${AJB_STOPTIME}"

As you can see, it’s fairly basic – altho I’ve added some extra output for timing and debugging, simply because I felt like it. I’m currently testing this long term to ensure it works as expected, but hopefully I’ll adopt this for all my systems within a couple of months.

The full /etc/crontab entries are now:

@hourly   root    /usr/local/sbin/run-crons-in hourly
@daily    root    /usr/local/sbin/run-crons-in daily
@weekly   root    /usr/local/sbin/run-crons-in weekly
@monthly  root    /usr/local/sbin/run-crons-in monthly

(Yes, I realize that, at least by the man page, those are all going to run at midnight, but daily is the only set of scripts that ever does anything really stressful on the system I’m currently testing on, and it’s a quad core, and I really don’t care otherwise I wouldn’t be testing cron daemons on it)

I’m also taking the opportunity to use cronie in place of vixie-cron. There’s little documentation for this project, but as far as I can see it’s a drop-in replacement for / fork of vixie-cron from redhat that’s actively maintained.

I did take a quick look at the other cron daemons in the portage tree, but cronie appears to be the only one that’s actively maintained.