CronJob: the 100-missed-schedules bug was fixed in 1.23
Two things everyone still tells you about Kubernetes CronJobs stopped being true four years ago, and the advice they came with is now the risk
Search for why a Kubernetes CronJob stopped running and you'll land on the same answer within two results. It missed more than 100 scheduled runs, the controller gave up permanently, and the fix is to set startingDeadlineSeconds. The posts are confident and recent.
The behaviour they describe was removed in Kubernetes 1.23, in December 2021.
I went and read the controller across five release branches to be sure, because I had repeated this advice myself. Here is what's actually in there.
What the old controller did
Through 1.22, getRecentUnmetScheduleTimes counted missed start times and bailed out hard once the count passed 100:
return []time.Time{}, fmt.Errorf(
"too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew")That's an error return, not a warning. The caller stopped, no Job was created, and status.lastScheduleTime never advanced. That last one is what made it permanent. Next sync, the controller measured the gap from the same stale timestamp, counted the same growing pile of misses, and errored again. The CronJob looked healthy in kubectl get cronjobs. It would never run again without human intervention.
That's a nasty bug and it deserved every blog post it got. The recommended mitigation was correct too: setting startingDeadlineSeconds narrowed the window the controller counted misses over, so the count stayed under 100 and the wedge never triggered.
What replaced it
The rewritten controller became the default in 1.21 and the only one from 1.23. In current code the same condition produces an event and nothing else:
if missedSchedules == manyMissed {
recorder.Eventf(cj, corev1.EventTypeWarning, "TooManyMissedTimes",
"too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew")
logger.Info("too many missed times", "cronjob", klog.KObj(cj))
}
return mostRecentTime, errThe message survived, which is why the myth has such a long half-life - people still see TooManyMissedTimes in their events and reasonably conclude the documented consequence followed. It didn't. mostRecentTime is returned, the Job gets created, scheduling continues.
The second stale fact travels with the first. The old controller ran wait.Until(jm.syncAll, 10*time.Second, stopCh) - a flat ten-second poll across every CronJob in the cluster. That's where "don't set startingDeadlineSeconds below 10 seconds or the controller will miss the window" comes from, and it was true. The new one uses a delaying work queue and requeues each CronJob at its own next scheduled time, with a 100 millisecond pad for NTP skew:
nextScheduleDelta = 100 * time.MillisecondNo polling interval to lose runs in.
What actually bites now
Three things, and startingDeadlineSeconds is one of them - not as a fix, as the cause.
The deadline still skips runs, and now that's its only job. The check is live in the current controller:
tooLate := false
if cronJob.Spec.StartingDeadlineSeconds != nil {
tooLate = scheduledTime.Add(time.Second * time.Duration(*cronJob.Spec.StartingDeadlineSeconds)).Before(now)
}Past the deadline, you get a MissSchedule warning event and the run is dropped. With the wedge gone, a field that used to buy you protection now only buys you skipped executions during any control-plane hiccup longer than the value you picked. Copying startingDeadlineSeconds: 30 from a 2020 answer onto a modern cluster is a straight downgrade.
There's a wrinkle worth knowing if you alert on events. The controller doesn't record the miss in status - there's a standing TODO in the source about it - so lastScheduleTime stays put and the same miss is re-detected and re-reported on later syncs. One skipped run can produce a stream of MissSchedule events.
`concurrencyPolicy: Forbid` drops runs silently. This is the failure I see most often in practice and it gets a fraction of the attention. A job that normally takes two minutes hits a slow dependency and runs for twenty. Every scheduled start in that window is skipped, not queued. The CronJob is working exactly as configured. Nothing is failing. You're simply not getting the runs you think you're getting, and the only trace is in events, which are gone in an hour by default.
`backoffLimit` fails the Job, not the CronJob. The claim that exhausting backoffLimit suspends future execution isn't something I can find in the controller, and the behaviour doesn't match it: the Job goes to Failed and future scheduled runs continue normally. The actual problem is quieter. Failed Jobs sit in the namespace until failedJobsHistoryLimit (default 1) garbage-collects them, so by the time somebody looks, the evidence of the earlier failures is gone. suspend: true on the CronJob spec is the field that stops scheduling, and only if someone set it.
What to do instead
Leave startingDeadlineSeconds unset unless you can name the thing you're protecting against. The honest cases are narrow: a job that's worse than useless if it starts late, like a report bound to a market close, or one whose late start would collide with a maintenance window.
Alert on staleness, not on failure. Failures are visible. The dangerous mode is a CronJob that stops producing runs without erroring, and the metric that catches all three paths above is one expression:
time() - kube_cronjob_status_last_schedule_time > <2x your interval>That covers the deadline skip, the Forbid skip, and someone leaving suspend: true after an incident. It doesn't care which one happened.
Raise failedJobsHistoryLimit to something like 5 before you need it. The default of 1 means a flapping job erases its own history, and you get exactly one artifact to debug from.
The limits worth knowing
If you're on a cluster older than 1.23 - which in the wild usually means a vendor appliance or something in a regulated environment on a long support contract - the original bug is real and the original advice applies. Check the version before deciding which set of rules you're living under.
And there's one condition that still errors out rather than warning: if two consecutive schedule times work out to less than a second apart, mostRecentScheduleTime returns time difference between two schedules is less than 1 second. That takes a malformed schedule to hit, but it's the one place where a CronJob really does stop dead.
The general lesson is cheaper than any of this. Controller behaviour is a moving target and blog posts aren't. When something in Kubernetes behaves differently from what every result on the first page says, the source is one gh api call away and it's dated.