TL;DR: Separate scheduling (who fires when) from execution (who runs it): a replicated leader storing schedules durably, workers leasing jobs with heartbeats, and every job idempotent or keyed. The failure modes to design out are double-firing across replicas, the midnight thundering herd, and silent non-execution after a missed window.
How to approach it
Start by naming why vanilla cron fails here: single-host binding (jobs die with the machine), no state about runs (no retries, no history), and silent failure as its default error mode. Then design in layers: schedule store, trigger loop, execution with leases.
A strong answer
Architecture.
The schedule store holds job specs declaratively (cron expression, command or service reference, owner, timeout, retry policy, priority). Declarative matters operationally: specs get reviewed like code instead of edited on one box via crontab -e, which is how teams lose track of what runs where.
Triggering: a leader scans for due jobs each tick. The hard problems live here:
- One fire, not two. Two replicas must not both launch the payroll job. Leader election handles steady state, but elections overlap, so the durable guarantee comes from the run key:
(job_id, scheduled_time)unique in the queue/store. Whichever replica claims the key first wins; the loser's enqueue is a no-op. - Missed windows. Scheduler down 02:00 to 02:20: what happens to the 02:00 job? Policy per job: fire-late if still useful, skip-with-alert if time-bound, never silently vanish. Vanilla cron's answer (nothing) is the bug you are selling against.
- Clock edges. Jobs pinned to wall-clock times around DST transitions need explicit policy. State yours.
Execution: workers lease a job for its expected duration, heartbeat to extend, and the lease expires on death so another worker can retry. At-least-once delivery falls out of this, which forces the real design constraint onto jobs themselves:
| Job type | Guarantee needed | Mechanism |
|---|---|---|
| Report generation | Effectively once | Deterministic output keyed by run ID; overwrite safe |
| Charge/payment | Deduplicated effect within provider contract | Run key sent downstream; retention and reconciliation checked |
| Cleanup/purge | At least once fine | Natural idempotence |
This table is the interview's centre of gravity: a scheduler can only offer at-least-once, so either jobs tolerate repetition or they carry keys that make repetition harmless. Candidates who promise "exactly-once execution" without this have not built it.
The thundering herd. Twenty thousand jobs specified as 0 2 * * * is twenty thousand simultaneous tasks. Mitigate by design: spread defaults automatically when users write midnight jobs (jitter within an hour unless pinned), enforce per-worker concurrency limits, priority queues so critical jobs jump the mass, and backpressure that delays rather than drops. The same herd arrives at dependencies too: if all those jobs hit one database, your scheduler just became a DDoS weapon pointed inward.
Operations. Every run recorded (start, end, exit, retries) because "did last night's job run?" must be answerable in seconds. Missed-window and timeout alerts route to owning teams, not a central queue nobody reads. And a kill switch per job class: when a bad deploy makes a job poison data, stopping it should take one API call, not an emergency crontab edit.
What interviewers probe next
"Leader dies mid-enqueue of a thousand jobs." Enqueues are individual writes under run keys; the successor rescans the same window, finds keys already present, enqueues only the rest. Idempotency again.
"How do you handle a job that hangs forever?" Enforce a maximum runtime independent of heartbeat renewal, stop/fence expired workers and alert on failed progress. A healthy heartbeat thread can otherwise renew a hung job forever; retrying must still respect the downstream idempotency window. Silent infinite hangs are a design defect, not an operational surprise.
"Why not just use Kubernetes CronJobs?" For many teams, correct answer! It solves placement and restarts. Its gaps: fleet-wide deduplication, rich run history, cross-cluster policies. Knowing when the off-the-shelf thing suffices scores points.
Common mistakes
Designing exactly-once delivery instead of exactly-once effects. Delivery guarantees beyond at-least-once cost more than making jobs idempotent.
Ignoring the dependency stampede. The scheduler works perfectly while the database it feeds falls over at 00:00 sharp.
No answer for missed windows. The question "what happens to jobs skipped while you were down?" has a right answer per job and no default.