Design a Job Scheduler: System Design Interview 2026

A distributed job scheduler looks like cron, scales like a distributed database, and fails like a coordination problem. Its job description sounds dull - fire each scheduled task at its time, on some pool of machines - until you account for what happens when two scheduler nodes both think they should fire the same job, when one of them is down for ten minutes and a thousand jobs were due in that window, or when an aligned cron expression makes ten thousand jobs come due at the same second. None of those questions has a "just throw more servers at it" answer, which is what makes this the natural closing problem of a system design series: the answer composes nearly every pattern that came before.

This walkthrough assumes the 6-step system design framework and applies it at senior-plus depth. It is Part 12 - and the final part - of a system design series.

Table of Contents

1. The Problem
2. Step 1 - Clarify Requirements
3. Step 2 - Estimate Scale
4. Step 3 - API and Data Model
5. Step 4 - High-Level Design
6. Step 5 - Deep Dive: Leader Election, Sharded Scheduling, and At-Least-Once Execution
7. Step 6 - Bottlenecks and Trade-offs
8. Reference Architecture
9. Common Mistakes in the Interview
10. Quick Reference
11. Frequently Asked Questions
12. Sources
13. Related Articles

The Problem

We are designing a distributed job scheduler - the engine behind cron jobs at platform scale, delayed retries, scheduled emails, periodic ETL, billing runs, and any other "run this at time T" workload. The canonical examples are Quartz, Kubernetes CronJobs, AWS EventBridge Scheduler, and the scheduler underneath any workflow engine like Airflow.

The design has two boundaries. The scheduling decision - “is this occurrence due, and who may claim it?” - needs a single authoritative state transition. The delivery and execution path is normally at-least-once because queue delivery and arbitrary external effects do not share one transaction. Coordination, a transactional outbox, stable occurrence IDs, idempotent effects, and reconciliation make lost or duplicate work detectable and recoverable; none of them alone creates a universal exactly-once guarantee.

Step 1 - Clarify Requirements

Functional requirements:

• Register a job: a definition plus a schedule (cron expression, fixed delay, or a one-shot runAt).
• Trigger jobs at the right time and dispatch them to workers.
• Track job runs and their outcomes.
• Retry failed jobs with backoff.
• Cancel, pause, and modify registered jobs.
• Define time zone, start deadline, overlap policy, and misfire behavior explicitly.

Out of scope (name, then defer): the job logic itself (we assume idempotent workers), workflow DAGs (Airflow is a scheduler plus a workflow engine - we design only the scheduler), and the authoring UI.

Non-functional requirements:

• Do not run a job twice when not asked to - and accept that this is "minimise duplicates", not "eliminate", because of at-least-once execution.
• Do not silently miss a scheduled run - and have an explicit policy for what to do about runs missed during an outage.
• Scale to the scenario target: millions of jobs and peaks of tens of thousands of due occurrences per second.
• Time precision to seconds; sub-second is a different problem.
• High availability, so a node failure does not stop firing.
• Tolerate clock skew between machines.

The decisive clarifying question is the exact correctness boundary. The occurrence row can be created once under a database uniqueness constraint. Queue delivery is at-least-once. A worker may achieve one durable state change when its deduplication record and effect share a transaction, but an arbitrary call to an external target needs that target's idempotency facility or reconciliation. The scheduler should make duplicate dispatches rare and missed or dropped occurrences observable, not hide them behind an absolute “exactly once” slogan.

Step 2 - Estimate Scale

Treat the following numbers as interview assumptions to validate with the interviewer, not properties of every scheduler.

Registered jobs. Assume 10 million across the platform - a mix of recurring crons and one-shot delayed jobs.

Trigger rate. Peak around ~10,000 triggers/sec, the peak coming from aligned cron expressions: "every hour at :00" makes thousands of jobs come due at the same instant. The scheduler must absorb that herd rather than serialise on it.

State storage. A 500-byte logical job payload makes 10 million jobs about 5 GB before indexes, row/version overhead, replication, backups, and headroom. Run history depends on the actual occurrence rate and retention; estimate it from runs/second × retained seconds × measured bytes/run, then add the same physical overheads instead of guessing “a few TB”.

The interesting number is not the totals but the spike-to-average ratio: the scheduler is mostly idle and occasionally has to fire ten thousand jobs at the same second. Sharding is what spreads that spike.

Step 3 - API and Data Model

POST /api/jobs
  Idempotency-Key: <client request key>
  body: { schedule, timeZone, payloadRef, targetRef, retryPolicy,
          misfirePolicy, startDeadline, overlapPolicy }
  201 Created
 
PATCH  /api/jobs/{id}  If-Match: <version>
DELETE /api/jobs/{id}
GET    /api/jobs/{id}/runs

The data model is small but carefully indexed:

Three constraints carry the design. nextRunAt is precomputed and indexed by (shard, nextRunAt) so an owner can scan a bounded due range. A unique occurrenceKey = (jobId, scheduledAt) makes repeated claims converge on one logical run. The occurrence row and dispatch outbox are committed with the schedule advance, closing the database-to-queue dual-write gap. misfirePolicy, startDeadline, and overlapPolicy state what to do when time was missed or an earlier run is still active.

Creation and update endpoints also need authentication, tenant quotas, schedule and IANA time-zone validation, optimistic concurrency, and an allowlisted target model. Do not accept arbitrary internal URLs or embed long-lived credentials in job payloads; that turns the scheduler into an SSRF and secret-exfiltration service.

Step 4 - High-Level Design

flowchart TD
    Cli([Client]) -->|register / cancel| API[Scheduler API]
    API --> Store[(Job Store<br/>indexed by shard, nextRunAt)]
    subgraph Sched["Scheduler tier - one owner per shard"]
        S1[Scheduler Node A]
        S2[Scheduler Node B]
        S3[Scheduler Node C]
    end
    Coord[Coordination Service<br/>etcd / ZooKeeper / Consul]
    S1 <-->|lease + fencing token| Coord
    S2 <-->|lease + fencing token| Coord
    S3 <-->|lease + fencing token| Coord
    S1 -->|tick: claim + advance due jobs| Store
    S2 --> Store
    S3 --> Store
    Store -->|unpublished outbox rows| Pub[Outbox Publishers]
    Pub -->|retry publish| Q[Worker Queue]
    Q --> W[Worker Pool<br/>idempotent jobs]
    W -->|run result| Store

Figure 1. Scheduler nodes coordinate shard ownership, then create occurrences and dispatch-outbox rows in the indexed job store. Separate publishers retry the outbox into the worker queue, so a scheduler crash cannot leave an advanced schedule with no durable dispatch intent.

The scheduler tier has multiple nodes; shard ownership lives in a consensus-backed coordination service. Each owner scans due jobs in its shards. In one job-store transaction it creates a uniquely keyed occurrence, advances the definition, and inserts an outbox event. Publishers deliver outbox events to the worker queue with retries; workers consume them and record results. The job store must enforce the fencing token on ownership-sensitive mutations. The queue and publishers still have their own durability, ordering, retention, and backpressure requirements.

Step 5 - Deep Dive: Leader Election, Sharded Scheduling, and At-Least-Once Execution

This is the core. Four mechanisms compose: shard ownership through leader election, fencing enforced by the job store, an atomic occurrence/schedule/outbox transaction, and at-least-once delivery to idempotent workers - with explicit time and recovery policies.

Part A - Why coordination is needed

Run N scheduler nodes that each scan and claim the whole job set without an atomic store-side guard and they contend or create duplicate occurrences. Two useful ownership structures are:

• A single global leader. One node evaluates schedules; the rest stand by. This can be the simplest adequate design until measured scan or dispatch load exceeds one node, but it concentrates capacity and failover.
• Sharded ownership with one authority per shard. Partition the job set into many logical shards and assign each shard to a current owner. Use more shards than nodes so ownership can rebalance without repartitioning every job.

The sharded option is a scaling choice, not a senior-interview ritual. Hashing jobId can distribute definitions evenly while leaving due-time load skewed: one shard may contain many aligned high-frequency schedules. Balance on measured scan cost, due occurrences, tenant quotas, and worker pressure, and split hot shards when needed.

Part B - Leader election and fencing tokens

Shard ownership is delegated to a consensus-backed coordination service such as etcd. A scheduler attaches an ownership key to a renewable lease and receives a monotonically ordered token, for example an etcd revision. A crashed or disconnected holder eventually loses the lease, after which another node may acquire ownership. Actual recovery time includes expiry, detection, election, shard loading, and backlog scanning; it is not bounded by the TTL alone.

flowchart TD
    Start([Node attempts to own shard S]) --> Try[Compare-and-swap on shard-S key]
    Try -->|success| Own["Owner: lease with TTL + fencing token N"]
    Try -->|already owned| Wait[Wait for lease to expire]
    Wait --> Try
    Own --> Renew[Periodic renew]
    Renew -->|ok| Renew
    Renew -->|crash or partition| Expire[Lease expires]
    Expire --> Try
    Own --> Work["Run scheduler tick for shard S<br/>writes carry fencing token N"]

Figure 2. The lease lifecycle for one shard. The lease limits how long ownership remains valid without renewal; the fencing token lets the job store reject a stale owner. The process must stop scheduling when it cannot establish that its lease is still valid.

What still has to be handled is a stale former owner. A paused process or network partition can leave an old node acting on an assumption that has expired while a new owner has already taken over. A lease held in etcd cannot by itself protect a separate SQL database, queue, or target.

The defence is a fencing token enforced by the protected resource. Ownership-sensitive job-store transactions carry the token, and the store atomically rejects any value older than the shard's accepted token. Rejection must be surfaced so the node stops; it is not a success to ignore. Fencing blocks a stale claim, but it cannot retract a message published while the former owner was still valid or protect a target that never compares the token. That is why occurrence uniqueness, the outbox, and downstream idempotency are separate requirements.

sequenceDiagram
    participant A as Scheduler A (former owner)
    participant Co as Coordination service
    participant B as Scheduler B (new owner)
    participant DB as Job store
 
    Note over A: owns shard, fencing token = 7
    Note over A,Co: network partition - A cannot renew
    Note over Co: A's lease expires
    B->>Co: claim shard S
    Co-->>B: granted, fencing token = 8
    Note over A: partition heals, A still thinks it owns S
    A->>DB: claim-and-advance, token = 7
    DB-->>A: REJECTED - latest token is 8
    B->>DB: claim-and-advance, token = 8
    DB-->>B: OK
    Note over A,B: stale state mutation is fenced while occurrence dedup protects delivery

Figure 3. The job store rejects the stale token after ownership changes. This prevents the old node from creating or advancing state, but it does not by itself guarantee zero duplicate delivery or exactly-once external effects.

Part C - The tick loop and atomic claim-and-advance

Each scheduler owns a set of shards. For every owned shard, every ~second:

1. Select a bounded batch: find active definitions in shard S with nextRunAt <= database_now, ordered by due time. A database-backed work queue can use row locks such as PostgreSQL FOR UPDATE SKIP LOCKED; this is intentionally a queue-like use, not a general consistent read.
2. Settle the schedule in one transaction: verify the fencing token and job version, insert a unique occurrence for (jobId, scheduledAt), compute the next occurrence according to time-zone, overlap, deadline, and misfire policies, and insert a dispatch-outbox row. Update nextRunAt only when those writes commit.
3. Publish the outbox: independent publishers retry unpublished rows to the worker queue and mark publication progress. The queue may accept a message and lose the acknowledgement, so publishing remains at-least-once.

If a scheduler crashes before the transaction commits, the next valid owner can still see the due definition. If it crashes after commit, the durable outbox remains publishable. If a publisher crashes after the queue accepts a message but before publishedAt is stored, it may publish again; the occurrence key lets consumers deduplicate. This closes the common “advanced nextRunAt, lost dispatch” gap while keeping delivery semantics honest.

The transaction is an atomic state transition: it prevents two contenders from creating distinct logical occurrences for the same (jobId, scheduledAt) and couples the schedule advance to durable dispatch intent.

The aligned cron herd is not solved by sharding alone. Spread logical shards across owners, cap each claim batch, apply per-tenant quotas, use backpressure, and offer flexible delivery windows or deterministic jitter where product semantics allow it. Capacity-plan the queue and workers for the accepted lateness SLO; otherwise sharding only moves the bottleneck.

Part D - At-least-once execution and idempotency

Once a trigger is on an at-least-once worker queue, a lost acknowledgement or worker crash can cause redelivery. “Exactly once” is meaningful only with a boundary: a worker can atomically deduplicate and update one database when both happen in the same transaction, but it cannot generally combine an arbitrary email, payment-provider call, or file write with the queue acknowledgement.

The usual contract is at-least-once delivery plus idempotent effects. The scheduler supplies the occurrence key. A worker stores durable deduplication with the effect in one local transaction, or passes the key to a downstream provider that guarantees idempotent handling. A best-effort cache is insufficient for high-value effects because eviction and retention expiry re-open duplicates. Reconciliation must find occurrences stuck in pending, running, retry, or terminal-failure states.

Job lifecycle

stateDiagram-v2
    [*] --> PENDING_DISPATCH: occurrence transaction commits
    PENDING_DISPATCH --> ENQUEUED: outbox publish attempted
    ENQUEUED --> RUNNING: worker claims delivery
    RUNNING --> SUCCEEDED: ok
    RUNNING --> RETRY_WAIT: transient failure
    RETRY_WAIT --> ENQUEUED: backoff elapsed
    RUNNING --> DEAD_LETTER: permanent / attempts exhausted
    SUCCEEDED --> [*]
    DEAD_LETTER --> [*]

Figure 4. Lifecycle of one occurrence, separate from the recurring job definition. The definition remains active with its nextRunAt while each occurrence is durably dispatched, executed, retried, and retained or archived for audit.

The recurring definition and its occurrences have different lifecycles. Creating an occurrence advances the definition's nextRunAt; execution changes only that occurrence. Failed attempts retry with bounded exponential backoff and jitter; an occurrence that exhausts its retry policy enters a terminal failure or DLQ workflow with replay authorization and audit history.

Part E - Missed schedules and time

A scheduler outage leaves definitions whose nextRunAt has passed. On recovery the owner can catch up within a bounded window, skip to the next future occurrence, or create one latest occurrence and resume. Combine that with a start deadline and maximum catch-up count so a long outage cannot create an unbounded storm. The right policy is per job: billing may require auditable catch-up, while a heartbeat usually skips.

Civil time is as important as clock skew. Store an IANA time-zone identifier with a cron schedule and define daylight-saving behavior for nonexistent and repeated local times. Distinguish fixed-rate from fixed-delay schedules and define whether runs may overlap (allow, forbid, or replace). Use the job store's transaction time or another authoritative service for due comparisons; workers consume already-created occurrences and do not recalculate whether they were due.

Consistency model

Ownership decisions and the occurrence-creation transaction require linearizable or serializable-enough compare-and-set semantics at their respective boundaries. Dispatch and execution are at-least-once. Whether a user observes one effect depends on durable deduplication and the downstream target's contract. State the lateness objective, allowed overlap, retry horizon, deduplication retention, and reconciliation process instead of claiming “effectively once” without a boundary.

Failure modes

• Scheduler crash. After lease expiry and takeover, another node loads the shard and scans due state. Recovery latency includes election and backlog work, and every catch-up remains subject to deadline and misfire policy.
• Coordination service unavailable. Nodes may act only while they can establish a still-valid lease and must stop before uncertainty crosses the safety margin. New ownership pauses; the durable job store accumulates due work for later bounded recovery.
• Worker queue down. Occurrence and outbox rows remain durable while publishers back off. Alert on outbox age and apply admission control before recovery overwhelms the queue.
• Worker crash mid-execution. The queue redelivers after its visibility/lease timeout; durable effect deduplication and reconciliation determine whether the outcome is safe.
• Stale owner reaches the store. Its ownership-sensitive transaction is rejected by the fencing predicate and the process relinquishes work. Messages already published can still redeliver, so consumers use the occurrence key.
• Poison job that always fails. Retries with cap then dead-letter, the Part 3 DLQ pattern. Alert on DLQ growth.

Multi-region

Start by asking whether schedules are region-local or global. Region-local ownership limits latency and blast radius. Cross-region takeover requires a control plane and job store whose consistency, replication lag, failover fencing, RPO, and RTO match the promise; “the same lease” is not enough if the data is stale or unreachable. A global occurrence should have one authoritative claim path and a globally unique occurrence key, but external execution can still duplicate across retries. During a partition, prefer an explicit pause over two unfenced writers when correctness matters more than availability.

Evolution path

Day-one needs depend on risk, but a stable occurrence key, durable schedule state, an atomic occurrence/outbox transaction, explicit time semantics, and observable retries are much cheaper to add before clients depend on the contract. Multi-region takeover and elaborate rebalancing can usually wait for measured need.

Observability

Track due-to-occurrence and occurrence-to-start lateness distributions, oldest unpublished outbox age, retry/DLQ volume, duplicate-suppression count, terminal drops, worker latency, lease churn, stale-token rejections, per-tenant throttling, and per-shard load skew. Define SLOs from product classes: a five-second target may suit operational automation and be absurd for monthly billing or best-effort batch work. Measure deadline misses separately from deliberate skip and overlap policies.

Step 6 - Bottlenecks and Trade-offs

• The coordination service sees ownership and renewal traffic in this design, not every occurrence. Size and benchmark it for shard count, churn, outages, and watch traffic rather than choosing a universal shard count.
• The (shard, nextRunAt) index narrows due scans, but batch size, table/index bloat, retention, tenant skew, and concurrent updates still matter. Verify plans and lock behavior on production-like data.
• The aligned cron herd needs capacity, bounded batches, fairness, backpressure, and optional flexible windows; shard count alone does not absorb it.
• Lease churn - frequent ownership changes - signals flaky nodes or too-tight TTLs and is itself a metric.
• Correctness telemetry includes duplicate attempts, suppressed effects, stale-token rejects, expired deadlines, unpublished outbox age, terminal drops, and reconciliation discrepancies. A duplicate attempt does not by itself mean the final effect duplicated.

Reference Architecture

The pattern this problem teaches, reusable far beyond schedulers:

Partition due-time scans, assign one current authority per shard, enforce fencing at the job store, atomically create a uniquely keyed occurrence with an outbox event, and deliver it at least once to consumers that make their effects idempotent and reconcilable.

flowchart LR
    subgraph Coord["Coordination - one owner per shard"]
        L[Lease + fencing token]
    end
    subgraph Tick["Owner's tick loop"]
        S[Scheduler] -->|occurrence + advance + outbox| D[(Durable job store)]
    end
    D --> P[Outbox publisher]
    P --> Q[Worker queue]
    Q --> W[Idempotent workers]
    L -.fences writes.-> D

Figure 5. The lease elects a preferred shard owner, while the durable store enforces its fencing token and commits occurrence, schedule advance, and outbox together. The publisher and workers may retry, so the occurrence key follows the request through every boundary.

The same concerns recur in systems with one current writer per partition, but their protocols are not interchangeable. The transferable questions are: who grants authority, how does the protected resource reject a stale owner, how is durable intent coupled to publication, and how are retries deduplicated at the effect boundary?

This is also the convergence post of the series: the atomic claim comes from Part 2, the durable queue and at-least-once + DLQ from Part 3, the producer-consumer feedback from Part 8, and idempotency at every boundary from Part 11. The scheduler exists at the seam where all of those patterns meet, which is why it is the right closing problem.

Common Mistakes in the Interview

• No coordination, so every scheduler fires every job and duplicates explode.
• A single global leader without a capacity and failover argument, or sharding by reflex when one database-coordinated scheduler would suffice.
• Leader election without fencing tokens, leaving split-brain duplicate dispatch under a partition.
• Fencing only in the coordinator, while the job store and target accept stale requests.
• Advancing nextRunAt before publishing directly to the queue, leaving a database-to-queue dual-write gap.
• Claiming universal exactly-once execution or, conversely, denying exactly-once state changes inside a controlled transaction boundary.
• No misfire policy, so a long outage silently skips runs or fires thousands of catch-ups.
• Letting workers decide whether a job is due, exposing the design to clock skew.
• Storing scheduler state in process memory rather than the durable job store, losing it on every restart.
• Ignoring time zones, daylight-saving transitions, overlap, cancellation races, tenant fairness, or unsafe target URLs.

Quick Reference

Frequently Asked Questions

How do you avoid running the same scheduled job twice?

Give every occurrence a unique key such as (jobId, scheduledAt), and create the run, advance the schedule, and write an outbox record in one database transaction. A publisher retries the outbox until the queue accepts it. Unique constraints suppress duplicate claims, while workers deduplicate durable effects with the occurrence key. Duplicates are still possible at queue and external-effect boundaries, so the scheduler minimises them rather than promising exactly-once execution.

What is leader election in a distributed scheduler?

Leader election selects the current authority for a scheduler shard. A consensus-backed service such as etcd can attach the ownership key to a renewable lease. The process must stop issuing work when it can no longer prove that its lease is valid; after expiry, another node may acquire ownership. Election reduces concurrent claim attempts, while fencing at the job store protects the external resource from a stale former owner.

What is a fencing token and why does a scheduler need one?

A fencing token is a monotonically increasing ownership version. The protected job store must compare it atomically and reject mutations from an older owner; a lock or lease alone cannot fence an external resource. Fencing prevents stale claim-and-advance writes, but it cannot retract a message or side effect already emitted by a previously valid owner, so occurrence keys and idempotent consumers remain necessary.

How do you handle scheduled jobs that were missed during an outage?

Use an explicit per-job misfire policy: catch up within a bounded window, skip to the next occurrence, or fire once and resume. Also define a start deadline and an overlap policy such as allow, forbid, or replace. The right combination depends on the workload: billing may require bounded catch-up, while a heartbeat should usually skip. Never allow an unlimited recovery storm after a long outage.

How do you scale a job scheduler horizontally?

Partition jobs into many more logical shards than scheduler nodes, assign shards to current owners, and query a due-time index inside each shard. Rebalance using measured due-rate and scan cost because hashing job IDs alone does not guarantee even trigger load. Workers scale independently behind the queue. In this design, coordination traffic is driven mainly by ownership changes, shard count, and lease renewals rather than by every job.

Can a job scheduler guarantee exactly-once execution?

It can provide exactly-once state changes only inside a transaction boundary it controls. It cannot generally make an arbitrary external side effect and a queue acknowledgement atomic across failures. Use at-least-once delivery, a stable occurrence key such as (jobId, scheduledAt), durable deduplication or a provider idempotency key, and reconciliation. State the precise boundary instead of claiming universal exactly-once or saying that every useful form is impossible.

Sources

• etcd v3.6 API: transactions, revisions, leases, and keep-alives
• etcd: why an external resource must enforce fencing/version validation
• Kubernetes CronJob: approximate scheduling, concurrency, deadlines, time zones, and idempotency
• AWS EventBridge Scheduler: at-least-once delivery and flexible windows
• AWS EventBridge Scheduler: retry policy and dead-letter queues
• PostgreSQL: SKIP LOCKED is suitable for queue-like tables, not general consistent reads

Related Articles

• System Design Interview Problems: A Senior's Roadmap - the full series index and pattern library.
• System Design Interview Guide: The 6-Step Framework - the method this walkthrough applies.
• Design a Rate Limiter - Part 2; the atomic-claim primitive generalised here to claim-and-advance.
• Design a Notification Service - Part 3; at-least-once dispatch, retries with backoff, and the dead-letter queue.
• Design a Payment System - Part 11; idempotency at every boundary, applied here to scheduled work.
• Design a Unique ID Generator - Part 13; the Tier 5 expansion opens with decentralised ID generation.

This is Part 12, the close of the core 12-part track in a system design series where each post solves one problem around one core pattern. The job scheduler sits where the series' threads converge - leases, atomic claims, durable queues, dead letters, and idempotency at every boundary - which is why it makes a fitting close. Continue with the Tier 5 expansion, or return to the series roadmap to revisit any pattern.