Design a Notification Service: System Design Interview 2026

A notification service looks like a thin wrapper around "send an email" - until you notice that it sits between hundreds of upstream services and a handful of slow, rate-limited, occasionally-down third-party providers, and that a single marketing campaign can ask it to deliver fifty million messages in five minutes. The service exists precisely to absorb that mismatch. This is why it is a favourite senior interview problem: almost every decision is about handling failure and load asymmetry, not about the happy path.

This walkthrough assumes the 6-step system design framework and applies it at the depth expected of a senior or staff candidate. It is Part 3 of a system design series.

Table of Contents

1. The Problem
2. Step 1 - Clarify Requirements
3. Step 2 - Estimate Scale
4. Step 3 - API and Data Model
5. Step 4 - High-Level Design
6. Step 5 - Deep Dive: Queue-Based Asynchronous Processing
7. Step 6 - Bottlenecks and Trade-offs
8. Reference Architecture
9. Common Mistakes in the Interview
10. Quick Reference
11. Frequently Asked Questions
12. Sources
13. Related Articles

The Problem

We are designing a service that delivers notifications to users across multiple channels - mobile push, email, SMS, and in-app - on behalf of many upstream services. An order service wants to confirm a purchase, a social service wants to announce a new follower, a marketing team wants to blast a campaign. They all hand work to one notification service.

The senior framing is that this is an event-driven pipeline bridging a fast, reliable producer side and a slow, unreliable consumer side. The provider boundary - APNs, FCM, an email or SMS gateway - is where latency, rate limits, and outages live. Every important decision in the design is about decoupling from that boundary and staying correct when it misbehaves.

Step 1 - Clarify Requirements

Scope the problem out loud before designing.

Functional requirements:

• Accept a notification request from any upstream service.
• Deliver across multiple channels: push, email, SMS, in-app.
• Render content from templates.
• Respect user preferences: channel choice, opt-outs, and quiet hours.
• Enforce consent, jurisdiction, sender identity, expiry, locale and per-tenant quotas at send time.
• Support two priority classes: transactional (an order confirmation - urgent, must not be lost) and bulk (a marketing campaign - tolerant of minutes of delay).

Out of scope (name them, then defer): the template-authoring UI and click/open analytics. We will note where analytics changes the design.

Non-functional requirements:

• Throughput is spiky. Steady-state load is modest; a campaign produces a massive short burst.
• Reliability. A transactional notification must not be silently dropped.
• Latency is tiered. Transactional delivery should complete in seconds; bulk can take minutes.
• The providers are the weak link. They are slow (100 ms+ per call), rate-limited, and they fail. The system must absorb that.
• Idempotency. Upstream services retry; the same logical notification must not be sent twice.

The decisive question is which boundary each status and guarantee covers. The service can accept a command exactly once in its own database transaction. Queue delivery and provider attempts are normally at-least-once. A provider may accept a request without the service receiving the response, and provider acceptance is not device delivery. Idempotency, attempt leases, expiry, reconciliation and provider-specific keys reduce duplicates and unknowns; do not call the whole path “effectively once”.

Step 2 - Estimate Scale

Make the arithmetic visible; it justifies the queue, the worker count, and the storage.

Treat these as interview assumptions and separate notification commands from per-channel deliveries; one command may produce several deliveries.

Throughput. Assume 1 billion channel deliveries/day.

• Average: 1B / 86,400 s ≈ ~11,600/sec.
• A campaign of 50 million messages pushed in ~5 minutes adds ~165,000/sec on top - so design for a peak near 200,000/sec while the steady state is twenty times lower. This spike-to-average ratio is the whole reason a buffer exists.

Worker count. At a scenario average of 150 ms, one serial slot has a theoretical ceiling near 6.7 calls/sec and 50 in-flight slots near 333/sec before CPU, connection, payload, provider and tail-latency overhead. The steady-state lower bound is about 35 processes, but provider quotas, connection limits, retry load and benchmarked headroom determine the fleet. Scaling workers cannot exceed the provider's allowed throughput.

Storage. 1B × 300 bytes is 300 GB/day of logical row payload before indexes, attempts, callbacks, outbox, encryption, replication, backups and storage-engine overhead. Retention differs by transactional audit, consent, provider reconciliation, privacy and legal requirements. Derive idempotency TTL from the longest upstream retry, internal retry, callback and audited replay horizon.

Step 3 - API and Data Model

The ingestion API is asynchronous by contract. It accepts work and returns immediately - it does not wait for delivery.

POST /api/notifications
  body: { "idempotencyKey": "...", "userId": "...", "templateId": "...",
          "payload": { ... }, "priority": "transactional" }
  202 Accepted   { "notificationId": "..." }

202 Accepted means the command, scoped idempotency record, and dispatch outbox are durably committed; it does not mean the queue already accepted an event or a provider accepted a delivery. 201 Created is also defensible if the API exposes a notification resource. Document the boundary instead of treating HTTP 200 as synonymous with synchronous provider delivery.

The core entities:

A notification moves through a well-defined lifecycle, and naming those states makes the failure handling concrete:

stateDiagram-v2
    [*] --> PENDING: accepted + outbox committed
    PENDING --> SENDING: worker obtains attempt lease
    SENDING --> PROVIDER_ACCEPTED: provider response + ID recorded
    SENDING --> UNKNOWN: timeout or crash around call
    SENDING --> RETRYING: classified transient failure
    RETRYING --> SENDING: budget and backoff allow
    UNKNOWN --> PROVIDER_ACCEPTED: reconcile provider record
    UNKNOWN --> RETRYING: safe retry decision
    RETRYING --> DEAD_LETTER: age or attempts exhausted
    SENDING --> FAILED: permanent error or expiry
    PROVIDER_ACCEPTED --> DELIVERED: channel receipt supports it
    PROVIDER_ACCEPTED --> UNDELIVERED: negative receipt
    DEAD_LETTER --> [*]
    FAILED --> [*]
    DELIVERED --> [*]
    UNDELIVERED --> [*]

Figure 1. Lifecycle of one channel delivery. UNKNOWN is first-class because a timeout or crash can hide whether the provider accepted the call. PROVIDER_ACCEPTED and DELIVERED are different claims, and some channels never provide end-device delivery evidence.

Statuses are provider- and channel-qualified. APNs describes best-effort behavior and acceptance is not proof of display. SMS providers may report carrier or handset-derived delivery states, and callbacks can be delayed, duplicated, reordered or lost. Persist provider IDs, verify callback signatures, apply monotonic transition rules, and reconcile where the provider offers a query API.

Step 4 - High-Level Design

The architecture is a pipeline. Each stage does one thing and is separated from the next by a durable queue.

flowchart TD
    Up[Upstream Services] --> API[Ingestion API]
    API -->|notification + dedup + outbox transaction| Store[(Notification Store)]
    Store --> Pub[Outbox Publishers]
    Pub --> QT[Transactional Queue]
    Pub --> QB[Bulk Queue]
    QT --> WT[Transactional Workers]
    QB --> WB[Bulk Workers]
    WT --> Disp[Channel Dispatchers]
    WB --> Disp
    Disp -->|throttled| Prov[Push / Email / SMS Providers]
    Disp -->|transient fail| RQ[Retry Queue]
    RQ --> Disp
    Disp -->|exhausted / poison| DLQ[Dead-Letter Queue]
    WT -.read current policy.-> Pref[(Preferences / Consent)]
    WB -.read current policy.-> Pref
    Disp -.update.-> Status[(Status Store)]

Figure 2. Ingestion atomically commits the notification, scoped idempotency record, and outbox event. Publishers retry queue delivery. Separate priority lanes reserve capacity, while current consent and channel policy are checked before dispatch.

The ingestion API authenticates the caller, validates a bounded payload/template reference, applies tenant quotas, and commits notification, deduplication and outbox rows in one transaction before returning 202. Publishers deliver at least once to priority lanes. Workers resolve a versioned template, locale, current consent/preferences and quiet-hour schedule, then create idempotent per-channel deliveries. Dispatchers use provider-specific quotas and status rules. Workers can be stateless, but queues, stores, rate-limit state and provider connections are not.

Step 5 - Deep Dive: Queue-Based Asynchronous Processing

This is the core of the problem. Four things make the pipeline correct under load and failure: the decoupling itself, the delivery semantics, the retry and dead-letter machinery, and provider-aware throttling.

Part A - Why the queue is the architecture

Putting a durable queue between the producer and the workers is not an implementation detail; it is the design.

• Acceptance avoids provider latency. The producer waits for the service's durable transaction, not for APNs, FCM, email or SMS. Storage saturation and admission control can still increase latency or reject work.
• The queue buffers within finite limits. Large campaigns should be expanded and paced from campaign state rather than inserting every recipient at once. Workers drain at provider-approved rates; expiry prevents stale messages from consuming capacity forever.
• Age and drain time matter more than depth alone. Track oldest message, arrival/service rate, retry amplification, partition skew and estimated time to recover. Scale workers only while stores and provider quotas have headroom; otherwise throttle bulk ingestion.
• Isolation requires end-to-end budgets. Separate lanes help, but shared stores, rate limiters, templates, provider accounts and callback processors can still propagate an outage.

Part B - Delivery semantics: at-least-once plus idempotency

A durable queue normally gives at-least-once processing. A worker can crash after a provider accepted a call but before the service recorded the provider ID or acknowledged the queue item. Redelivery then faces an ambiguous outcome. Exactly-once state changes are possible inside one database transaction, but the service cannot atomically combine an arbitrary provider call, queue acknowledgement and device display.

• At ingestion. A tenant-scoped idempotency key is inserted atomically with the notification and outbox. The same key and equivalent request return the original resource; the same key with a different payload is a conflict.
• At delivery. A time-bounded lease prevents two workers from calling concurrently, but it does not prove the call happened. Reuse a stable provider idempotency/collapse key only when its documented semantics match the product. Record provider message IDs and a hash/version of the rendered request.
• On ambiguity. Move to UNKNOWN, query or reconcile provider state where possible, and retry only under the channel's duplicate-versus-loss policy. Some channels cannot eliminate the uncertainty.

Retain keys and attempt lineage through the maximum upstream retry, internal retry, provider callback, campaign replay and audit horizon. Bound storage through explicit retention classes, not a universal 24-48 hours.

Part C - Retries, backoff, and the dead-letter queue

Provider failures split into two kinds, and conflating them is a classic mistake:

• Retryable under provider rules - selected timeouts, throttling responses and server errors. Honor Retry-After, preserve an overall retry budget, and remember that a timeout can be an unknown outcome rather than a known failure.
• Permanent or expired - invalid token, malformed payload, revoked consent, unsupported destination, or a message past its usefulness window. Stop and update destination health where appropriate.

Transient retries must use exponential backoff with jitter. Backoff prevents hammering a struggling provider; jitter prevents every failed message in a spike from retrying in lockstep and re-creating the spike.

delay = min(cap, base * 2^attempt) * random(0.5, 1.0)

Cap retries by attempts and event age, using provider guidance and the notification's expiry. A message that exhausts its budget enters a terminal-failure store or DLQ. Replay is a new audited decision: re-check consent, expiry, template availability and idempotency, then create a traceable attempt. Alert on DLQ arrival rate and oldest age, not only total size, and avoid retaining unnecessary PII in queue payloads.

sequenceDiagram
    participant W as Dispatcher
    participant P as Provider
    participant R as Retry Queue
    participant D as Dead-Letter Queue
 
    W->>P: deliver (attempt 1)
    P-->>W: 503 - transient
    W->>R: requeue with backoff + jitter
    R->>W: redeliver after delay
    W->>P: deliver (attempt 2)
    P-->>W: timeout
    Note over W: attempts exhausted (cap reached)
    W->>D: move to dead-letter queue
    Note over D: alert fires on DLQ growth

Figure 3. A provider-classified retry consumes an attempt and age budget. In production, a timeout may first enter UNKNOWN for reconciliation, and a DLQ replay must pass current consent, expiry and idempotency checks.

Part D - Provider rate limits and channel isolation

Providers expose different quotas by project, credential, destination, region or message type. Pace each dimension with a distributed limiter or provider-aware scheduler, honor response feedback such as FCM Retry-After, smooth on-the-hour bursts, and reserve headroom for retries and transactional traffic. A single token bucket per provider is only the starting model.

Channels must also be isolated from one another: a slow SMS gateway must not starve push delivery. Give each channel its own queue and worker pool so one degraded provider cannot consume the whole fleet's capacity.

Priority isolation

Transactional traffic needs reserved end-to-end capacity and stricter age targets. Separate queues and workers are a simple design; weighted-fair scheduling with quotas can also work. Neither prevents a bulk campaign from consuming a shared database, template service, provider quota or callback path. Apply admission control and per-tenant budgets at every shared bottleneck, and do not let “transactional” become an ungoverned priority label.

Consistency model

The status model is a projection of provider evidence. PROVIDER_ACCEPTED means a successful provider response and identifier were durably recorded; a timeout around the call is UNKNOWN, not failed or sent. DELIVERED means only what that channel's documented receipt means - it may be carrier-derived rather than proof that a person saw the message, and APNs is explicitly best effort. Callbacks can be delayed, duplicated, reordered or missed, so verify signatures, process them idempotently with monotonic/versioned transitions, and reconcile provider state where supported.

Failure modes

• Worker crash around delivery. A lease prevents concurrent attempts, but a crash after the provider call creates an unknown result. Reconcile by provider ID/idempotency key or retry under the channel's duplicate-versus-loss policy.
• Queue unavailable. If the notification/outbox transaction still commits, ingestion can return 202 and publishers retry later until the accepted-work age SLO is threatened. If that durable store is unavailable or admission limits are reached, return a retryable error without claiming acceptance.
• Total provider outage. Circuit breakers stop futile calls, bulk intake is throttled, retry age is bounded and transactional backlog consumes reserved capacity. A secondary provider is used only with compatible consent/content/routing and fencing that accounts for ambiguous primary attempts.
• Poison message. Bounded retries isolate it into terminal handling; ordered partitions need a deliberate skip/park policy so one record does not block later work.

Multi-region

Regional queues and workers reduce latency and blast radius, but the acceptance authority must be explicit. A tenant/user home region can keep deduplication local only while routing is stable; failover needs a fenced ownership epoch and replicated notification/outbox state, otherwise both regions may accept the same key. Define RPO/RTO and what happens during a partition. Preferences, consent, suppressions and invalid-token updates need propagation guarantees strict enough for their risk; stale marketing consent should fail closed. Provider credentials, sender identities and quotas may also be region-specific.

Evolution path

Build the accepted-work boundary, scoped idempotency behavior, consent checks and status vocabulary early because callers depend on them. Defer multi-region and multi-provider failover until the measured risk justifies their ambiguity and operational cost.

Observability

Track acceptance latency/errors, oldest outbox and queue age, ingress/service/drain rates, expiry before attempt, delivery latency by priority/channel/provider, response code and Retry-After, unknown outcomes, lease expiry, retries, provider throttling, circuit state, callback lag/signature failures, status reconciliation gaps, invalid destinations, suppression/consent decisions, DLQ rate/age and replay results. Define SLOs per notification class; a 30-second provider-acceptance target may suit one transactional product but is not universal.

Step 6 - Bottlenecks and Trade-offs

• Provider throughput is an external ceiling. Smooth traffic, batch where documented, reserve quota, honor throttling and negotiate capacity; do not evade limits with extra accounts.
• Pipeline throughput includes acceptance store, outbox, queue partitions, preferences, templates, dispatch and callbacks. Partition and isolate based on measured bottlenecks.
• The idempotency store is on the hot path - every notification reads and writes it, so size it for peak QPS the way any shared counter store is sized.
• Campaign spikes versus transactional latency needs reserved end-to-end capacity, tenant quotas and pacing; queue separation alone is insufficient.
• Retry storms are self-inflicted: backoff without jitter synchronises every retry and re-spikes a provider that is already struggling.

Reference Architecture

The pattern this problem teaches, reusable far beyond notifications:

Atomically accept a scoped idempotent command and outbox intent, publish it at least once into capacity-isolated channel lanes, enforce current consent and expiry, pace provider attempts, preserve UNKNOWN outcomes, and reconcile callbacks under an explicit duplicate-versus-loss policy.

flowchart LR
    subgraph Fast["Producer side - fast, reliable"]
        F1[Ingestion API] --> F2[(Durable Queue)]
    end
    subgraph Slow["Consumer side - slow, unreliable"]
        S1[Workers] --> S2[Rate-limited dispatch]
        S2 --> S3[External providers]
    end
    F2 --> S1
    S2 -.retry / backoff.-> S1
    S2 -.exhausted.-> DLQ[(Dead-Letter Queue)]

Figure 4. The queue buffers a finite mismatch between accepted work and provider capacity. Correctness also requires an atomic acceptance/outbox boundary, provider-specific attempt state, current policy checks, bounded retries and audited terminal handling.

The same shape helps at unreliable external boundaries, but it does not make them exactly once. Always name the durable acceptance point, ambiguous side-effect window, expiry, reconciliation source and replay authorization.

Common Mistakes in the Interview

• Claiming exactly-once end-device delivery, or claiming every exactly-once state change is impossible without naming a boundary.
• Synchronous provider calls in the request path, defeating the entire point of the service.
• Retrying without jitter, which synchronises retries into a storm that re-overwhelms the provider.
• One queue for transactional and bulk traffic, letting a marketing campaign delay order confirmations.
• No dead-letter queue, so a poison message either blocks the pipeline or is retried forever.
• Ignoring per-provider rate limits, which gets the platform throttled or banned.
• Not deduplicating, so an upstream retry delivers the same notification several times.
• Using a permanent send marker before the provider call, which trades duplicates for lost sends after a crash.
• Treating provider acceptance as device delivery, or ignoring UNKNOWN, callback authentication/order and reconciliation.
• Checking opt-out only at ingestion, missing consent, block or token changes before delayed delivery and DLQ replay.

Quick Reference

Frequently Asked Questions

Why does a notification service need a message queue?

A durable queue decouples acceptance from slow, quota-limited provider calls and gives workers controlled retries, fairness, and backpressure. It does not make capacity infinite or isolate upstream automatically: the service still needs admission control, queue-age limits, reserved transactional capacity, and overload behavior. Commit the notification, idempotency record, and outbox event atomically, then publish to channel queues at least once so a queue outage cannot lose accepted work.

Can a notification service guarantee exactly-once delivery?

Not for an arbitrary provider call and end-device display. A claim before the call can lose a send after a crash; retrying an expired claim can duplicate a call whose response was lost. Use a leased attempt to prevent concurrent workers, pass a stable provider idempotency or collapse key when the channel supports the required semantics, record provider IDs, represent ambiguous outcomes explicitly, and reconcile callbacks or polling. Promise exactly-once only inside a transaction boundary you control.

How do you handle a notification provider failing?

Classify outcomes using each provider's documented codes. Retry eligible timeouts, 429s and server errors with a retry budget, exponential backoff and jitter; honor Retry-After and stop when the notification is no longer useful. Do not retry invalid tokens, malformed payloads or revoked consent. A circuit breaker and admission control protect the provider and backlog. Failover is safe only when routing is fenced and an ambiguous primary attempt cannot create an unacceptable duplicate.

What is a dead-letter queue and why is it needed?

A dead-letter queue or terminal-failure store retains messages that exceeded retry age or attempts, failed validation, or hit a non-retryable processing error. It prevents infinite retry loops and supports diagnosis, but it is not an archive or automatic recovery plan. Encrypt and limit sensitive payloads, set retention, alert on rate and age, and require an audited replay workflow that revalidates consent, expiry and idempotency before creating a new attempt.

How do you stop a marketing campaign from delaying transactional notifications?

Give transactional work reserved end-to-end capacity and separate backlog controls. Separate queues and worker pools are a simple option; weighted-fair scheduling with quotas can also work when implemented and tested. Isolation must include shared stores, template rendering, provider quotas and callback processing, not only the first queue. Throttle or pause bulk campaigns before they consume the transactional latency and provider budget.

How do you prevent users from receiving duplicate notifications?

Scope an ingestion idempotency key by tenant and operation and persist it atomically with the accepted notification. Give every channel delivery a stable attempt lineage and use leases only to prevent concurrent workers, not as proof that a call happened. Reuse a provider idempotency or collapse key where its semantics fit, deduplicate callbacks, and reconcile unknown outcomes. Retain keys for the maximum retry, callback, replay and upstream-retry horizon derived from the product, not a universal 24-48 hours.

Sources

• Apple Developer: APNs best-effort delivery, expiry, throttling and collapse behavior
• Firebase: FCM scaling, Retry-After, backoff, jitter and traffic smoothing
• Firebase: FCM error classification and retry guidance
• Twilio: outbound accepted, queued, sent, delivered and undelivered statuses
• Twilio: persistent status logging, callback gaps and reconciliation

Related Articles

• System Design Interview Problems: A Senior's Roadmap - the full series index and pattern library.
• System Design Interview Guide: The 6-Step Framework - the method this walkthrough applies.
• Design a Rate Limiter - Part 2; the token-bucket primitive reused here to pace provider calls.
• Design a Distributed Cache - Part 4; consistent hashing for the partitioned stores this pipeline relies on.
• Apache Kafka Interview Questions - the durable queue and consumer-group mechanics behind this pipeline.

This is Part 3 of the core track in a 16-part system design series. Next: Design a Distributed Cache.