Design a Payment System: System Design Interview 2026

A payment system is a correctness- and evidence-heavy workflow across boundaries that cannot share one transaction. Authorization, capture, settlement, refund, fee, payout and chargeback are different events; a provider timeout may leave the outcome unknown. The design therefore combines explicit state machines, scoped idempotency, an immutable accounting journal, durable event intent, provider webhooks/query APIs and reconciliation. It reduces duplicate or lost effects and makes remaining uncertainty operationally visible rather than promising magic exactly-once behavior.

This walkthrough assumes the 6-step system design framework and applies it at senior-plus depth. It is Part 11 of a system design series.

Table of Contents

1. The Problem
2. Step 1 - Clarify Requirements
3. Step 2 - Estimate Scale
4. Step 3 - API and Data Model
5. Step 4 - High-Level Design
6. Step 5 - Deep Dive: Idempotency, the Outbox, and Sagas
7. Step 6 - Bottlenecks and Trade-offs
8. Reference Architecture
9. Common Mistakes in the Interview
10. Quick Reference
11. Frequently Asked Questions
12. Sources
13. Related Articles

The Problem

We are designing a payment system: a customer pays a merchant, the platform takes a fee, and the money moves through an external payment provider that ultimately settles with the bank rails. The canonical examples are Stripe and the payment subsystem inside any marketplace or platform.

Correctness dominates, but availability, latency, fraud, compliance and operational recovery still shape the product. The hardest boundary is an external operation whose response is lost. The system needs one durable workflow per logical payment, stable provider references and a state such as UNKNOWN or PROCESSING until authoritative evidence arrives.

Step 1 - Clarify Requirements

Functional requirements:

• Process a payment: debit a customer, credit a merchant, take a platform fee.
• Issue refunds.
• Integrate with an external payment provider that talks to the bank rails.
• Expose transaction history.

Out of scope (name, then defer): fraud detection, KYC, fiat conversion, the customer-facing payment UI, and DRM-style chargeback handling beyond a basic refund flow.

Non-functional requirements:

• Correctness and controlled availability. Reject or pause unsafe transitions instead of creating two authorities; make unknown outcomes visible and recoverable.
• Idempotency at every boundary that can retry.
• Auditability. Every state change recorded, no in-place edits.
• Strong transaction boundaries and invariants around each owned ledger/payment partition; cross-system status is reconciled.
• Modest throughput, but every operation matters.
• Security and compliance scope - tokenise payment methods, minimise cardholder-data exposure, use current PCI DSS requirements where applicable, encrypt, segregate duties and retain evidence according to jurisdiction.

Clarify the boundary instead of saying “exactly once”. The service can create one internal payment under a unique constraint and post one ledger transaction in a controlled database transaction. It cannot atomically combine that commit with an arbitrary external provider and bank rail. Provider idempotency, webhook deduplication, status queries and reconciliation narrow the uncertainty; the API must expose processing or unknown rather than infer success or failure.

Step 2 - Estimate Scale

The numbers here are deliberately small compared to the rest of the series, because the difficulty is per-operation, not per-second.

Treat these as candidate assumptions. Throughput: 10 million payments/day is ~116/sec average; use 500/sec as the interview peak and separately estimate webhook, refund, ledger, reconciliation and retry traffic.

Storage. A 2 KB logical estimate gives ~20 GB/day before indexes, multiple attempts, journal postings, outbox/webhook records, encryption, replication, backups and audit evidence. Retention is defined by business and jurisdiction, not one universal number.

Latency. Use 100-500 ms as a scenario for synchronous card-provider calls, while authentication and asynchronous payment methods can remain processing for much longer. Separate API acknowledgement, authorization/capture and settlement SLOs.

The arithmetic suggests correctness and external dependency behavior dominate this scenario, but benchmark the ledger's hottest account partitions, webhook bursts, reporting and reconciliation scans before dismissing capacity.

Step 3 - API and Data Model

POST /api/payments
  Idempotency-Key: <uuid>
  body: { customerId, merchantId, amountMinor, currency, paymentMethodToken }
  201 Created   { paymentId, status: "requires_action" | "processing" | "authorized" | "captured" | "failed" }
 
POST /api/payments/{id}/refund
  Idempotency-Key: <uuid>
  body: { amount }
 
GET  /api/payments/{id}   -> the payment state

The data model has four pieces, and the relationships between them are the design:

Represent money in integer minor units (or an exact decimal model tied to currency exponent), never binary floating point. Journal postings balance under the chart-of-accounts rule for one currency; multi-currency conversion needs explicit legs, rates and gain/loss accounts. Serving balances may be materialised transactionally for speed but must reconcile to the immutable journal. Payment state separates authentication, authorization, capture, settlement, partial/full refund, dispute/chargeback and terminal failure. An ambiguous provider call is UNKNOWN/PROCESSING, not a guessed result.

Step 4 - High-Level Design

flowchart TD
    Client([Client]) -->|idempotency key| PS[Payment Service]
    PS -->|atomic: payment + idempotency + command outbox| DB[(Transactional DB)]
    DB --> Pub[Outbox Publisher]
    Pub --> Cmd[Payment Command Queue]
    Cmd --> Orch[Payment Orchestrator]
    Orch -->|stable provider operation key| Prov[External Payment Provider]
    Prov -->|response| Orch
    Orch -->|atomic: outcome + ledger + event outbox| DB
    Prov --> Hook[Verified Webhook Inbox]
    Hook --> DB
    Pub --> Bus[Event Bus]
    Bus --> Notif[Notification Service]
    Bus --> Fulfil[Fulfilment Service]
    Bus --> Analytics[Analytics]
    Recon[Reconciliation Service] -.read.-> DB
    Recon -.read provider reports.-> Prov
    Recon -->|exceptions / approved adjustments| DB

Figure 1. Acceptance atomically creates the payment, scoped idempotency record and durable command intent. The orchestrator calls the provider with a stable operation key, then atomically records the observed outcome, appropriate ledger postings and downstream event intent. Verified webhooks and reconciliation can advance the same versioned state machine.

The first transaction inserts the tenant-scoped idempotency record, canonical request hash, CREATED/PROCESSING payment and provider-command outbox. An orchestrator consumes that command, creates or resumes the provider object with a stable operation key, and records the response. Only then does a second transaction advance state, post the accounting event appropriate to authorization/capture/refund, and add downstream outbox events. A timeout remains unknown until a provider query, verified webhook or reconciliation resolves it. Every transition uses optimistic versioning and a legal state-machine guard.

Step 5 - Deep Dive: Idempotency, the Outbox, and Sagas

This is the core. Five mechanisms cover different failure windows: scoped idempotency, an immutable accounting journal, transactional outbox intent, durable saga/orchestration state with semantic compensations, and reconciliation across independent records.

Part A - Idempotency at every boundary

A retry that re-processes is the most common way payment systems double-charge. The fix is to make every boundary that can be retried idempotent, which means recognising a retry as the same logical operation and returning the original result instead of redoing the work.

Client to service. The client generates an Idempotency-Key per logical operation. In one transaction, the service inserts the scoped key, canonical request hash, payment record and command outbox. The same key plus same request returns the current payment, whether it is processing, unknown or complete; the same key plus different parameters is rejected. Retention follows the maximum supported client retry and workflow/reconciliation horizon.

sequenceDiagram
    participant C as Client
    participant PS as Payment Service
    participant DB as Transactional DB
    participant O as Orchestrator
    participant Prov as Payment Provider
 
    C->>PS: POST /payments (key=K)
    PS->>DB: tx: K + request hash + payment + command outbox
    DB-->>PS: committed payment P (PROCESSING)
    PS-->>C: 201 Created (P, PROCESSING)
    O->>DB: consume command for P
    O->>Prov: create/confirm (provider key T)
    Prov-->>O: accepted (providerIntentId)
    O->>DB: tx: outcome + postings + event outbox
    Note over C: network timeout - retry
    C->>PS: POST /payments (key=K, again)
    PS->>DB: lookup K and compare request hash
    DB-->>PS: same payment P and current status
    PS-->>C: 201 Created (P, same workflow)

Figure 2. The idempotency key and payment are one transaction, so a retry always finds the same workflow even before provider work completes. The provider operation has its own stable key and the result, ledger postings and downstream event intent commit only after evidence of that transition.

Service to provider. Use a distinct stable key for each provider operation (create intent, confirm, capture, refund) and follow that provider's scope, parameter-comparison and retention rules. Stripe API v1, for example, permits pruning keys after at least 24 hours; reusing a pruned key can create a new operation. Persist the provider object ID and query it on ambiguity. If the provider lacks a suitable idempotency contract, do not blind-retry a charge: route the outcome through investigation/reconciliation policy.

This is the same at-least-once-plus-idempotency reasoning from Part 3, but here it is layered at every boundary instead of just one, and the consequence of getting it wrong is real money rather than a duplicate email.

Part B - The append-only double-entry ledger

A real payment system keeps an immutable accounting journal. Each recognised accounting event posts debits and credits under a defined chart of accounts; a per-currency invariant balances the transaction according to that model. Serving balances are commonly materialised for performance, but they are derived state with sequence/version and reconciliation back to journal postings.

This carries three properties no mutable column gives you:

• Auditability. Every change is a row, with a reason, and history is replayable.
• Self-checking. Balanced transaction, currency, account and sequence invariants catch classes of bugs; a balanced but semantically wrong posting is still possible.
• Correctability without rewriting history. A mistake is fixed with new compensating entries that reverse the bad ones - the bad entries stay, annotated.

This is the standard pattern in real financial systems and is the same idea event-sourced architectures generalise to other domains.

Part C - The transactional outbox

Once a payment is committed, downstream services need to know - notifications, fulfilment, analytics. The naive approach commits the payment, then publishes to a message bus. If the publish fails, the database and the bus disagree forever (a dual-write problem), and there is no way to atomically span the two.

The transactional outbox sidesteps this. In the same database transaction that writes the payment and the ledger entries, the service writes one row per outgoing event to an outbox table. A separate outbox publisher polls the outbox and emits the events to the bus, marking each as published.

Because publication intent is in the transaction, it cannot disappear between the database commit and broker call. If the publisher crashes after broker acceptance but before marking the row, it republishes. Consumers therefore deduplicate stable event IDs and apply aggregate version/order checks. The outbox closes the lost-intent dual-write gap; it does not provide exactly-once delivery or make every downstream projection current.

Part D - Sagas for cross-service work

A single payment can touch several services and external systems. A local ACID transaction cannot span a provider that does not participate in the same commit protocol. Two-phase commit may be valid among controlled compatible resources, but it introduces coordination and blocking/failure trade-offs and is unavailable across typical payment-provider APIs.

A saga models a workflow as durable local transitions. Failure may trigger forward recovery or a semantic compensation such as release authorization, refund, or inventory release. Compensation is a new external action that can fail, arrive late, cost money, or be impossible after fulfilment; it is not rollback.

flowchart TD
    S1[1. Authorize<br/>provider operation] -->|ok| S2[2. Reserve inventory<br/>local transaction + outbox]
    S2 -->|ok| S3[3. Capture<br/>provider operation]
    S3 -->|ok| S4[4. Record capture postings<br/>+ completion outbox]
    S1 -.order cancelled before capture.-> C1[Release authorization]
    S2 -.inventory reservation fails.-> C2[Release authorization<br/>and mark failed]
    S4 -.fulfilment fails after capture.-> C3[Business decision:<br/>retry fulfilment or refund]

Figure 3. Saga outcomes depend on state and business policy. Before capture, release may compensate an authorization; after capture and fulfilment failure, retrying fulfilment may be safer than an automatic refund. Publishing completion is an outbox concern and is never a reason to reverse a valid payment.

Every step and compensation needs its own operation key, deadlines, legal transitions and recovery owner. Choreography can fit simple independent reactions; orchestration is often clearer when one payment state machine, provider ambiguity and manual intervention must be visible. Neither style creates atomicity. Two-phase commit remains an option only among controlled compatible participants whose blocking and failure behavior the product accepts; external processors normally do not participate.

Part E - Reconciliation - the safety net

Independent records can disagree because a provider call timed out, a webhook was missed, settlement was delayed, a mapping changed or a bug posted the wrong account. Reconciliation compares these books rather than declaring one system universally authoritative.

Use stable provider and internal references to match payment/attempt state, journal postings, provider objects/reports and bank settlement by amount, currency, date and lifecycle. Separate expected timing differences with aging windows from true breaks. Exceptions enter an auditable workflow; some unknown statuses can be resolved automatically from provider state, while monetary adjusting entries require reason codes, approvals and segregation of duties. Never “repair” an unexplained difference merely to make totals match.

Failure modes

• Provider call times out. Keep UNKNOWN/PROCESSING, query the persisted provider object, accept verified webhooks and retry with the same still-valid operation key only under documented semantics. Never infer failure and generate a new charge key.
• DB update fails after provider success. The payment and attempt already exist before the call. Resume by provider ID/key, then atomically record the observed outcome, journal postings and outbox. Reconciliation catches attempts that remain unknown beyond their age SLO.
• Outbox publisher down. Intent accumulates durably; alert on oldest age. Recovery may publish duplicates, so consumers deduplicate and enforce aggregate versions.
• Concurrent updates on one account. Row/advisory locks, serializable transactions or optimistic versions protect the local invariant; hot accounts need measured partitioning without splitting one invariant across authorities.
• Webhook duplicated or reordered. Verify signature and endpoint version, deduplicate event IDs/object transitions, retrieve the current provider object when required, and apply only legal monotonic state transitions.
• Discovered fraud or bug. Preserve history, investigate scope, and post approved reversing or adjusting entries with reason and authorization; do not silently rewrite journal rows.

Multi-region

Choose an authoritative region/partition per account or payment and fence failover so two regions cannot advance the same workflow. Synchronous replication can target RPO 0 for specific committed records but increases latency and may reduce write availability; it does not protect against application bugs, operator error or correlated corruption. Define tested RPO/RTO per ledger, payment, webhook and outbox data, keep immutable backups, and decide explicitly whether a partition pauses writes. Cross-region merchant/customer accounting may require due-to/due-from accounts and a controlled settlement process, not an informal saga alone.

Evolution path

Define scoped idempotency, exact money representation, payment/attempt state machines, journal rules, provider references, webhook inbox and outbox before production money moves. Defer multi-region and elaborate orchestration until requirements justify them, but rehearse unknown-outcome and reconciliation procedures early.

Observability

Track transitions and latency by method/provider/region; authentication, authorization, capture, refund and settlement success; unknown outcome count/age; duplicate-key conflicts; provider/webhook errors and ordering gaps; oldest outbox age; consumer dedup; journal invariant violations; materialised-balance drift; reconciliation unmatched amount/count by age and reason; manual adjustments; compensation failures; hot-account contention; and DR replication/failover health. SLOs differ for cards and asynchronous bank methods. Zero unexplained aged breaks is a goal; transient unmatched items are expected before cutoff and should not all be labelled incidents.

Step 6 - Bottlenecks and Trade-offs

• Provider latency dominates per-payment time and is largely out of your control - keep the rest of the path lean.
• Provider rate limits call for a per-provider token-bucket limiter - exactly the primitive from Part 2.
• Per-account contention protects invariants but can create hot accounts; measure lock/serialization retries and design explicit sub-ledgers only when accounting can reconcile them.
• Outbox lag is the visible health signal for the event pipeline; alert on growth.
• Reconciliation breaks need aging, materiality and reason classification; unexplained or overdue differences trigger controlled incident/exception workflows.

Reference Architecture

The pattern this problem teaches, reusable far beyond payments:

Create one durable workflow per logical command, use stable provider operation identities, record each observed monetary transition in an immutable balanced journal with outbox intent, preserve unknown outcomes, and reconcile independent internal/provider/bank records through an auditable exception process.

flowchart LR
    subgraph Atomic["Atomic transaction"]
        A1[State change] --- A2[Ledger entries] --- A3[Outbox event]
    end
    Atomic --> Pub[Outbox publisher]
    Pub --> Bus[Event bus]
    Atomic <-->|saga steps + compensations| Ext[External systems]
    Recon[Reconciliation] -.compare.-> Atomic
    Recon -.compare.-> Ext
    Recon -->|compensating entries| Atomic

Figure 4. Each owned state transition commits its payment state, accounting postings when applicable, and outgoing intent atomically. External calls remain outside that transaction, so stable operation IDs, UNKNOWN states, webhooks/status queries and reconciliation bridge the boundary.

The reusable lesson is to distinguish owned atomic transitions from external evidence. Idempotency, outbox, saga and reconciliation solve different failure windows; none substitutes for the others or makes irreversible effects roll back.

Common Mistakes in the Interview

• Claiming exactly-once end to end without naming the controlled transaction and external-provider boundaries.
• Dual-write - committing to the DB and then publishing to a bus separately - opening a permanent inconsistency window.
• Proposing two-phase commit across services and an external provider, which the provider does not actually support.
• Treating a mutable balance as authority, or claiming materialised balances are forbidden instead of reconciling them to the journal.
• Retrying a provider call without an idempotency token, the canonical way to double-charge.
• No reconciliation across internal ledger/payment state, provider records and settlement/bank statements.
• Treating a provider timeout as a definite failure, then retrying and double-charging.
• A saga without compensations, or with non-idempotent ones that fail on retry.
• Treating compensation as rollback, automatically refunding on an event-publication failure, or ignoring irreversible fulfilment.
• Using binary floating point for money, omitting currency/exponent, or mixing currencies in one balance invariant.
• Blindly applying reconciliation adjustments instead of aging timing differences and approving explained journal entries.

Quick Reference

Frequently Asked Questions

How do you prevent double charges in a payment system?

Atomically create one payment and idempotency record for a tenant-scoped key plus request hash. The same key and equivalent request resumes or returns that payment; a different request is rejected. Call the provider with a stable operation-specific idempotency key when its documented retention and parameter rules cover the retry. Persist provider IDs and represent timeouts as UNKNOWN until queried or reconciled. A database constraint, provider idempotency and reconciliation are complementary, not an absolute end-to-end guarantee.

What is an idempotency key?

It identifies one logical command within a documented scope such as tenant, endpoint and operation. Store it with a canonical request hash, payment ID, status and response in the same transaction that creates the payment. A retry may return a completed result or resume an in-progress or unknown workflow; it must not blindly return an empty placeholder. Retention follows client, internal and provider retry/reconciliation horizons, and reusing a key with different parameters is an error.

What is the transactional outbox pattern?

Write an outbox row in the same database transaction as the payment or ledger transition, then publish it asynchronously. This makes publication intent durable without a database-to-broker dual write. The publisher can crash after the broker accepts an event but before publishedAt is recorded, so duplicate publication remains possible. Consumers need stable event IDs, idempotency and ordering/version checks; monitor oldest unpublished age and reconcile stuck rows.

When should you use a saga instead of two-phase commit?

Use a saga when independently committed services or providers cannot share one transaction and the business accepts intermediate states plus forward recovery or semantic compensation. Compensation is a new fallible action such as refund or release, not an atomic undo, and some effects are irreversible. Two-phase commit can be valid among controlled compatible participants, but an external payment provider usually does not join it. For money flows, orchestration, durable state, idempotent steps and manual recovery are often clearer than event-only choreography.

Why use a double-entry ledger as the source of truth for money?

An immutable journal records each business transaction as balanced debit and credit postings under a defined chart of accounts, currency and accounting rule. It provides an audit trail and lets invariant checks detect imbalance. Serving balances can be materialised for performance but must be derived and reconcilable with the journal. Authorization, capture, settlement, refund, fee and chargeback are distinct events and accounts; corrections use approved reversing or adjusting entries rather than history edits.

What is reconciliation in a payment system?

Reconciliation matches independent records from the internal payment state and ledger, provider objects and reports, and bank or settlement statements using stable references, amount, currency and business date. Expected timing differences are aged separately from true breaks. Missing, duplicate or mismatched items enter an auditable exception workflow; some can be auto-resolved from provider state, while monetary adjustments require controlled approval. Reconciliation detects drift but does not make the provider the sole truth for internal accounting.

Sources

• Stripe API: idempotency-key scope, parameter comparison and retention
• Stripe: PaymentIntent lifecycle, authentication, processing and capture
• Stripe: webhook signatures, duplicates, retries and unordered events
• Stripe: payment status updates and asynchronous fulfilment
• PCI Security Standards Council: current PCI DSS v4.x resources

Related Articles

• System Design Interview Problems: A Senior's Roadmap - the full series index and pattern library.
• System Design Interview Guide: The 6-Step Framework - the method this walkthrough applies.
• Design a Notification Service - Part 3; at-least-once delivery and idempotency at one boundary - here generalised to every boundary.
• Design a Rate Limiter - Part 2; the per-provider token bucket reused to respect provider limits.
• Design a Job Scheduler - Part 12; idempotency at every boundary applied to scheduled work, with leader election added.
• SQL Interview Questions - transactions, isolation levels, and the database guarantees the outbox depends on.

This is Part 11 of the core track in a 16-part system design series. Next: Design a Job Scheduler.