Design a Chat System: System Design Interview 2026

A large chat system inverts the usual request-response architecture. Its gateways hold long-lived connections and push events to whichever devices are online. The data model—messages in conversations—looks simple, but the hard parts are connection routing, durable hand-off, ordering, multi-device state, recovery, and deciding which guarantees still hold during regional failure.

This walkthrough assumes the 6-step system design framework and applies it at senior depth. It is Part 6 of a system design series.

Table of Contents

1. The Problem
2. Step 1 - Clarify Requirements
3. Step 2 - Estimate Scale
4. Step 3 - API and Data Model
5. Step 4 - High-Level Design
6. Step 5 - Deep Dive: Real-Time Delivery and Connection Routing
7. Step 6 - Bottlenecks and Trade-offs
8. Security and Abuse Controls
9. Reference Architecture
10. Common Mistakes in the Interview
11. Quick Reference
12. Frequently Asked Questions
13. Sources
14. Related Articles

The Problem

We are designing a real-time chat system supporting one-to-one and group messaging, with delivery and read receipts, presence, and message history - the shape of WhatsApp, Messenger, or Slack.

The senior framing is that this is a routing problem over a stateful connection layer. Unlike every prior system in this series, the front tier is not stateless: a connection server owns the live WebSocket connections of the users attached to it. Delivering a message means finding which server holds the recipient and getting the message there - and staying correct when a server holding a hundred thousand connections suddenly dies.

Step 1 - Clarify Requirements

Functional requirements:

• One-to-one messaging and group messaging.
• Real-time delivery when the recipient is online.
• Offline delivery: store messages for an offline recipient, deliver on reconnect.
• Delivery and read receipts.
• Presence: online / offline / last-seen.
• Message history.

Out of scope (name, then defer): media and file storage, end-to-end encryption details, and voice/video calls.

Non-functional requirements:

• Real-time latency. Delivery to an online recipient within ~200 ms.
• Massive concurrent connections. Hundreds of millions of simultaneous persistent connections.
• Reliability. A successful send ACK means the message is durably committed according to the chosen replication policy. Delivery is at-least-once with deduplication and gap repair; regional-disaster loss is governed by the declared RPO.
• Per-conversation ordering. Messages in one conversation appear in a consistent order; global ordering is not required.

The clarifying questions that shape the design: the client boundary uses retries plus idempotency rather than a literal exactly-once promise. Ordering is per-conversation only. Multi-device delivery must be defined: should a message reach every active device, and is “delivered” a per-device or per-user state? And group size matters: a small group and a 100,000-member broadcast channel need different fan-out, the same way Part 5's celebrity accounts did.

Step 2 - Estimate Scale

These are interview assumptions, not industry benchmarks. State them, calculate consistently, then replace them with measured connection, CPU, memory, and network limits during capacity testing.

Connections. Assume 1 billion users and 500 million online at peak, with one active connection each for a first-pass estimate: 500 million concurrent connections. If a tested gateway safely holds 100,000 connections after headroom, the connection layer needs roughly 5,000 gateways. Multiple devices, rolling deploys, regional spare capacity, and uneven load increase that number.

Messages. At ~40 messages/user/day, that is 40 billion messages/day ≈ ~460,000 messages/sec average, with peaks past 2 million/sec.

Storage. At an assumed 300 bytes per message, 40B/day is ~12 TB/day of raw message payload and metadata. Indexes, replicas, encryption overhead, edit/delete history, receipts, backups, and attachments are additional.

Connection memory. At an assumed 10 KB of application and runtime state per connection, 500M connections is ~5 TB across the gateway fleet. Benchmark the actual runtime and TLS/network buffers rather than treating 10 KB or 100,000 sockets as constants.

Two numbers define the problem: 500 million persistent connections, and the fact that a message must hop between two arbitrary servers among 5,000.

Step 3 - API and Data Model

Messaging does not use request-response REST - it uses a persistent WebSocket carrying typed frames: SEND, ACK, RECEIPT, PRESENCE, TYPING. A thin REST endpoint serves history: GET /conversations/{id}/messages?cursor=<opaque>.

Two IDs do two jobs. The messageId is generated by the client and used for idempotency—a retried send carries the same ID, with a uniqueness constraint such as (senderId, messageId). The seq is a monotonic per-conversation sequence number assigned by the authoritative partition and used for ordering and gap detection. Authorization, ID allocation, message insertion, and the durable delivery event must share one atomic boundary or an outbox/log design; otherwise a crash between “stored” and “published” can strand an online delivery. Messages are partitioned by conversationId, so one extremely hot conversation can become a hot partition.

Step 4 - High-Level Design

flowchart TD
    CA([User A]) <-->|WebSocket| SA[Connection Server A]
    CB([User B]) <-->|WebSocket| SB[Connection Server B]
    SA --> MS[Message Service]
    MS -->|atomic message + outbox commit| Store[(Message Store<br/>partitioned by conversation)]
    Store --> DW[Delivery Workers]
    SA -->|register endpoints / renew leases| Reg[(Connection Registry)]
    SB -->|register endpoints / renew leases| Reg
    DW -->|lookup all recipient endpoints| Reg
    DW -->|publish by gateway| BP[Pub/Sub Backplane]
    BP --> SA
    BP --> SB
    SB -.push.-> CB
    SA -.presence.-> Pres[(Presence Store - TTL)]

Figure 1. Stateful gateways own live client connections. The registry maps a user to every leased connection endpoint, delivery workers consume committed message events, and the backplane routes batches to the gateways that currently own those endpoints. Durable history makes a dropped push repairable by resync.

The connection layer is a fleet of stateful gateways. The connection registry maps each user to a set of live endpoints because a phone, desktop app, and browser tab may sit on different gateways. The message service commits the message and a delivery event atomically through a transactional outbox or an append-only log. Delivery workers resolve conversation membership and online endpoints, then publish gateway-addressed batches over the backplane. Presence lives in leased, TTL-backed records. A missed push is acceptable because durable history plus cursors can repair it; a lost accepted message is not.

Step 5 - Deep Dive: Real-Time Delivery and Connection Routing

This is the core. Four things make real-time chat work: the transport, cross-server routing, presence, and ordered reliable delivery.

Part A - The transport

The transport choice depends on the product and client environment. HTTP polling repeatedly asks “anything new?”, adding request overhead and up to roughly one poll interval of detection latency. Long polling holds a request until data or a timeout and can batch events, but still renews requests. Server-Sent Events provide server-to-client streaming while sends use ordinary HTTP, which can be a sound simpler design. WebSocket provides a persistent, bidirectional message channel and is a strong fit when the same client exchanges frequent low-latency events such as messages, receipts, and typing indicators.

The price is architectural: the gateway holds live connection state. That drives the registry, backplane, flow control, draining, and reconnect story below. The browser WebSocket API does not provide stream-style backpressure, so gateways and clients also need bounded outbound queues, message-size limits, slow-consumer policy, and resync rather than unbounded buffering.

Part B - Connection routing

With a large gateway fleet, sender and recipient endpoints usually do not share a server. Routing works in five steps:

1. The sender's gateway authenticates the connection, validates size/rate limits, and the message service authorizes conversation membership.
2. The message service atomically commits the message, its per-conversation sequence, the idempotency record, and a durable delivery event; only then does it acknowledge acceptance.
3. A delivery worker resolves recipients and looks up all current endpoints for those users.
4. It batches the event by gatewayId and publishes over the backplane.
5. Each gateway pushes to its local sockets. Missing, stale, or slow endpoints recover through cursor-based sync.

sequenceDiagram
    participant A as User A
    participant SA as Server A
    participant MS as Message Service
    participant DL as Delivery Log
    participant R as Registry
    participant BP as Backplane
    participant SB as Server B
    participant B as User B
 
    A->>SA: SEND (messageId, content)
    SA->>MS: authorize + idempotent create
    MS->>DL: atomic message + delivery event
    DL-->>MS: committed (seq)
    MS-->>SA: accepted (seq)
    SA-->>A: ACK (persisted)
    DL->>R: resolve User B endpoints
    R-->>DL: Server B / connection IDs
    DL->>BP: publish gateway batch
    BP->>SB: deliver
    SB->>B: push over WebSocket
    B->>SB: RECEIPT (delivered)

Figure 2. The acceptance ACK follows the durable commit. Delivery then proceeds asynchronously from the committed event through endpoint lookup and a gateway-addressed backplane. If any post-commit step fails, the worker retries and the client can still repair a gap from history.

Each gateway registers a distinct endpoint on connect and periodically renews its lease. Deletion and renewal include a connection or generation token, so a delayed disconnect from an old socket cannot erase a newer session. TTL expiry cleans up crashed gateways; lifecycle events are merely an optimization because delivery correctness comes from history and resync. The backplane decouples gateways: each subscribes only to traffic addressed to that gateway and does not need peer-to-peer knowledge of the fleet.

Part C - Group messaging and fan-out

A 1:1 message may still target several device connections. A group of N members creates up to N user deliveries and potentially more endpoint pushes. Store the message once in conversation history; maintain lightweight inbox/change entries and read/delivery cursors rather than copying full message content for every offline member. Deduplicate gateway batches when many recipients share one gateway.

Large broadcast channels—the celebrity problem from Part 5 in a new guise—need a different threshold-driven policy. Do not block the sender on synchronous fan-out to 100,000 members. Commit once, enqueue asynchronous fan-out, optionally push a lightweight “conversation changed” hint to online subscribers, and let clients pull ordered history. The exact push/pull threshold comes from workload measurements.

Part D - Presence

Presence is deceptively expensive. Broadcasting every connect and disconnect to all of a user's contacts produces a presence storm - at 500M users churning connections, the fan-out dwarfs the actual messaging traffic.

The scalable approach treats each connection as a leased record with a short TTL. A user is online while at least one unexpired device lease exists. Presence is queried for visible contacts or delivered through bounded, coalesced subscriptions rather than broadcast globally. It remains approximate: failure detection takes at least a heartbeat/TTL interval, mobile apps sleep, and networks partition. “Last seen” should come from the last trusted heartbeat or lease transition; relying only on a graceful disconnect misses crashes.

Ordering and reliable delivery

Ordering uses the per-conversation seq. Route a conversation to one authoritative partition leader, which assigns a monotonically increasing sequence or log offset; clients sort by that value, not device timestamps. Global cross-conversation ordering is deliberately not provided. The trade-off is explicit: a single hot conversation is limited by one ordering authority unless the product relaxes ordering or introduces a more complex protocol.

Reliability is at-least-once plus idempotency and deduplication. The client retries SEND with the same messageId until it gets a persistence ACK; an atomic unique constraint returns the original result for a duplicate request. Delivery events and pushes may repeat, so clients deduplicate by the server message ID and detect sequence gaps. Receipts—accepted, delivered, read—are idempotent state transitions, not proof that a human saw a message. Define whether delivered/read aggregates across devices or is tracked per device.

Offline delivery and resync share one mechanism. An inbox or changed-conversation index tells a reconnecting device which conversations advanced; per-conversation cursors then fetch messages after the last contiguous seq. Pagination, tombstones, edits, membership changes, and retention boundaries must be part of that protocol. A single seq without a conversation ID is not a global sync cursor.

Consistency model

Within the availability of its authoritative partition, a conversation receives one server-defined total order; during leader failover, fencing prevents two sequencers from assigning overlapping positions. Across conversations there is no ordering guarantee. Presence is eventually consistent and approximate: a crashed client may read as online until its lease expires. Delivery is at-least-once; idempotent creation and client deduplication aim for an effectively-once user-visible result, while observability must still count retries and duplicates.

Failure modes

• Gateway crash. Its sockets drop. Clients reconnect through the load balancer with backoff and jitter, obtain new connection IDs, renew registry leases, and resync. Messages acknowledged under the durable-store policy survive this process; unacknowledged sends are retried with the same client ID.
• Reconnect storm. A dead server dumps 100,000 clients reconnecting at once onto the rest of the fleet and the registry. Clients must reconnect with backoff and jitter - the Part 3 discipline.
• Stale registry entry. A route pointing at a dead gateway causes a missed push, but durable history and gap repair preserve correctness. Leases expire stale routes; generation tokens prevent an old disconnect from deleting a new route.
• Backplane outage. Real-time push stops. Durable delivery workers retry after recovery, while connected clients can poll/sync as a degraded path. If the backplane itself is ephemeral, the durable delivery log—not the pub/sub channel—owns retry state.
• Slow consumer. A blocked device must not grow an unbounded gateway buffer. Cap per-connection queues, disconnect or drop replaceable events such as typing indicators, and require history resync for durable messages.

Multi-region

Users can connect to the nearest region, while a conversation has a home region or authoritative shard for ordering. Cross-region sends pay the latency to that authority. Failover requires an epoch or fencing token so the old and new leaders cannot both assign sequences. Synchronous cross-region replication increases commit latency but can reduce RPO; asynchronous replication is faster but may lose recently acknowledged messages in a regional disaster unless acknowledgements use a stronger policy. State the chosen RPO/RTO rather than claiming both low latency and zero loss. A durable inter-region stream carries delivery events to endpoints elsewhere.

Evolution path

Build the client messageId, idempotent commit, per-conversation seq, changed-conversation index, and cursor-resync protocol from day one—they are the contracts every reliability and recovery property depends on. Defer presence sophistication, large-group hybrids, and multi-region until measurements require them.

Observability

Track connections and outbound-queue depth per gateway, connection churn, accepted-message latency, online-delivery latency, delivery-log lag, duplicate/retry rate, sequence gaps, registry lease failures, reconnect storms, and sync volume. Define SLOs from product needs and measure them by region and message class; for example, an interview design might propose 99% of ordinary messages reaching an online device within 500 ms, then validate whether that target is useful and affordable.

Step 6 - Bottlenecks and Trade-offs

• Connection count makes the front tier stateful and memory-bound - hence thousands of servers and a registry, instead of a stateless fleet.
• Cross-server routing is bounded by the backplane's throughput, so it must be partitioned.
• Presence fan-out would dominate all other traffic if pushed; TTL heartbeats plus on-demand reads contain it.
• Large-group fan-out repeats the celebrity problem and needs the push/store/pull hybrid.
• Connection server failover is the defining hard case - clients reconnect and resync, and statefulness is precisely what makes it non-trivial.
• A hot conversation is bounded by its ordering authority; partitioning by conversation scales the fleet, not a single channel.
• Multi-device state multiplies endpoints and makes delivered/read semantics a product decision, not merely a boolean column.

Security and Abuse Controls

• Authenticate the handshake and re-authorize every send, history read, membership change, and attachment access. A known conversationId is not authorization.
• Use wss:// in production, validate the browser Origin against an allowlist, and protect cookie-authenticated handshakes from cross-site WebSocket hijacking. Rotate or expire credentials without trusting a connection forever.
• Bound frame and decompressed payload sizes, validate message types, apply per-user/device/conversation rate limits, and isolate expensive fan-out from the gateway event loop.
• Encrypt data in transit and at rest, define retention/export/deletion behavior, and avoid sensitive content in logs. End-to-end encryption changes server-side search, moderation, previews, key backup, and multi-device key distribution, so it needs a separate threat model.
• Design blocking, reporting, spam controls, malware scanning for attachments, and abuse investigation as first-class flows. Availability controls alone do not make a chat product safe.

Reference Architecture

The pattern this problem teaches, reusable well beyond chat:

A stateful gateway layer holding live connections, a leased registry mapping each user to all active endpoints, and a backplane routing committed delivery events to those gateways—all backed by durable history and cursor sync, so a dropped push becomes a repairable gap.

flowchart LR
    subgraph Conn["Stateful connection layer"]
        direction TB
        S1[Connection server]
        S2[Connection server]
    end
    S1 <-->|endpoint leases| Reg[(Connection registry)]
    S2 <-->|endpoint leases| Reg
    S1 <-->|route events| BP[Pub/Sub backplane]
    S2 <-->|route events| BP
    Durable[(Durable messages + delivery log)] --> BP
    Conn --> Durable

Figure 3. The reusable pattern is a stateful edge, leased endpoint registry, durable event hand-off, gateway-addressed backplane, and cursor repair. The durable log owns retry; the backplane optimizes live delivery.

The same shape recurs in systems that push real-time events to many clients: live notifications, collaborative-editing presence, multiplayer game state, and the live driver-rider link of a ride-sharing service. The exact consistency, ordering, and durability requirements differ, but endpoint leases plus durable hand-off and repairable delivery remain a useful toolkit.

Common Mistakes in the Interview

• Using polling, or choosing WebSocket without naming the statefulness it imposes.
• No cross-server routing story - failing to explain how a message reaches a recipient on another server.
• Mapping a user to one socket, ignoring multiple devices, tabs, and concurrent sessions.
• A persist-then-publish dual-write gap, with no transactional outbox or durable log to retry post-commit delivery.
• Broadcasting every presence change, producing a presence storm that dwarfs real traffic.
• Ordering by client timestamps instead of a server-assigned per-conversation sequence.
• Forgetting offline delivery and the reconnect-and-resync protocol.
• Treating a 100,000-member channel like a small group with no fan-out hybrid.
• Ignoring connection-server failover and the reconnect storm it triggers.
• Claiming end-to-end exactly-once delivery without defining the transactional boundary, idempotency key, deduplication, and gap repair.
• Skipping authorization, slow-consumer limits, abuse controls, and data-retention requirements.

Quick Reference

Frequently Asked Questions

Why use WebSocket instead of polling for a chat system?

WebSocket removes repeated polling requests and supports frequent bidirectional events over one live channel. It is a strong default for interactive chat, but not an automatic winner: SSE plus HTTP can suit simpler server-push products, and any long-lived transport needs capacity limits, reconnect, flow control, and resync.

How do you route a message between two users on different servers?

Register each device or tab as a leased endpoint. After atomically committing the message and delivery event, workers resolve every recipient endpoint, batch by gateway, and publish to those gateway channels. A stale route can miss a push without losing the message because cursor sync repairs the gap.

How do you track online presence at scale?

Keep a short-TTL lease per live endpoint and consider the user online while any lease remains. Renew leases with heartbeats and protect updates with connection-generation tokens. Query visible contacts or coalesce subscribed changes instead of broadcasting every transition globally.

How do you guarantee message ordering in a chat?

Route each conversation to an authoritative partition leader and use its monotonic sequence or log offset. This defines order inside that conversation; it does not create a global order and it deliberately limits the throughput of one very hot conversation.

How are messages delivered to a user who is offline?

Durable conversation history is the source of truth. A per-user inbox or change cursor identifies conversations that advanced, and per-conversation cursors fetch messages after the last contiguous sequence. The same protocol repairs gaps after a gateway crash or stale route.

Can a chat system guarantee exactly-once message delivery?

Not literally across an unreliable client boundary: the commit may succeed while its ACK is lost. Use a client-generated idempotency key enforced atomically, at-least-once retries, deduplication, sequence-gap detection, and resync. Narrow transactional subsystems may provide exactly-once processing within their own boundary.

Sources

• WHATWG WebSockets Living Standard
• IETF RFC 6455: The WebSocket Protocol
• Apache Kafka design: ordering, durability, and delivery semantics
• Redis documentation: key expiration and TTL

Related Articles

• System Design Interview Problems: A Senior's Roadmap - the full series index and pattern library.
• System Design Interview Guide: The 6-Step Framework - the method this walkthrough applies.
• Design a Notification Service - Part 3; at-least-once delivery, deduplication, and backoff with jitter.
• Design a News Feed - Part 5; the large-group fan-out is the same celebrity problem.
• Design a Ride-Sharing Service - Part 7; the connection layer reused for live location streaming.
• WebSockets Interview Questions - the transport behind the connection layer in depth.

This is Part 6 of a 12-part system design series where each post solves one problem around one core pattern. Next: Design a Ride-Sharing Service.

Frequently Asked Questions

Why use WebSocket instead of polling for a chat system?

HTTP polling forces the client to repeatedly ask whether new messages exist, wasting requests and adding latency equal to the poll interval. WebSocket establishes a single persistent, bidirectional connection so the server can push a message the instant it arrives. The cost is that connections are long-lived and stateful, which makes the connection servers stateful and complicates load balancing and failover.

How do you route a message between two users on different servers?

Each gateway registers every live device or tab as a leased endpoint in a shared registry, so one user can map to multiple connections on multiple servers. After the message and its delivery event are durably committed, a fan-out worker resolves the recipients' endpoints and publishes to the relevant gateway channels. Each gateway pushes to its local sockets; stale routes are harmless because clients resync from durable history.

How do you track online presence at scale?

Treat presence as approximate, multi-device state. Each live connection refreshes a leased record with a short TTL; a user is online while at least one lease remains. Use a generation token so a late disconnect cannot delete a newer session. Query presence only for visible contacts or coalesce subscription updates instead of broadcasting every connect and disconnect to everyone.

How do you guarantee message ordering in a chat?

Define the ordering scope first. A common design routes a conversation to one authoritative partition leader and uses its monotonically increasing offset or sequence number. Clients order by that server-assigned value, not a device clock. This provides a total order within a conversation, not across conversations, and makes a very hot conversation a deliberate throughput bottleneck.

How are messages delivered to a user who is offline?

Commit each accepted message to durable conversation history before acknowledging it. Keep a per-user inbox or changed-conversation index plus per-conversation cursors, so reconnecting clients can discover which conversations changed and fetch messages after their last contiguous sequence. The same cursor protocol repairs gaps caused by disconnects, stale routes, or gateway crashes.

Can a chat system guarantee exactly-once message delivery?

Do not claim literal end-to-end exactly-once delivery across an unreliable client network. An acknowledgement can be lost after a commit, so retries are necessary. Make message creation idempotent by atomically enforcing a unique client message ID scoped to the sender, then use at-least-once delivery, recipient-side deduplication, gap detection, and cursor resync. Transactional subsystems can offer narrower exactly-once processing guarantees, but they do not remove the client boundary.