Design a real-time chat/messaging system at scale
Expected question
"Design a real-time chat/messaging system at scale. How do you deliver messages with low latency, ordering guarantees, and presence across millions of users?"
Variant forms
Same design, different framing:
- "Design WhatsApp-scale 1:1 messaging with delivery and read receipts."
- "How do you build group chat for 500 members with acceptable fan-out?"
- "Design message sync across mobile, web, and desktop with offline support."
- "Scale from 1M to 500M daily active users — what breaks in your websocket layer?"
- "Design end-to-end encryption without sacrificing search and moderation."
- "How do you handle 'last seen' and typing indicators at scale?"
- "Design a system where messages are never lost even if a client is offline for a week."
Where this actually gets asked
The best-sourced entry in this folder. Meta is well-documented as asking a version of this — multiple independent interview-prep sources (IGotAnOffer, Exponent, DesignGurus) consistently list "Design WhatsApp/Messenger" as one of Meta's most common system-design questions; no single verbatim Blind quote was captured, but the convergence across independent sources, plus Meta actually owning both products, makes this credible. Microsoft has moderate secondary sourcing for "design the chat service for Microsoft Teams." No company-specific evidence was found for Google, Apple, OpenAI, or Anthropic on this exact topic. What's unusually strong here: the real system grounding, independent of interview attribution. WhatsApp's own engineering team publicly documented their architecture directly — Rick Reed's "Scaling to Millions of Simultaneous Connections" talk (Erlang Factory SF, 2012) and WhatsApp's own blog post "1 Million is so 2011." Separately, Meta's own engineering blog published "Building Facebook Messenger" (2011) and "Building Mobile-First Infrastructure for Messenger" (2014), describing the real MQTT-based push architecture and the "Iris" ordered-queue system — genuine primary sources, not aggregator claims.
Executive summary
30-second thesis
I'd treat this as a connection-and-ordering problem first — WebSocket/MQTT gateways, a connection registry, and a per-user ordered offline inbox — not as "shard the messages table."
2-minute answer
Clarify 1:1 vs large groups, offline retention, and whether E2E crypto is in scope (usually defer). Default: durable append with per-conversation seq, fanout to inboxes, push on open sockets, client ack + client_msg_id dedup. Presence and typing shed first under overload. Offline sync is gap-fill from last seq — Iris-style — not "hope pub/sub remembered." Multi-device: fan to each device connection; read receipts idempotent per device. Groups: fanout-on-write until member count hurts, then pull/hybrid like feeds. Connection count per box is the first ceiling; DB throughput is the second. Refuse exactly-once at the server alone.
Quantitative trade-offs
| Decision | Trade-off and reversal evidence | Evidence class |
|---|---|---|
| WS push vs polling | Push wins latency; reverse to poll only as degraded mode when WS fleet is unhealthy. | H |
| Fanout-on-write vs on-read for groups | Write fanout is simple until ~hundreds of members (H); reverse to hybrid/read for large rooms. | H |
| At-least-once + client dedup vs exactly-once dream | Dedup is cheap and real; reverse only if you can fund end-to-end transactional inbox semantics. | R/H |
What I'd ask them
- Max group size and offline retention window?
- Multi-device / multi-platform sync required day one?
- Is E2E encryption a hard requirement or a later deep dive?
Requirements
Functional
- Users can send/receive messages 1:1 and in groups, with delivery working even when the recipient is briefly offline.
- Message delivery status (sent, delivered, read) should be trackable and shown to the sender.
- Users should see whether their contacts are currently online (presence).
Non-functional
- Low latency for message delivery when both parties are online — this is a real-time system, not an eventually-consistent one.
- At-least-once delivery, with the client responsible for deduplication via message IDs — never silently drop a message, even under server failure.
- Must scale to a connection count in the hundreds of millions concurrently — this is fundamentally a long-lived-connection-management problem, not just a database-throughput problem.
Core entities
- Message: sender, recipient(s), content, timestamp, a client-generated unique ID (for dedup), and delivery status.
- Conversation: 1:1 or group, with membership and the ordered message history.
- Connection: a client's currently-open long-lived connection (WebSocket or MQTT), mapped to which server instance is holding it — needed to route a message to the right server when the sender and recipient aren't connected to the same one.
- Presence record: a user's current online/offline/last-seen status.
API / interface
Auth: user session; device-scoped connections.
POST /v1/conversations
{"member_ids":["u_1","u_2"],"type":"direct"} → 201 {"conversation_id":"cv_..."}
POST /v1/conversations/{id}/messages
Idempotency-Key: <uuid>
{"client_msg_id":"cmsg_...","body":"hello","reply_to":null}
→ 201 {"message_id":"msg_...","seq":1842,"server_ts":"..."}
GET /v1/conversations/{id}/messages?after_seq=1800&limit=50
→ {"messages":[...],"next_after_seq":1850}
WebSocket /v1/realtime?device_id=dev_...
→ server pushes {"type":"message","conversation_id":"cv_...","message":{...}}
→ client sends {"type":"ack","conversation_id":"cv_...","seq":1842}
→ client sends {"type":"typing","conversation_id":"cv_..."}
POST /v1/conversations/{id}/read
{"up_to_seq":1842} → 200 {"ok":true}
Staff+ callout: client_msg_id + seq give idempotent send and gap-free sync; read receipts are a separate write.
Data Flow
Send writes an ordered message, fanout updates inboxes, websocket pushes to online devices, acks advance cursors.
Rendering architecture diagram…
High-level design
Rendering architecture diagram…
What the diagram is really about: sender and recipient are very often connected to different gateway servers, so message delivery requires a routing layer that knows which server currently holds which user's connection — a stateful routing problem, unlike a typical stateless request/response API.
Deep dive 1: connection management at scale
| Approach | Concurrent connections per server | Real-time push | When it's the right call |
|---|---|---|---|
| Polling (client asks "any new messages?") | High (stateless) | Poor — adds latency proportional to poll interval | Never for a real chat product; only ever a fallback |
| Long-lived WebSocket | Moderate-high, bounded by server memory/file descriptors | Good | The standard modern default |
| Erlang/BEAM-based connection handling (WhatsApp's real, documented approach) | Very high — the actor model and lightweight processes are specifically suited to millions of concurrent lightweight connections | Good | When connection count per server is the dominant scaling axis, as WhatsApp's own published talks describe |
Trap: treating this as a database-scaling problem (shard the messages table) when the actual first-order bottleneck is connection count — a server has a hard ceiling on concurrent open connections well before it hits a database throughput limit, and the real engineering challenge WhatsApp's own talks describe is specifically about maximizing connections-per-machine.
Deep dive 2: offline delivery and ordering — Meta's real "Iris" pattern
When a recipient is offline, messages need to queue until they reconnect, and arrive in the correct order. Meta's own published architecture ("Building Mobile-First Infrastructure for Messenger," 2014) describes exactly this: an ordered, per-user queue system (internally named Iris) that assigns each message a sequence number per recipient, so a reconnecting client can request "everything since sequence N" and receive a gap-free, correctly-ordered backlog — a more pattern than a raw pub/sub fan-out, which doesn't inherently guarantee ordering or completeness across a disconnect/reconnect cycle.
Deep dive 3: delivery guarantees and deduplication
At-least-once delivery (never silently drop a message) necessarily means a client might receive
the same message twice — after a connection drop and retry, for example. The correct design
puts deduplication responsibility on the client, using the client-generated client_message_id
from the send request: if a client sees a message with an ID it's already displayed, it discards
the duplicate rather than showing it twice. Trap: trying
to achieve exactly-once delivery at the server layer alone — this is either impossible or
requires far more coordination overhead than the at-least-once-plus-client-dedup pattern real
systems actually use.
Deep dive 4: hot conversations and backpressure
Shard message storage by conversation_id; large groups/bots are hot partitions — call them out.
Under overload, shed typing/presence before durable messages; cap offline inbox growth and fall
back to pull-sync. In 45 minutes, cover fanout + ordering + offline sync; defer full E2E crypto
unless asked.
What's expected at each level
- Mid-level: DB + polling/pubsub; misses connection-count and ordering.
- Senior: WebSocket + cross-server routing via a connection registry.
- Staff+: per-recipient seq + gap-fill sync; at-least-once + client dedup (no exactly-once cosplay).
- Principal: connection count is the first ceiling; can discuss why BEAM-style runtimes showed up in WhatsApp's published talks.
Follow-up questions to expect
- "Two devices at once?" — Fan the message to both device connections. Read receipts need to be idempotent per device so reading on phone doesn't leave desktop confused.
- "How does group chat change this?" — Fan-out cost scales with members. Past a few hundred (H), it starts looking like the feed problem — hybrid write/read fanout, same instinct as 03.
Related
- general-system-design/03: News feed / ranking system — the fan-out trade-off this entry's group-chat follow-up connects to
- general-system-design/07: Distributed cache / CDN layer — Meta's real Memcache-at-scale pattern, relevant to this system's presence/status lookups