Real-time delivery

Short-polling is the right answer more often than candidates think. If the product can tolerate a 5-10 second update delay — dashboards, leaderboards, inventory counts, delivery tracking — polling eliminates every operational headache of persistent connections: no WebSocket upgrade negotiation, no connection state tracking, no pub/sub bus, no reconnect logic, no mobile keepalive.

The implementation is one HTTP endpoint: GET /updates?since=<timestamp>&limit=50. The server queries the database (or cache) for rows newer than the timestamp, returns them with the latest timestamp for the next poll. The client sets a setInterval and fires the request every N seconds. HTTP caching (ETag, 304 Not Modified) eliminates redundant payload transfer when nothing has changed.

The failure mode is obvious: wasted requests. At 100K clients polling every 5 seconds, the server handles 20K req/s even when nothing is happening. This is fine for a CDN-cacheable endpoint (all clients get the same leaderboard) but ruinous for per-user endpoints (each client has a different since value, so caching doesn't help). The break-even point is roughly: if the update frequency is higher than 1 update per poll interval, polling becomes efficient because every response carries data. If updates are rare (1/minute), 90%+ of polls are empty.

When to choose: dashboards with >5s freshness tolerance, public data that can be CDN-cached, systems where simplicity outweighs latency, or as a last-resort fallback when SSE and WebSocket are blocked.

When to move on: latency requirement drops below the poll interval, or per-user poll traffic exceeds the server's request budget.

SSE is the most underused transport in system design interviews. Candidates skip it because they think WebSocket is strictly better, but SSE wins on three axes: simplicity, proxy compatibility, and built-in reconnection.

The protocol is a single HTTP response with Content-Type: text/event-stream. The server writes events as newline-delimited text: `event: message data: {"text":"hello"} id: 42

. The browser's EventSource API handles parsing, buffering, and automatic reconnection. On disconnect, the browser reconnects and sends Last-Event-ID: 42` as a header, allowing the server to replay missed events from a durable store.

SSE works through HTTP/2 multiplexing, which means multiple SSE streams can share a single TCP connection — eliminating the "6 connections per domain" limit that plagued HTTP/1.1 SSE. It works through corporate proxies that block WebSocket upgrades. It works through CDNs that support streaming responses (most do). The only thing it cannot do is send data from client to server on the same connection — the client uses normal POST/PUT requests alongside the SSE stream.

This makes SSE the correct choice for: notification feeds (server → client only), live comment streams, stock tickers, CI/CD build logs, and any scenario where the client is a consumer, not a producer. Even in chat applications, SSE can handle the downstream path (receiving messages) while regular HTTP handles the upstream path (sending messages) — but at that point, WebSocket's single-connection model is simpler.

When to choose: one-directional server push, need proxy compatibility, want built-in reconnect without custom client code, or when HTTP/2 is available and you need multiple event streams.

When to move on: client sends data at high frequency (typing indicators, cursor positions), or the product requires true bidirectional streaming.

WebSocket is the right answer when the client and server both need to send data at high frequency on the same connection. Chat messages, collaborative editing cursors, multiplayer game state, real-time trading — these all require bidirectional streaming that SSE cannot provide.

The connection starts as an HTTP upgrade (Upgrade: websocket). Once upgraded, both sides send binary or text frames with minimal overhead (2-14 bytes per frame vs ~100 bytes for HTTP headers). The connection is persistent — no per-message handshake, no header repetition, no HTTP parsing overhead.

The operational cost is where most candidates underestimate WebSocket. Every persistent connection is state that must be tracked: which user is on which edge server, which rooms they have joined, what their last-seen message is. The load balancer must support long-lived connections (L4 sticky by connection, not L7 round-robin per request). Horizontal scaling requires a pub/sub bus — when a message arrives at one edge server, it must be forwarded to all other edge servers that hold connections from the recipient's conversation members.

Connection lifecycle adds complexity: heartbeat/ping frames to detect dead connections, reconnect with gap replay, graceful shutdown during deploys (drain connections before killing the process), and admission control to prevent thundering herds.

The edge tier architecture that emerges: N stateless edge servers, each holding ~50K connections, subscribing to a Redis pub/sub bus. The publisher writes to the database, then publishes to the bus. Each edge receives events for the channels its clients are subscribed to, and forwards them over the WebSocket. Adding capacity = adding edge servers and subscribing them to the bus.

When to choose: bidirectional communication (chat, collaborative editing, gaming), sub-100ms latency requirement, or when the client sends data at high frequency.

When to move on: there is no simpler transport beyond this — WebSocket is the terminal transport. Evaluate whether you truly need bidirectional or if SSE + HTTP POST would suffice.

In production, no single transport works for every client. Corporate proxies block WebSocket upgrades. Some CDNs buffer SSE responses. Older mobile browsers lack EventSource support. The production answer is a transport negotiation protocol that tries the best option first and falls back gracefully.

The negotiation sequence: (1) attempt WebSocket upgrade — if the handshake succeeds within 5 seconds, use WebSocket for the session. (2) If the upgrade fails (HTTP 400, proxy rejection, timeout), fall back to SSE — open an EventSource for downstream and use HTTP POST for upstream. (3) If SSE also fails (response buffered, no events received within 10 seconds), fall back to long-poll. (4) If long-poll also fails (unlikely), fall back to short-poll at 5-second intervals.

The server exposes a /negotiate endpoint that returns the supported transports and their URLs. The client tries them in order. Once a transport is established, the session is pinned to it — no mid-session switching, which would require state synchronisation. If the connection drops, the client re-negotiates from step 1.

Socket.IO popularised this pattern (though its implementation conflates transport negotiation with application protocol). The key engineering insight is that the application layer should be transport-agnostic: messages are messages regardless of whether they arrive over WebSocket frames, SSE events, or poll responses. The transport layer handles framing, reconnection, and buffering; the application layer handles message routing and business logic.

Feature flags per client segment reduce complexity. If analytics show that 95% of clients successfully connect via WebSocket, the remaining 5% (enterprise customers behind aggressive proxies) can be routed to an SSE-only edge tier, avoiding the complexity of runtime negotiation for the majority.

When to choose: any production system with diverse client environments — enterprise customers behind proxies, mobile users on unreliable networks, or global users with varying ISP configurations.

The simplest real-time delivery: client polls GET /updates?since=<ts> every 5 seconds. The server queries the database for new rows and returns them. No persistent connections, no pub/sub, no connection tracking. A single app server handles this with standard request/response routing.

This works for the first prototype and for low-traffic dashboards. At 1K concurrent clients polling every 5 seconds, that is 200 req/s — trivial. The latency floor is the poll interval: a message sent 0.1s after a poll waits 4.9s to be seen. For a chat app, this is unacceptable; for a delivery-tracking dashboard, it is invisible.

The transition trigger is clear: when the product demands sub-5-second latency, or when empty polls exceed 80% of traffic, short-poll becomes wasteful and you need a persistent-connection transport.

Upgrade from short-poll: the client sends GET /updates, the server holds the connection open until either (a) a new event arrives, or (b) a 30-second timeout expires. The client immediately re-requests after each response. Latency drops from the poll interval to near-instant for the first event.

Implementation requires an async event loop: the server registers each waiting request in a lookup table keyed by user or room. When a new event arrives (via database trigger, internal pub, or polling), the server finds the matching requests and responds. Frameworks like Node.js, Go, and Python asyncio handle this naturally.

At 10K concurrent clients, this means 10K held connections. An async server handles this comfortably. At 100K+ concurrent connections, the memory and file-descriptor overhead starts to matter, and you need to move to a dedicated connection tier with SSE or WebSocket.

The production architecture: clients connect via WebSocket (or SSE) to a fleet of stateless edge servers. Each edge subscribes to a Redis pub/sub bus for the rooms/channels its connected clients belong to. When a publisher emits an event, Redis routes it to every subscribing edge, which forwards it to the relevant clients.

Connection budget: 50K idle connections per edge server. For 3M concurrent connections (10M DAU × 30% online), that is 60 edge servers. Each edge is stateless — if it crashes, clients reconnect to any other edge and re-subscribe. The pub/sub bus is the coordination layer, not the edge.

Durability: the publisher writes to Postgres first, then publishes to Redis. If the pub/sub message is lost, the data is still in the DB. On reconnect, the client sends its last-seen cursor and the edge replays the gap from the database.

This architecture handles millions of concurrent connections, sub-100ms delivery latency, and survives edge failures. The next iteration adds multi-region edge deployment and global pub/sub.

Deploy edge servers in every major region (US-East, EU-West, AP-Southeast). Each region has its own Redis pub/sub cluster. A global pub/sub tier (NATS JetStream or Kafka with MirrorMaker) replicates events across regions. Clients connect to the nearest edge via GeoDNS or Anycast.

Presence is promoted to a first-class service: a dedicated presence cluster receives heartbeats from all edge servers, maintains TTL-based online status in Redis, and broadcasts status changes to contacts via the pub/sub bus. Typing indicators are handled as ephemeral events — no persistence, no gap replay, just fire-and-forget over the bus.

Connection draining on deploy: before shutting down an edge server, it sends a GOAWAY frame to all connected clients, then waits 30 seconds for in-flight messages to drain. Clients reconnect with jittered backoff to prevent thundering herds. The load balancer's health check removes the draining server from the pool immediately, so new connections go elsewhere.

At this stage, the system handles tens of millions of concurrent connections across multiple regions, with sub-50ms delivery within a region and sub-200ms cross-region.

The transport decision is a function of four variables, and interviewers test whether you can navigate it rather than defaulting to WebSocket.

Direction: Does the client only receive (notifications, live scores), or does it also send at high frequency (chat messages, cursor positions)? If the client only receives, SSE is simpler and more proxy-compatible than WebSocket. If the client sends infrequently (one message per minute), SSE for downstream + HTTP POST for upstream is viable. WebSocket is only necessary when the client sends at high frequency.

Latency: What is the freshness budget? If 5-10 seconds is acceptable, short-poll is correct and eliminates all persistent-connection complexity. If sub-second is required, you need SSE or WebSocket. Long-poll sits in between — near-instant for the first event, then poll-interval latency for subsequent events within the same response cycle.

Connection cost: Every persistent connection consumes memory (connection state, buffers, kernel file descriptors). A server handles ~50K idle WebSocket connections before memory pressure becomes noticeable. At 10K concurrent users, connection cost is irrelevant. At 10M, it dominates the architecture. Short-poll has zero persistent connection cost; SSE and WebSocket have one connection per client.

Proxy compatibility: Corporate proxies routinely block WebSocket upgrades (they don't understand the Upgrade header). SSE works through any HTTP-compatible proxy. Short-poll works everywhere. If your users include enterprise customers behind restrictive proxies, you need a fallback strategy or SSE as the primary transport.

The decision matrix: choose short-poll if latency tolerance > 5s and the data is cacheable. Choose SSE if the client is a consumer (server → client only) and you want built-in reconnection. Choose WebSocket if bidirectional streaming is required. Choose hybrid fallback in production when client environments are diverse.

The edge tier is the core architectural innovation that makes real-time delivery scale. Without it, every publisher needs to know which server holds which client — a coupling that makes horizontal scaling impossible.

Each edge server maintains a local subscription table: a map from channel/room id to a set of connected WebSocket handles. When a client connects and joins room "conv:123", the edge server adds the client to its local table and subscribes to the Redis pub/sub channel "conv:123". When a message arrives on that channel, the edge iterates its local subscribers and writes the message frame to each.

Adding an edge server is trivial: deploy it, register with the load balancer. It starts with zero subscriptions. As clients connect and join rooms, it subscribes to channels incrementally. Removing an edge: drain connections (GOAWAY), unsubscribe from all channels, deregister from the load balancer.

The subscription management has a subtle efficiency concern: if 10,000 clients on the same edge are in the same room, the edge receives one copy of each message from the pub/sub bus and fans it out locally to 10,000 WebSocket connections. This is efficient. But if each of those 10,000 clients is in a unique room, the edge subscribes to 10,000 channels. Redis pub/sub handles millions of channels with O(1) subscribe/unsubscribe, so this is fine — but the edge server now receives messages for 10,000 channels and must dispatch each to the correct client.

Connection count per edge is bounded by memory: each idle WebSocket connection costs ~5-10 KB (kernel buffers, userspace state). At 50K connections, that is 250-500 MB — well within a modern server's capacity. The CPU cost of message fan-out is negligible for idle connections; the bottleneck is the burst when a popular channel fires (e.g., a group chat with 1,000 members on the same edge). This is handled by buffering and batching writes: accumulate frames for 1-5ms, then write them in a single syscall (writev / scatter-gather I/O).

Networks drop. Mobile radios sleep. Deploys restart edge servers. Every real-time system must handle reconnection, and the critical question is: what about the messages the client missed while disconnected?

The protocol: every message has a monotonically increasing id (typically a database sequence or a Snowflake id). The client tracks the id of the last message it received. On reconnect, it sends last_seen_id as a parameter. The edge server (or a dedicated catch-up service) queries the message store for all messages with id > last_seen_id, replays them in order, then switches to live streaming from the pub/sub bus.

The gap between "replay from DB" and "live from pub/sub" is the tricky part. While the server is querying the DB for missed messages, new messages may arrive on the pub/sub channel. If the server subscribes to the channel before replaying, some messages appear twice (once from DB replay, once from live). If it subscribes after replaying, some messages may be missed (arrived between the DB query and the subscription).

The solution: subscribe to the pub/sub channel first, buffer incoming messages, then replay from the DB. After replay is complete, drain the buffer, deduplicating by message id. Any message in the buffer with an id already replayed from the DB is skipped. This ensures exactly-once delivery from the client's perspective.

For SSE, this is handled automatically by the Last-Event-ID header — the browser sends it on reconnect, and the server replays from that point. For WebSocket, you must build this protocol yourself: a reconnect handshake that includes the cursor, followed by a gap-fill phase before switching to live mode.

Performance: the gap-fill query should hit a recent-message cache (Redis sorted set, keyed by room, with the last 1,000 messages) rather than the primary database. Only if the gap exceeds the cache window should the query fall back to the database. In practice, most disconnections are brief (seconds to minutes), so the cache handles 99%+ of reconnections without touching the DB.

Presence ("Alice is online") and typing indicators ("Bob is typing...") are the two most common ephemeral real-time features. They seem simple but have subtle scaling implications that interviewers love to probe.

Presence via heartbeat + TTL: The client sends a heartbeat ping over the WebSocket connection every 30 seconds. The edge server forwards this to a presence service, which sets a Redis key presence:{user_id} with a TTL of 60 seconds. If the TTL expires without a refresh, the user is considered offline. On status change (online → offline or vice versa), the presence service publishes a status_changed event to the pub/sub bus, and the edge tier forwards it to the user's contacts.

The scaling concern is fan-out of status changes. When a user with 500 contacts goes online, the presence service must notify 500 other users. If those 500 users are spread across 50 edge servers, that is 50 pub/sub messages. At 10M online users, status changes fire at a rate of thousands per second (users constantly going online/offline). The presence service must batch these changes and rate-limit broadcasts.

Typing indicators are simpler because they are ephemeral — no persistence, no gap replay, no catch-up on reconnect. When a user starts typing, the client sends a "typing" event over WebSocket. The edge forwards it to the conversation's pub/sub channel. All members of the conversation see "X is typing..." for 3 seconds (refreshed if the typing continues, cleared on timeout or on message send).

The key design decision: typing indicators should never be persisted and never be included in missed-message replay. They are fire-and-forget. If a user reconnects and someone was typing while they were disconnected, the typing indicator is irrelevant. This simplifies the system: typing events bypass the message store entirely and go straight to the pub/sub bus.

Presence can be batched for efficiency: instead of broadcasting every individual status change, the presence service batches changes over a 2-second window and sends a single "batch status update" with all changed users. Clients that care about real-time presence (the active chat window) can subscribe to individual status channels for instant updates; the contact list uses the batched channel.

The thundering herd is the most predictable failure in real-time systems, and one of the easiest to prevent — yet most candidates miss it entirely.

The scenario: you deploy a new version of the edge server. The rolling deploy kills edge server 1, which holds 50K WebSocket connections. All 50K clients detect the disconnection (via TCP RST or close frame) and immediately reconnect. If the load balancer distributes them evenly across the remaining 59 edge servers, each receives ~850 new connections in under a second — manageable. But if the reconnect is truly simultaneous (within 100ms), the connection establishment rate spikes: TLS handshakes, WebSocket upgrades, room re-subscriptions, and gap-fill queries all hit at once.

At scale, a rolling deploy that takes down 5 edge servers over 5 minutes means 250K reconnections concentrated in 5 bursts of 50K each. Without mitigation, each burst saturates the remaining edges' CPU (TLS) and the message store (gap-fill queries).

Client-side fix: jittered exponential backoff. On disconnect, the client waits a random delay before reconnecting: delay = min(base * 2^attempt + random_jitter, max_delay). With a base of 100ms, jitter of 0-1000ms, and max of 30s, 50K clients spread their reconnections over 1-2 seconds instead of 100ms. This alone reduces the peak reconnection rate by 10-20x.

Server-side fix: connection draining. Before killing an edge server, send a GOAWAY frame (WebSocket close with code 1001 "Going Away") and wait for clients to disconnect gracefully. The edge stops accepting new connections but continues serving existing ones for a drain period (30-60 seconds). During this window, clients receive the close frame and reconnect with backoff. The drain period spreads reconnections over the entire window rather than concentrating them at the kill instant.

Server-side fix: admission control. The load balancer rate-limits new WebSocket connections per edge server. If an edge receives more than N connection attempts per second, excess requests receive HTTP 503 with a Retry-After header. The client backs off and retries. This prevents any single edge from being overwhelmed during a reconnection storm.

The combination of all three — jittered backoff on the client, connection draining on the server, admission control on the load balancer — makes deploys invisible to users. This is the level of operational maturity interviewers look for in a staff-level answer.

Mobile networks introduce three problems that don't exist on desktop: NAT timeout, battery constraints, and backgrounding.

NAT timeout: Mobile carriers use Network Address Translation (NAT) gateways to share public IPs across devices. These gateways maintain a mapping table: (device private IP:port) → (public IP:port). The mapping has a timeout — typically 30-120 seconds, varying by carrier. If no traffic flows through the mapping within the timeout, the NAT gateway silently drops it. The WebSocket connection is now broken, but neither the client nor the server knows — the TCP connection appears alive until the next write attempt, which may not happen for minutes.

Fix: WebSocket ping frames. The client sends a ping frame every 25 seconds (comfortably within any carrier's NAT timeout). The server responds with a pong. This keeps the NAT mapping alive. The ping/pong is at the WebSocket protocol level — it does not hit the application layer, so it is cheap. Without pings, the connection silently dies, and the user stops receiving messages until the app detects the failure (often only when the user tries to send).

Battery impact: Sending a ping every 25 seconds means the mobile radio wakes up every 25 seconds. On LTE, the radio consumes ~1W when transmitting and ~0.01W when idle. Waking the radio 2,400 times per day (every 25s × 16 waking hours) adds measurable battery drain. The trade-off is acceptable for a chat app (users expect instant delivery) but may not be for a news feed (where 30-second latency is fine and SSE or long-poll with longer intervals saves battery).

Backgrounding: When the user switches to another app, the OS suspends the app's network connections after a brief grace period (iOS: ~30s, Android: varies). The WebSocket connection dies. On resume, the app reconnects with gap replay. But what about messages that arrive while the app is backgrounded?

Fix: push notification fallback. When the edge server detects that a client's connection has been idle for >60 seconds (no pings received), it marks the user as "push-eligible." New messages for that user are sent via APNs (iOS) or FCM (Android) as push notifications. The push notification wakes the app, which reconnects, replays the gap, and resumes the WebSocket. This dual-path (WebSocket when foregrounded, push when backgrounded) is the standard pattern for every mobile messaging app.

The server must track delivery state: a message is "delivered via WebSocket" or "delivered via push" or "pending." This prevents duplicate deliveries (push + WebSocket) and ensures no message is lost in the transition between states.