Multi-region active-passive / active-active

The single-region topology is where every system begins and where many should stay. All users — regardless of geography — hit one region's load balancer, application tier, and database. There is no replica, no standby, no failover path. If the region goes down, the product is offline.

Why it works at small-to-medium scale. Operational simplicity is dramatically undervalued. One region means one database, one deploy pipeline, one set of monitoring dashboards, one on-call rotation. There are no replication lag alerts, no conflict-resolution bugs, no split-brain incidents at 3 AM. For a team of 5–20 engineers serving users primarily in one geography, single-region is not a compromise — it is the right choice.

Cost model. Multi-region roughly triples infrastructure cost (two full stacks + replication bandwidth + cross-region data transfer). It also triples operational complexity. If the business SLA allows 4 hours of downtime per year (99.95%), a well-architected single-region setup with multi-AZ redundancy can meet it.

When to leave single-region. Three triggers: (1) regulatory requirement to store data in a specific geography (GDPR, PIPL), (2) latency SLO that a single region cannot meet for a global user base, (3) business-critical RTO that requires automated regional failover. If none of these apply, stay single-region and invest the engineering effort elsewhere.

AZ redundancy is not multi-region. Running across 3 availability zones within one region provides hardware-level fault tolerance (rack, power, network diversity) without the complexity of cross-region replication. Multi-AZ is the standard baseline; multi-region is the next step only when AZ redundancy is insufficient.

Active-passive is the workhorse DR topology. The primary region handles 100% of reads and writes. The standby region maintains a continuously-updated replica via asynchronous WAL streaming (PostgreSQL) or binlog replication (MySQL). The standby application tier is deployed but idle — it receives no user traffic until failover.

Failover mechanics. When the primary region fails: (1) health checks detect the failure (30–60 seconds), (2) the standby database is promoted to primary (10–30 seconds), (3) DNS is updated to point at the standby region (propagation 60–300 seconds depending on TTL). Total RTO: 2–5 minutes with automation, 15–60 minutes with manual intervention. RPO depends on replication lag — typically 0.1–5 seconds of async lag.

The untested-failover trap. The single most common failure mode of active-passive is an untested failover. Configs drift: the standby's environment variables point to a stale secret, the connection string references the wrong DB, the SSL certificate expired. Quarterly failover drills in production are not optional — they are the only way to know your failover actually works.

Cost model. You pay for a full second region (compute + storage + networking) that handles zero traffic in steady state. This feels wasteful, but the cost of a 4-hour outage at a company doing $10M ARR is roughly $4,500 in direct revenue loss, plus customer trust and SLA credits. The standby region costs far less.

Semi-sync for tighter RPO. If the business requires RPO < 1 second, enable semi-synchronous replication: the primary waits for at least one standby ack before confirming the write. This adds 40–100 ms to write latency (half the cross-region RTT for one ack) but guarantees that committed data exists in both regions. Full synchronous replication (wait for all replicas) doubles the latency penalty and is rarely worth it.

Active-active with eventual consistency is the target state for global-scale systems. Both regions accept reads and writes. Async bidirectional replication propagates changes between regions with a lag window of 0.1–5 seconds. During that window, the two databases may have different values for the same key — this is the consistency window, and the system must handle it.

Partition-by-user: the no-conflict pattern. The most common active-active architecture avoids conflicts entirely by partitioning data by user or tenant. Each user has a home region determined by geography, account creation location, or explicit choice. All writes for that user go to their home region only. Other regions have a read replica for cross-region reads (e.g., when a US user views an EU user's profile). This is active-active at the fleet level but single-writer at the row level.

When partition-by-user breaks. Shared resources that multiple users write concurrently: a collaborative document, a shared shopping cart, an inventory counter, a leaderboard. These entities have no single "home user" and cannot be cleanly partitioned. For these, you need real conflict resolution — LWW, CRDTs, or application merge.

Replication lag monitoring. Async replication lag is the heartbeat of active-active. Monitor it continuously. Normal: <1 second. Concerning: 1–5 seconds. Critical: >5 seconds (risk of significant data divergence on failover). Alert on lag spikes and investigate causes: network congestion, large transactions, schema migrations that block replication.

Read-your-own-writes. After a US user writes a post, they immediately read the page and expect to see their post. If the read hits the EU replica (which has not received the write yet), the user sees stale data. Solutions: (1) sticky sessions route the user to their home region for reads after a write (simplest), (2) read-after-write consistency via synchronous cross-region read confirmation (expensive), (3) client-side optimistic UI shows the write immediately and reconciles later.

Active-active with strong consistency eliminates the conflict-resolution problem entirely: every write is confirmed in all regions before being acknowledged. No replication lag, no stale reads, no conflict resolution. The price is write latency — every write pays the full cross-region round-trip time (80–200 ms).

How it works. Distributed databases like CockroachDB, Google Spanner, and YugabyteDB use Raft or Paxos consensus across regions. A write is proposed by the local node, replicated to a quorum of nodes across regions, and committed only when a majority acknowledges. The write latency equals the time to reach quorum — typically one cross-region RTT for a 3-region setup (the local node + one remote node form a majority of 3).

Spanner and TrueTime. Google Spanner achieves globally consistent reads by using GPS-synchronised clocks (TrueTime) with bounded uncertainty (~7 ms). Reads wait out the uncertainty window to guarantee they see all committed writes. This is an engineering marvel but requires specialised hardware (GPS receivers in every data centre). CockroachDB approximates this with hybrid logical clocks (HLCs) — slightly larger uncertainty windows but no specialised hardware.

When strong consistency is worth the latency. Financial systems where double-spending or phantom reads cause real money loss. Inventory systems where overselling is worse than slower checkout. Any domain where "eventually consistent" means "eventually wrong." Name the use case explicitly — most systems do not need cross-region strong consistency and should not pay the latency tax.

Geo-partitioned leaseholders. CockroachDB and Spanner allow pinning table partitions to specific regions. EU user data has its leaseholder in the EU region, so EU reads and writes hit local storage with no cross-region penalty. Only cross-partition transactions (rare) pay the full RTT. This combines the performance of partition-by-user with the consistency of distributed consensus.

All users hit one region. Multi-AZ for hardware fault tolerance. No cross-region replication. Simplest to operate. Regional outage = total outage.

Primary region serves all traffic. Standby region maintains an async replica. Automated health checks trigger DNS failover on primary failure. Quarterly failover drills validate the setup.

Both regions accept traffic. Users partitioned by home region — each user writes only to their home region. Async replication provides cross-region read replicas. No conflicts by construction. Zero-RTO for the fleet — if one region fails, the other absorbs its users.

Distributed consensus (Raft/Paxos) across regions. Every write confirmed by a quorum before ack. No conflicts, no replication lag. Write latency includes cross-region RTT. Geo-partitioned leaseholders minimise latency for partitioned data.

Active-passive failover has three phases: detection, promotion, and redirection. Each phase has its own failure modes, and the total RTO is the sum of all three.

Phase 1: Detection. Health checkers (Route 53, Cloudflare Health Checks, custom probes) ping the primary region every 10–30 seconds. A failure is declared after N consecutive failures (typically 3). Detection time: 30–90 seconds. False positives (transient network blip triggers failover) are the main risk — set the threshold high enough to avoid flapping but low enough for timely detection.

Phase 2: Promotion. The standby database is promoted from read-replica to read-write primary. In PostgreSQL, this is 'pg_ctl promote' or an RDS API call. The promotion itself takes 10–30 seconds. During promotion, the standby replays any remaining WAL segments — this is the RPO window. If async replication was 2 seconds behind, those 2 seconds of transactions are lost (or must be reconciled manually after the original primary recovers).

Phase 3: Redirection. DNS records are updated to point the domain at the standby region's load balancer. DNS propagation depends on TTL: a 60-second TTL means most resolvers switch within 60–120 seconds; a 300-second TTL means up to 5 minutes. During propagation, some users still hit the (dead) primary. Use low DNS TTLs (30–60 s) in production, and consider AWS Global Accelerator or Cloudflare Anycast for instant redirection that bypasses DNS TTL entirely.

Total RTO. Detection (60 s) + promotion (20 s) + DNS propagation (60 s) = ~2.5 minutes best case. Manual failover with a human in the loop: add 5–30 minutes for paging, assessment, and decision. Automated failover is the only way to hit sub-5-minute RTO consistently.

The recovery problem. After failover, the old primary is now a stale, possibly corrupted node. Recovery options: (1) rebuild it as a new replica from the new primary (safest, slowest), (2) replay its divergent WAL and reconcile (risky, complex), (3) discard it and provision a fresh standby. Most teams choose option 1 — treat the old primary as disposable.

Conflict resolution is the defining challenge of active-active architectures. Two regions write the same key concurrently — which write survives?

Last-write-wins (LWW). Each write carries a timestamp. When two writes to the same key arrive via replication, the one with the higher timestamp wins; the other is silently discarded. Implementation: DynamoDB Global Tables, Cassandra, Cosmos DB default policy.

Strengths: trivially simple, no application changes. Weaknesses: data loss. If Region A sets price=100 at T1 and Region B sets price=200 at T2, LWW keeps 200 and drops 100 — even if 100 was the correct price. Clock skew between regions can cause the "older" write to win if its clock is ahead. NTP synchronisation helps but cannot eliminate skew entirely.

CRDTs (Conflict-free Replicated Data Types). Mathematical structures that merge without conflicts:

• G-Counter (grow-only counter): each region maintains its own counter; the merged value is the sum. Perfect for view counts, like counts.
• PN-Counter (positive-negative counter): two G-Counters — one for increments, one for decrements. Net value = positive - negative. Works for inventory with add/remove.
• OR-Set (observed-remove set): elements can be added and removed without conflicts. A remove only affects adds the remover has observed. Good for shopping carts, tag sets.
• LWW-Register: a single value with a timestamp — essentially LWW for one field. Useful for profile fields where the latest edit should win.

CRDTs are powerful but limited: they cannot express arbitrary business rules (e.g., "reject if balance would go negative"). They shine for specific data shapes — counters, sets, collaborative text — and should be used surgically, not as a general solution.

Partition-by-user. Assign each user (or tenant, or entity) a home region. All writes for that entity go to its home region only. Other regions have read replicas. No conflicts because each row has exactly one writer. This is the default recommendation for most active-active systems: WhatsApp, Slack, Notion, most SaaS platforms. The 1% of requests from travelling users are proxied cross-region — a small latency penalty for operational simplicity.

Application merge. Present both versions to the user and let them choose. Google Docs uses Operational Transformation (OT); Figma uses CRDTs. This is the right approach for collaborative editing where human intent matters, but it is heavy to build and confusing for non-collaborative use cases.

DNS is the first routing layer in a multi-region architecture. The DNS resolver decides which region's IP address to return to the client, and that decision shapes the user's entire request path.

Latency-based routing. The DNS provider measures round-trip time from the resolver to each region's health endpoint and returns the IP of the region with the lowest latency. AWS Route 53, Cloudflare, and Google Cloud DNS all support this. It works well when regions are well-distributed and the resolver is geographically close to the user (which is usually but not always true — some ISPs route DNS through distant resolvers).

Geolocation routing. Returns the IP of the region mapped to the user's geographic location (determined by the resolver's IP or EDNS Client Subnet). Simpler than latency-based but less accurate — a user in eastern Canada might be closer to EU-West than US-East. Use geo routing when data residency requires region affinity (EU users must hit EU region) rather than pure latency optimisation.

Failover routing. Primary/secondary configuration: DNS returns the primary region's IP unless health checks fail, then switches to the secondary. This is the DNS layer for active-passive failover. Health check interval (10–30 s) × failure threshold (3) = detection time. TTL determines propagation speed. Low TTLs (30–60 s) speed up failover but increase DNS query volume.

Weighted routing. Distribute traffic by percentage: 90% to US, 10% to EU. Useful for canary deployments (route 5% of traffic to a new region) or gradual migration from one region to another. Not a steady-state routing strategy.

DNS TTL trade-off. Low TTL (30 s): fast failover, high DNS query volume, slightly higher latency (more DNS lookups). High TTL (300 s): slow failover, lower DNS volume, cached results serve faster. Production recommendation: 60 s TTL for services that need automated failover. If using Anycast (Cloudflare, AWS Global Accelerator), DNS TTL matters less because the Anycast IP does not change — routing shifts at the network layer instead.

Beyond DNS: Anycast and Global Accelerator. DNS-based routing has a fundamental limitation: TTL-bound propagation delay. Anycast eliminates this: the same IP address is announced from multiple regions via BGP. The network routes each packet to the nearest announcing region. Failover is instant — BGP reconvergence happens in seconds, not minutes. AWS Global Accelerator and Cloudflare use Anycast for the edge, terminating TCP at the nearest PoP and forwarding to the backend region. This is the fastest failover mechanism available.

Automated failover is the difference between a 2-minute RTO and a 30-minute RTO. Manual failover requires paging an on-call engineer, assessing the situation, deciding to fail over, and executing the runbook — all while the product is down. Automated failover compresses this to detection + promotion + redirection, all executed by software.

Health check design. A good health check verifies the entire request path, not just "is the server up." The health endpoint should: (1) query the database (proves DB connectivity), (2) check cache connectivity, (3) verify that the application can process a request end-to-end. Return 200 if all checks pass, 503 if any fail. The health checker (Route 53, CloudWatch, custom) polls this endpoint every 10–30 seconds.

Failure detection thresholds. Require N consecutive failures (typically 3) before declaring a region down. This prevents flapping: a transient network glitch or a brief GC pause should not trigger a failover. But the threshold must be low enough for timely detection. At 10-second intervals and 3 failures: detection takes 30 seconds worst case. At 30 seconds × 3: 90 seconds. Choose based on your RTO budget.

Traffic shifting strategies. (1) DNS failover: update DNS records. Propagation bounded by TTL — 30–300 seconds. (2) Global Accelerator / Anycast: shift traffic at the network layer. Propagation in seconds via BGP. (3) Application-layer routing: a global load balancer (Cloudflare, AWS ALB with Global Accelerator) routes based on backend health. Instant for new connections; existing connections drain.

Automated promotion. Cloud providers offer managed failover: RDS Multi-AZ promotes the standby automatically, Aurora Global Database promotes a secondary cluster. For self-managed databases, promotion scripts must be battle-tested: stop replication, promote replica, update connection strings, validate data integrity, report status.

Failover drills. The most important operational practice in multi-region is scheduled failover in production. At least quarterly, intentionally fail over to the standby region during business hours with the team watching. This catches: expired credentials, drifted configs, broken runbooks, untested promotion scripts, DNS TTL surprises, and monitoring gaps. Netflix's Chaos Monkey and Gremlin are tools for this, but a simple "turn off the primary region in the console" is the minimum viable drill.

Failback. After failover, the old primary must be recovered (rebuilt as a replica of the new primary) and traffic gradually shifted back. Failback is often harder than failover because the old primary may have divergent data. Plan for it explicitly — many teams discover failback is broken only after their first real failover.

Data residency requirements constrain where user data can physically reside and how it can be transferred across borders. These constraints are not optional — GDPR fines reach 4% of global revenue, and PIPL violations can result in app bans in China.

GDPR (EU). Personal data of EU residents must be processed under EU data protection law. This does not strictly require data to stay in the EU — it requires "adequate protection" (adequacy decisions, Standard Contractual Clauses, Binding Corporate Rules). However, the simplest compliance architecture is region-pinning: EU users' data lives in an EU region and never crosses borders.

PIPL (China). Stricter than GDPR for cross-border transfers. Personal information of Chinese users generally must be stored in China and can only be exported after security assessment by the Cyberspace Administration of China. In practice, most companies run a fully isolated China region with no cross-border data flow.

Architecture implications.

1. 1Region affinity per user. Each user is assigned a home region based on their legal jurisdiction at account creation. All their personal data is stored in that region. DNS routing sends them to their home region. This is partition-by-user with a legal constraint on the partition key.

1. 1No cross-border replication of PII. Async replication of EU PII to a US region may violate GDPR. Either exclude PII from cross-region replication (replicate only anonymised or aggregated data) or encrypt PII with EU-controlled keys that cannot be decrypted in the US.

1. 1Audit trail. Maintain records of where each piece of data resides, when it was created, and whether it was ever transferred. GDPR Article 30 requires this. Use database-level tagging or a metadata service that tracks data residency per user.

1. 1Right to erasure (Article 17). When a user requests deletion, all copies across all regions must be deleted — including replicas, backups, and cached data. Multi-region makes this harder because data may exist in multiple replicas. Build a deletion pipeline that propagates erasure requests to all regions and confirms completion.

Interview signal. When a candidate mentions multi-region, asking "how do you handle GDPR?" tests whether they understand that multi-region is not just a latency and DR pattern — it is a compliance architecture. Name region-pinning, PII exclusion from replication, and deletion propagation.

In a multi-region architecture, each region typically has its own cache cluster (Redis, Memcached). The challenge: when data changes in Region A's database, Region A's cache is invalidated — but Region B's cache still holds stale data. Without cross-region cache invalidation, users in Region B see stale content until the cache's TTL expires.

Option 1: Short TTLs (simplest). Set cache TTL to 30–60 seconds. Stale data is bounded by the TTL. No cross-region invalidation needed. Works when seconds of staleness are acceptable — product catalogues, user profiles, content feeds. Most systems should start here.

Option 2: Event-driven invalidation. When a write occurs in Region A, publish an invalidation event to a cross-region message bus (Kafka MirrorMaker, SNS + SQS, Pub/Sub). Each region's cache subscriber receives the event and deletes the stale key. Propagation time: 0.5–5 seconds (replication lag + processing). More complex but bounds staleness to seconds regardless of TTL.

Option 3: Write-through to remote caches. On every write, the application updates the local cache AND sends an update to remote region caches. This is eager invalidation — the remote cache has fresh data almost immediately. The risk: if the remote cache update fails (network partition), the remote cache is silently stale. Use a retry queue to handle transient failures.

Option 4: Redis Global Replication. AWS ElastiCache Global Datastore and Redis Enterprise Active-Active replicate cache data across regions automatically. The cache replication runs independently of database replication. This is the managed solution: zero application code for cross-region cache consistency. Trade-off: vendor lock-in and cost.

Cache stampede across regions. If a popular key expires simultaneously in all regions, all regions miss and hit the database concurrently — a global cache stampede. Solutions: (1) jitter TTLs by ± random seconds so expiry is desynchronised, (2) use SWR-style lazy refresh (serve stale, refresh async), (3) single-flight / request coalescing per region so only one process refills the cache.

Interview tip. When discussing multi-region caching, name the trade-off: short TTLs are simple but allow seconds of staleness; event-driven invalidation is tighter but requires a cross-region messaging layer. Don't claim "Redis handles it" — Redis replication lag is real (100–500 ms cross-region), and the application must decide what "consistent" means for each cached entity.