High Availability

The simplest deployment: a single application server talking to a single database. There are no load balancers, no replicas, no health checks — every component is a single point of failure.

When this is fine: prototypes, internal tools with < 100 users, hobby projects, anything where "restart the server" is an acceptable incident response.

Why it breaks: a server reboot, a bad deployment, a DB disk full, a network blip — any single event causes total downtime. There is no automatic recovery. MTTR is "however long it takes a human to SSH in and fix it," which at 3 AM is measured in hours.

Availability math: if the server is 99.5% and the DB is 99.5%, combined availability is 99.5% x 99.5% = 99.0% — almost four days of downtime per year. Acceptable for a team wiki; disqualifying for anything customer-facing.

Upgrading from here: the first step is always separating the app from the DB (so you can restart one without the other) and adding a health-checked load balancer in front of at least two app instances. This moves you to v2.

Add a load balancer and multiple app instances. The app tier is now redundant — losing one instance does not cause an outage. The DB has a read replica for read scaling and a warm standby for manual failover.

What changes: the load balancer performs health checks every 10-30 seconds. An unhealthy instance is removed from the pool within one check interval. The remaining instances absorb the traffic. Deployments can use rolling updates with zero downtime.

What remains fragile: the database. The primary is still a SPOF. If it crashes, the replica can be promoted manually, but this takes 5-15 minutes and may lose the last few seconds of writes (async replication lag). The load balancer itself may be a SPOF if using a single instance (use a managed LB or VRRP pair).

Availability math: app tier with 3 instances at 99.5% each in parallel: 1 - (0.005)^3 = 99.999987%. But the DB at 99.5% caps the system at 99.5%. The weakest link dominates.

Upgrading from here: add synchronous replication and automatic DB failover (PostgreSQL Patroni, MySQL Group Replication, or managed RDS Multi-AZ). Add a second load balancer. This moves you to v3.

Every layer has redundancy. Dual load balancers (active-passive or active-active). N+2 app instances across three AZs. Quorum-replicated database cluster with automatic leader election. Redis Sentinel or Cluster for cache failover.

Load balancers: managed cloud LBs (AWS ALB/NLB, GCP GLB) are multi-AZ by default. On-prem, use a VRRP pair with keepalived — one active, one standby, virtual IP floats between them. Health checks at the LB level detect app failures in < 30 seconds.

App tier: N+2 means you can lose two instances simultaneously and still handle peak load. Stateless design: no in-memory sessions. Deploy across at least 3 AZs so that losing an entire AZ still leaves you with > 50% capacity.

Data tier: three-node quorum cluster (Raft-based: etcd, CockroachDB; or Paxos-based: Google Spanner). Synchronous replication within a region; leader election completes in < 10 seconds on failure. RPO = 0 (no data loss within region).

Cache tier: Redis Sentinel watches the primary and promotes a replica on failure. Cache loss causes a thundering herd to the DB — pre-warm caches during failover and use request coalescing to absorb the spike.

Availability: with every component at 99.95% and all layers redundant, the combined system approaches 99.99% — 52 minutes of downtime per year. Achieving this requires not just infrastructure but operational discipline: automated runbooks, sub-5-minute MTTR, and quarterly failover drills.

Cell architecture divides the system into independent, self-contained units called cells. Each cell has its own app tier, database, cache, and message queue. A cell router at the edge hashes the request (typically by user ID or tenant ID) and routes to the correct cell.

Why cells? In a monolithic HA deployment, a correlated failure (bad config push, poison message, cascading retry storm) can take down the entire system. Cells contain the blast radius: a failure in Cell 1 affects only its shard of users. Cells 2-N continue serving their users as if nothing happened.

Cell sizing: each cell serves 5-10% of total traffic. This means a cell failure affects at most 10% of users. Cells are sized identically for operational simplicity — same instance types, same DB schema, same deploy pipeline. You scale by adding cells, not by making cells bigger.

Cell router: the router is the one shared component and must be extremely simple and highly available itself. It does one thing: hash the shard key and route. No business logic. Typically implemented as a thin L7 proxy with a hash ring. The router itself is redundant (multi-AZ, health-checked).

Deployments: cells enable safe deployments. Deploy to one cell first (canary). Monitor error rates. If clean, roll to the next cell. A bad deployment affects only the canary cell. This is fundamentally safer than deploying to the entire fleet simultaneously.

Who uses this: AWS uses cell architecture for Route 53, S3, and DynamoDB. Azure uses it for Storage. These are the most available services on the internet — not by accident, but by design.

One server, one database. Every component is a SPOF. Acceptable for prototypes and internal tools. Availability target: ~99% (3.65 days downtime/year). Bottleneck: any single failure causes total outage.

Add LB + multiple app instances + DB read replica. App tier is now redundant; DB primary is still SPOF. Availability target: 99.9%. Rolling deployments eliminate downtime. Health checks detect and remove failing instances.

Dual LBs, N+2 app instances across 3 AZs, quorum-replicated DB, cache cluster. No SPOF. Automatic failover at every layer. Availability target: 99.99% (52 min/year). Requires operational discipline: automated runbooks, chaos testing, quarterly failover drills.

Partition into independent cells. Each cell is self-contained. Cell router hashes shard key. Blast radius limited to one cell (5-10% of users). Canary deployments per cell. Availability target: 99.999%. Used by AWS (S3, Route 53, DynamoDB) and Azure (Storage).

Availability math is the foundation of every HA conversation. Two rules govern everything:

Rule 1 — Serial composition multiplies failure rates. If Service A is 99.9% and Service B is 99.9%, and a request must go through both, the combined availability is 99.9% x 99.9% = 99.8%. Three services in series: 99.9%^3 = 99.7%. The takeaway: every synchronous dependency you add to the request path drags down your overall availability. This is why microservice architectures with deep call chains struggle with availability — each hop multiplies failure probability.

Rule 2 — Parallel composition multiplies uptime. Two instances at 99.9% each in active-active: 1 - (1 - 0.999) x (1 - 0.999) = 1 - 0.001 x 0.001 = 99.9999%. This is the engine behind redundancy — put two of everything and the combined availability soars. Three instances: 1 - (0.001)^3 = 99.9999999%.

SLA composition across dependencies. A real system calls multiple dependencies: DB, cache, message queue, third-party APIs. The overall SLA cannot exceed the product of all serial dependencies' SLAs. If your DB is 99.99% but your payment processor is 99.9%, your checkout flow is at most 99.9% x 99.99% = 99.89%.

Implications for design:

• Minimise the number of synchronous dependencies in the critical path.
• Where possible, make calls parallel (fan-out) rather than serial (chain).
• Use caching to short-circuit dependencies: a cache hit avoids calling the dependency entirely, removing its failure rate from the calculation.
• For non-critical features, use async processing: if analytics fails, the user doesn't notice.
• Set your SLO honestly: if your dependency chain yields 99.7%, don't promise 99.9%. Either improve the chain or adjust the promise.

Error budgets (Google SRE). If your SLO is 99.95%, you have a budget of 0.05% — about 22 minutes/month. When you're within budget, ship features fast. When you're burning through it, freeze deployments and focus on reliability. This turns availability from a vague aspiration into a measurable resource that product and engineering negotiate over.

Health checks are the nervous system of high availability. Without them, your load balancer sends traffic to dead instances, your orchestrator doesn't restart crashed containers, and your failover never triggers.

Liveness probes answer: "is this process alive?" A simple HTTP 200 from /healthz. If the liveness check fails, the orchestrator kills and restarts the container. Use this to detect deadlocked processes, OOM-killed workers, and stuck event loops. Keep liveness checks simple — they should not call external dependencies. A liveness check that queries the database will kill healthy instances when the database is slow.

Readiness probes answer: "can this instance serve traffic?" A readiness check verifies that the instance has completed startup, its caches are warm, its connection pools are established, and its downstream dependencies are reachable. If readiness fails, the load balancer removes the instance from the pool but does NOT restart it. This is critical for graceful deploys: a new instance that's still warming up should not receive traffic until ready.

Tuning parameters:

• Interval: how often the probe fires. 10s is a good default. Too frequent = noise; too infrequent = slow detection.
• Timeout: how long to wait for a response. 3-5s. A timeout is a failure.
• Failure threshold: how many consecutive failures before action. 3 is typical. This prevents flapping on transient network blips.
• Success threshold: how many consecutive successes before marking healthy. 1 for liveness, 2-3 for readiness.

Deep health checks go beyond HTTP 200: they verify the DB connection is alive, the cache is reachable, and the last heartbeat from a critical dependency was recent. Use these for readiness probes only — never for liveness. A deep health check that fails because Redis is slow should remove the instance from the LB pool, not restart it.

Health check cascading: if Service A's readiness check calls Service B's readiness check, and Service B checks Service C, a failure at C cascades up and takes all services out of the LB pool. Design readiness checks to check only direct dependencies, not transitive ones.

A circuit breaker prevents a failing dependency from dragging down the entire system. Without one, a slow or failing downstream causes threads/connections to pile up in the caller, eventually exhausting resources and cascading the failure upstream.

Three states:

1. 1Closed (normal): all requests pass through to the dependency. The breaker monitors failure rate. If the failure rate exceeds a threshold (e.g., 50% failures over a 30-second window), the breaker trips to Open.

1. 1Open (failing fast): all requests immediately fail (or return a fallback response) without calling the dependency. This gives the downstream time to recover. After a cooldown period (e.g., 60 seconds), the breaker transitions to Half-Open.

1. 1Half-Open (probing): the breaker allows one (or a few) probe requests through. If they succeed, the breaker resets to Closed. If they fail, it goes back to Open for another cooldown period.

Tuning the breaker:

• Failure rate threshold: 50% over 30 seconds is a reasonable starting point. Too low = false trips on normal error rates; too high = slow to detect real failures.
• Cooldown period: 30-60 seconds. Long enough for the downstream to recover; short enough that you're not failing fast for minutes after the issue resolves.
• Half-open probe count: 1-3 requests. Enough to get a signal; not so many that you flood a recovering dependency.
• Sliding window: use a rolling window (not fixed) to avoid edge-boundary artifacts.

Fallback strategies when the breaker is open:

• Return cached data (stale but available).
• Return a degraded response ("recommendations unavailable, showing popular items").
• Queue the request for retry when the breaker closes.
• Return a clear error with retry guidance.

Implementation: libraries like resilience4j (Java), Polly (.NET), and opossum (Node.js) provide battle-tested circuit breakers. In service meshes like Istio, circuit breaking is configured at the infrastructure level via DestinationRules.

Per-dependency breakers: each downstream dependency gets its own circuit breaker with independently tuned thresholds. A slow payment processor should not trip the breaker for the recommendation engine.

Graceful degradation is the principle that when a non-critical component fails, the system continues operating with reduced functionality rather than failing entirely. It requires advance planning — you must decide what to shed before the incident happens.

The degradation matrix: for every dependency, document what the user experience should be when it's unavailable:

Load shedding under pressure: when the system is under extreme load and cannot serve all requests at full quality, shed non-critical work:

1. 1Priority tiers: classify requests into tiers. Tier 1 (checkout, login) always served. Tier 2 (search, browse) served when capacity allows. Tier 3 (analytics, recommendations) shed first.
2. 2Feature flags: use feature flags to disable expensive features in real-time. "Disable personalised recommendations" reduces CPU load by 30% and lets the core survive.
3. 3Response quality: serve lower-quality responses: smaller images, fewer results, no auto-complete. Each reduction frees capacity for more requests.

Implementation patterns:

• Fallback decorators: wrap each dependency call with a fallback that returns cached/default data when the call fails or times out.
• Bulkheads: isolate thread pools per dependency. The recommendation service getting slow should not consume threads needed for checkout.
• Feature flags: runtime toggles that disable non-critical features without a deployment.
• Read-only mode: if the write path fails, switch to read-only. Users can browse but not purchase. Better than nothing.

The key interview signal: candidates who proactively build the degradation matrix, rather than waiting to be asked "what if X fails," demonstrate senior-level thinking. The matrix should be a design artifact, not an afterthought.

Cell architecture is the ultimate blast-radius containment pattern. Instead of running one large system where a correlated failure takes down everyone, you partition into N independent cells, each serving a fraction of traffic.

How it works:

1. 1Cell router at the edge receives every request. It extracts a shard key (user ID, tenant ID, or account ID) and hashes it to determine which cell should handle the request.
2. 2Each cell is independent: its own app tier, its own database, its own cache, its own message queue. No shared state between cells except the routing table.
3. 3Cell failure is contained: if Cell 3 has a bad deployment, a DB failure, or a cascading retry storm, only Cell 3's users are affected. Cells 1, 2, and 4-N continue serving normally.

Cell sizing and operations:

• Size each cell identically. Same instance types, same DB schema, same configuration. This simplifies operations enormously — you operate one cell template, not N snowflakes.
• Each cell serves 5-10% of traffic. This means a cell failure affects at most 10% of users. For five-nines availability across the system, each individual cell only needs 99.99% availability.
• Scale by adding cells, not by making cells bigger. This avoids the "blast radius grows with scale" problem.

Deployment safety:

Deploy to one cell first (canary cell). Monitor error rates, latency, and business metrics for 15-30 minutes. If clean, roll to the next batch of cells. If not, roll back only the canary cell — 95% of users never saw the bad code.

Cross-cell concerns:

The hard part of cell architecture is cross-cell queries. If a user in Cell 1 wants to see data about a user in Cell 3, the request must be routed cross-cell. Strategies:

• Avoid cross-cell reads by routing all data for a user/tenant to the same cell.
• Federation layer for the rare cross-cell queries (admin dashboards, global search).
• Async replication of aggregated data to a global read store for analytics.

Real-world usage: AWS Route 53 uses cells. Each cell handles DNS queries for a subset of hosted zones. A cell failure affects only those zones. This is how Route 53 achieves 100% SLA — the only AWS service with that guarantee.

Chaos engineering is the discipline of experimenting on a production system to build confidence in its ability to withstand turbulent conditions. Put simply: break things on purpose, before they break by accident.

The chaos engineering cycle:

1. 1Hypothesise: "If we kill a random app instance, the LB should route around it within 30 seconds and error rate should not spike above 0.1%."
2. 2Inject failure: kill the instance (or inject latency, corrupt a response, block a network path).
3. 3Observe: measure error rate, latency, and user impact.
4. 4Learn: did the system behave as hypothesised? If not, fix the gap and re-test.

Types of failure injection:

Principles:

• Start small. First chaos experiment: kill one instance in staging. Build confidence before touching production.
• Minimise blast radius. Use canary/cell architecture to limit impact. Netflix runs Chaos Monkey only during business hours when engineers are watching.
• Automate. Manual chaos testing doesn't scale. Schedule experiments to run continuously. AWS FIS and Gremlin support automated experiment plans.
• Measure everything. Without observability, chaos engineering is just breaking things. You need real-time dashboards showing error rates, latency percentiles, and business metrics.

Game days: quarterly exercises where the team simulates a major failure (region outage, DB corruption, DDoS) and practices the incident response. Game days validate not just the technology but the runbooks, communication channels, and escalation paths. If your last game day was "we discovered our runbook was wrong," that's a success — you found it before a real incident.

The interview signal: mentioning chaos engineering shows operational maturity. Saying "we'd run chaos experiments to validate our HA claims" is worth more than describing perfect architecture that's never been tested.