Loading…

Design a Webhook Delivery Service — SystemRound

Design a Webhook Delivery Service

hardSystem design50 min5 stages

Asked atStripeGitHubTwilioShopifyPlaidSlackSquare

Design the webhook layer for a B2B platform (think Stripe events, GitHub webhooks). When something happens internally (charge succeeded, PR opened), an event must be delivered to every customer endpoint subscribed to that event type — durably, signed, and with retries on failure. Customers get a dashboard to inspect delivery attempts and replay failed events. Endpoints fail constantly: timeouts, 5xx, slow consumers, customers down for hours. The hard problems are slow-customer isolation, retry-storm protection, and dashboard replay without overwhelming a recovering customer.

Best after a few full reps. Expect follow-up questions, edge cases, and deeper trade-off discussion.

What this problem tests

Subscription Management APIOutbound Webhook ContractAPI QualityPipeline ArchitecturePer-Customer IsolationRetry Loop & Backoff

Round shape

5 stages

Time budget

50 min

Feedback loop

Grade anytime

Guided practice·Primary loop

Guided practice

Workspace-first, hints visible, stage retry available. The cheap, repeatable loop — build the answer shape before you take it under pressure.

Stage-by-stage workspace instead of a blank page.
Grade one stage or the whole answer whenever you want.
Compare your answer against reference checkpoints and model answers.

Solve once, compare against the checklist, then come back to the weak stage instead of starting over.

Mock interview·Pressure test

Mock interview

Strict timer, hints hidden, debrief deferred to the end. Use this once you can already structure a clean answer and want to pressure-test pacing and pushback.

Best once the answer shape is already in your head.
Pressure-test pacing, pushback handling, and communication.
Use diagnosis after the interview for exact misses and next study steps.

Best after one structured rep · timed · focused on pacing and communication.

Requirements

This is the framing pass. A strong answer quickly defines what the system must do, what quality bar it has to hit, and the numbers that will justify the rest of the design.

First 5 min of the round

What must exist

Functional Requirements

7 items

1At-least-once delivery — every accepted event lands on every active subscription, or terminates as dead-letter. Customer dedupes by event_id. Exactly-once is impossible across an unreliable HTTP boundary.

2Retry policy — exponential backoff with jitter; up to 12 attempts spread across 24 hours before dead-lettering.

3HMAC-SHA256 signing over (timestamp + body) using per-subscription secret. Receivers reject if timestamp older than 5 minutes (replay protection).

4Event-type filter per subscription (e.g. `charge.*` matches `charge.succeeded`, `charge.failed`).

5Dashboard with delivery history (each attempt: status, response, latency, body) and one-click replay for any failed event in last 30 days.

6Per-customer fairness — one slow endpoint must not delay deliveries for other customers.

7Below the line: dead-letter notification (email), per-subscription FIFO ordering as nice-to-have (not promised — Stripe-style).

What good looks like

Non-Functional Requirements

5 items

1p95 first-attempt latency < 10s from event creation. Slow customer endpoints affect their own latency only.

2Zero silent drops — every accepted event in a durable queue (Kafka 3x replicated) BEFORE producer ack.

3Audit retention 30 days for replay; older summaries only.

4Produce-path availability 99.95% — delivery-path availability is bounded by customer endpoint health, not gateway-controlled.

5Body limit 1 MB — anything larger is rejected at ingest with structured error.

Numbers to anchor the design

Scale Estimation

5 items

110K customers, ~200K subscriptions (avg 20 subs/customer; Stripe's number is similar).

250K events/sec internally; fan-out factor depends on how many subscriptions match each event (often 1-5). Average ~5K outbound deliveries/sec, peak 20K during business hours.

3Storage: `delivery_attempts` at ~5K/sec × ~500 bytes (no body in this row) = ~2.5 MB/sec → ~215 GB/day → ~6.5 TB/month with 30-day retention. Postgres + S3 archive, or columnar (ClickHouse).

4Event bodies: 50K/sec × ~2 KB avg = 100 MB/sec → ~8.5 TB/day with 7-day hot retention; archive to S3.

5Peak retry storm: a customer down 30 min then recovering = ~30 min × ~167 deliveries/sec for them = 300K queued retries for that customer alone. Multiplied across customers, this can dwarf steady-state by 5-10x. Plan for it.

What to cover

Day-1 failure modes to design for

4 items

1Slow customer endpoint (30s response time) → must not block other customers.

2Customer down for hours → retries pile up; recovery floods them.

3Customer ships a regex bug → 100% 5xx → eats retry budget pointlessly.

4Replay-all click on dashboard → 10K queued retries hit the customer's recovering endpoint at once.

How the round unfolds

Each stage has a distinct job. Treat them like separate deliverables instead of one giant answer, and the round becomes much easier to navigate.

4 design stages · 40 pts after framing

🔌

Stage 2~7 min10 pts

API Design

Define the contract clearly: the endpoints, auth boundary, error semantics, and the one or two decisions that matter most.

What you should produce

Define the customer-facing API and the outbound contract.

Strong answers cover

Subscription Management APIOutbound Webhook ContractAPI Quality

🏗️

Stage 3~12 min10 pts

High-Level Architecture

Lay out the main components and trace the write path, read path, and any async path cleanly.

What you should walk through

Walk me through the high-level architecture.

Strong answers cover

Pipeline ArchitecturePer-Customer IsolationRetry Loop & Backoff

💾

Stage 4~10 min10 pts

Data Model & Storage

Pick the store, show the schema or key model, and explain why that storage choice fits the access pattern.

What you should lock in

Walk me through the data model.

Strong answers cover

Schema DesignStorage ChoicesIndexes & Queries

📈

Stage 5~16 min10 pts

Scaling & Deep Dive

Name the first bottleneck, failure modes, and the trade-offs that keep the system fast and reliable under pressure.

What you should pressure-test

Time for the deep dive. The interesting failure modes here are slow-customer head-of-line blocking, retry storms when a down customer recovers, and...

Strong answers cover

Slow-customer isolationRetry-storm protectionReplay-burst handlingObservability

What a strong first rep looks like

Scope clearly

Translate the prompt into concrete requirements, scale, and trade-offs before drawing architecture.

Stay stage-specific

Give APIs in the API stage, data models in the storage stage, and failure modes in scaling. Don't blur them together.

Iterate fast

Grade early, compare to the reference checkpoints, fix the biggest misses, and re-submit the weak stage instead of starting over.