Event-driven / saga

The monolith transaction is not a saga at all — it is the starting point you are evolving away from. A single relational database handles all writes inside one BEGIN/COMMIT block. If any step fails, the database rolls back atomically. No compensations needed, no event bus, no distributed state.

Why it works. Relational databases have spent 40 years perfecting ACID transactions. Write-ahead logs, MVCC, lock managers — the entire stack exists to make BEGIN/COMMIT reliable. At moderate scale (single-digit thousands of writes per second), this is the right answer. The monolith gives you strong consistency, simple debugging (one stack trace, one log stream), and trivial rollback.

Why it stops working. The moment you need independent deployment per team, independent scaling per component, or different storage engines per domain (SQL for orders, NoSQL for recommendations), the monolith transaction breaks. You cannot wrap a Postgres INSERT and a DynamoDB PutItem in the same BEGIN/COMMIT. You also cannot hold database locks across a network call without killing throughput.

When to stay here. Until your team is large enough that deploy coupling is the bottleneck, or until your read/write patterns diverge enough that one database cannot serve all access patterns. Many startups should stay here longer than they think.

In a choreographed saga, each service listens for events from upstream services and publishes its own domain events when its step completes. The workflow emerges from the union of all subscriptions. There is no single process that "knows" the full workflow.

Mechanics. Order service publishes OrderPlaced. Payment service subscribes, captures payment, publishes PaymentCaptured. Inventory service subscribes to PaymentCaptured, reserves stock, publishes StockReserved. Shipping service subscribes to StockReserved, creates the label, publishes ShipmentCreated. Each service owns its own database and its own event publication.

Compensations in choreography. If Inventory fails to reserve stock, it publishes StockReservationFailed. Payment service subscribes to that event and issues a refund. Order service subscribes and marks the order as failed. Each compensation is triggered by an event, not by a coordinator — so every service must know which failure events to listen for and what to do.

The debugging problem. "What is the state of order #12345?" To answer, you must query Order, Payment, Inventory, and Shipping services and piece together the timeline. In practice, teams build a saga-status aggregator that materialises a view from all events — which is halfway to building an orchestrator anyway. This is why choreography works for simple flows and collapses at 4+ steps.

When choreography fits. Simple reactive workflows: on user signup, send welcome email and create analytics profile. Two consumers, no compensation needed, stable event schemas. The moment you need compensations or the workflow has branches, switch to orchestration.

An orchestrated saga uses a central coordinator — Temporal, AWS Step Functions, Cadence, Netflix Conductor — to drive the workflow. The coordinator calls each service's activity endpoint in sequence, waits for the result, decides the next step, and invokes compensations in reverse order on permanent failure.

The workflow is code. In Temporal, a saga is a function: call reserve_inventory(), then capture_payment(), then create_shipping_label(). If capture_payment throws a non-retryable error, the framework invokes release_inventory(). You can read the workflow top-to-bottom, unit-test it with mocked activities, and deploy it independently from the services.

State is durable. The orchestrator persists the saga state to its own database after every step. If the orchestrator process crashes, a new instance picks up from the last persisted state. This is fundamentally different from choreography, where a crash means the workflow is partially complete with no single owner to resume it.

Retry semantics. Temporal distinguishes transient errors (retry with backoff) from permanent errors (invoke compensations). Step Functions use a Catch/Retry DSL. In either model, the orchestrator handles retries so individual services do not need their own retry loops — they fail fast and let the coordinator decide.

Cost and coupling. The orchestrator is a piece of infrastructure you must operate: database, workers, monitoring. Each service couples to the orchestrator's activity interface (gRPC, HTTP, or SDK-specific). For high-request-volume workflows (>10K/s), orchestrator throughput and database write amplification become sizing concerns.

The production-grade orchestrator adds three concerns on top of the basic coordinator pattern: an explicit state machine, the outbox pattern for every event publication, and a dead-letter queue for poison messages.

Explicit state machine. Each saga instance has a state: STARTED → PAYMENT_CAPTURED → INVENTORY_RESERVED → SHIPPING_CREATED → COMPLETED (happy path) or → COMPENSATING → COMPENSATED → FAILED (rollback path). The orchestrator persists this state in a database row keyed by saga_id. Every transition is a conditional UPDATE (WHERE state = expected_state) — so concurrent duplicate deliveries cannot advance the saga past a step it has already completed.

Outbox for every event. When the orchestrator advances the state machine, it writes the new state AND an outbox event in the same database transaction. A CDC connector (Debezium) or a poller reads the outbox and publishes to the event bus. This eliminates the dual-write hazard. If the orchestrator crashes between the DB write and the event publish, the outbox retains the event and the publisher picks it up on restart.

Dead-letter queue. Messages that cannot be processed after N retries land in a DLQ. A human or automated remediation process inspects them. Without a DLQ, a poison message blocks the entire partition. The DLQ also serves as an audit trail of infrastructure failures.

Saga timeouts. A background sweeper queries for sagas stuck in a non-terminal state beyond a configurable TTL (e.g., 30 minutes for checkout). It moves them to COMPENSATING. This catches coordinator crashes, network black holes, and services that accepted a request but never responded. The sweeper must be idempotent — running it twice on the same saga must not double-compensate.

All writes happen inside a single BEGIN/COMMIT block. Simplest possible approach. Works until deploy coupling or single-DB throughput becomes the bottleneck.

Services communicate via domain events on a shared bus. No coordinator. Each service subscribes to the events it cares about. Compensations are event-driven too.

A Temporal/Step Functions coordinator calls each service in sequence, persists state, retries transient failures, and invokes compensations on permanent failure. Workflow is readable code.

Outbox eliminates dual writes. Explicit state machine prevents double-processing. DLQ catches poison messages. Timeout sweeper detects dead sagas. Distributed tracing (saga_id as trace context) gives end-to-end visibility.

The choreography-vs-orchestration decision is the first fork in saga design. Getting it wrong costs 6-12 months of migration.

Choreography wins when the workflow is simple (2-3 linear steps), the event schemas are stable, and no compensation is needed. Example: on UserRegistered, send a welcome email and create an analytics profile. Two subscribers, both idempotent, no rollback scenario. Choreography also wins when you need extreme decoupling — adding a 10th consumer to UserRegistered requires zero changes to the publisher.

Orchestration wins the moment any of these are true: (a) the workflow has 4+ steps, (b) compensations exist, (c) the workflow has conditional branches (if payment fails, do X; if inventory fails, do Y), (d) you need to query the saga state by business ID, (e) the workflow must complete within an SLA (timeout detection). In practice, this is most production workflows.

The hybrid model. Some teams use orchestration for the core saga (order → payment → inventory → shipping) and choreography for side-effects (publish OrderConfirmed to Kafka for analytics, email, search indexing). The orchestrator drives the critical path; the event bus fans out to non-critical consumers. This is the most common production pattern at scale.

Migration path. Starting with choreography and migrating to orchestration is painful — you must introduce a coordinator, replay in-flight sagas, and deprecate the old event subscriptions without losing messages. Starting with orchestration and adding choreographic side-effects is cheap. When in doubt, start with orchestration.

Interviewer signal. If a candidate says "choreography" for a 5-step order workflow, the interviewer will probe: "How do you know the current state of order #12345?" If the candidate cannot answer without describing an orchestrator, they have proved the point.

A compensating transaction is not a database rollback. It is a new forward action that semantically reverses a previous action's business effect. The original action's database writes are committed and visible; the compensation writes new data that corrects the outcome.

Design rules for compensations:

1. 1Every forward step must have a defined compensation before you build it. If you cannot articulate the compensation, the step is not saga-safe. Example: sending a push notification has no compensation — you cannot un-notify a user. Put non-compensable steps last in the saga so they execute only after all compensable steps have succeeded.

1. 1Compensations must be idempotent. The compensation may be invoked more than once due to retries, replays, or duplicate event delivery. A refund compensation must check whether a refund has already been issued (by idempotency key) before issuing another one.

1. 1Compensations must be ordered. If the saga executed steps 1, 2, 3 and step 3 failed, compensations run in reverse: compensate 3, compensate 2, compensate 1. The orchestrator guarantees this ordering. In choreography, ordering is implicit in the event graph — and much harder to verify.

1. 1Compensations must handle concurrent state. Between the original step and the compensation, other processes may have read or acted on the intermediate state. A stock reservation that is compensated (released) may conflict with another saga that reserved the same stock. The compensation must handle this — typically via optimistic concurrency (version checks) or pessimistic locks at the service level.

1. 1Compensations must never fail permanently. A compensation that throws a non-retryable error leaves the system in a partially compensated state with no automated recovery. Design compensations to retry with exponential backoff. If they still fail after max retries, route to a dead-letter queue for human intervention.

The semantic gap. Some business actions have imperfect compensations. A payment capture can be refunded, but the refund may take 5-10 business days to appear on the customer's statement. An email confirmation was sent and cannot be un-sent. These are "imperfect compensations" — they correct the system state but not the user experience. Acknowledge this in your design and handle it with customer communication (apology email, status page).

The dual-write problem is the #1 source of silent data drift in event-driven systems. The service writes to its database, then publishes an event to the message bus. If the process crashes between the two operations, the database has the new state but the event was never published — downstream services are permanently out of sync, and nothing detects the drift.

How the outbox works:

1. 1The service starts a database transaction.
2. 2It writes the business data (e.g., INSERT INTO orders).
3. 3In the same transaction, it writes an outbox row (INSERT INTO outbox(event_id, topic, payload, created_at)).
4. 4It commits the transaction. Both writes succeed or both fail — atomicity is guaranteed by the local database.
5. 5A separate process — a CDC connector (Debezium on Postgres WAL, DynamoDB Streams, MongoDB change streams) or a polling loop — reads unpublished outbox rows.
6. 6It publishes each event to the message bus (Kafka, SQS, Pub/Sub).
7. 7It marks the outbox row as published (or deletes it).

CDC vs polling. CDC (Change Data Capture) tails the database's write-ahead log and emits change events in real time. Debezium + Kafka Connect is the canonical stack. Latency: single-digit milliseconds. Polling queries the outbox table on a schedule (every 100ms-1s). Simpler to operate but higher latency and more database load. Start with polling; switch to CDC when latency matters.

Exactly-once at the publish boundary. Because the publisher marks rows as published after a successful bus write, a crash between publish and mark means the event is published again on restart. This is at-least-once at the publisher. Consumers must still be idempotent.

Outbox table schema. Minimal: event_id (UUID), aggregate_type, aggregate_id, event_type, payload (JSONB), created_at, published_at (nullable). Index on (published_at IS NULL, created_at) for the poller. Partition by created_at for cleanup.

Cleanup. Outbox rows are write-once, read-once artifacts. Delete or archive rows where published_at is not null and created_at is older than the retention window (e.g., 7 days). Without cleanup, the outbox table grows unbounded.

Every production saga needs an explicit state machine. Without one, the system cannot answer "what step is this saga on?" and cannot recover from crashes.

State transitions. A checkout saga might have these states: CREATED → PAYMENT_PENDING → PAYMENT_CAPTURED → INVENTORY_RESERVED → SHIPPING_CREATED → COMPLETED. On failure: any state → COMPENSATING → COMPENSATED → FAILED. Each transition is a conditional database update: UPDATE sagas SET state = 'PAYMENT_CAPTURED' WHERE saga_id = ? AND state = 'PAYMENT_PENDING'. The WHERE clause ensures that concurrent duplicate deliveries cannot advance the saga past a state it has already left.

State machine + outbox. The state transition and the outbox event are written in the same database transaction. This means the state machine is always consistent with the events published. If the orchestrator crashes, the new instance reads the saga state from the database and resumes from the last completed step.

Saga table schema. Columns: saga_id (UUID, PK), saga_type, state, payload (JSONB — the input data), created_at, updated_at, completed_at, error (nullable text). Index on (state, updated_at) for the timeout sweeper. Index on (saga_type, state) for operational dashboards.

Timeout sweeper. A background process runs on a schedule (every 30s) and queries: SELECT * FROM sagas WHERE state NOT IN ('COMPLETED', 'FAILED', 'COMPENSATED') AND updated_at < NOW() - INTERVAL '30 minutes'. It moves these sagas to COMPENSATING. This catches: (a) orchestrator crashes, (b) services that accepted a request but never responded, (c) network partitions that silently dropped the response.

Concurrency control. When the sweeper moves a saga to COMPENSATING, it uses the same conditional update pattern: UPDATE ... WHERE state = 'INVENTORY_RESERVED'. If the saga advanced between the sweeper's read and write, the update affects zero rows, and the sweeper skips it. This prevents double-compensation.

Operational dashboards. With an explicit state machine, you can build dashboards: sagas by state (how many are in COMPENSATING?), sagas by age (any older than 1 hour in a non-terminal state?), compensation success rate. These are impossible with implicit choreography state.

Every production message bus delivers at-least-once. Kafka consumers that crash after processing but before committing the offset will re-receive the message. SQS visibility timeout expiry causes re-delivery. Pub/Sub acknowledgement failures cause re-delivery. Temporal retries activities on transient errors. Every saga step and every compensation must be idempotent.

Strategy 1: Idempotency key in a seen-events table. Before processing, the consumer checks: SELECT 1 FROM seen_events WHERE event_id = ?. If the row exists, skip. Otherwise, process and INSERT the event_id in the same transaction. This is the most general approach and works for any operation. The seen_events table needs a TTL-based cleanup (delete rows older than the message bus retention period).

Strategy 2: Database upsert keyed on business ID. Instead of INSERT, use INSERT ... ON CONFLICT (order_id) DO UPDATE SET status = 'CAPTURED'. The upsert is naturally idempotent — replaying the same message produces the same database state. This works when the operation is setting a value, not incrementing one.

Strategy 3: Conditional updates with version checks. UPDATE orders SET status = 'SHIPPED', version = version + 1 WHERE order_id = ? AND version = ?. If the version has already advanced, the update affects zero rows. This is optimistic concurrency control and works for state machine transitions.

Strategy 4: Naturally idempotent operations. Some operations are idempotent by design. Setting a boolean flag (is_paid = true) is idempotent. Sending a DELETE for a resource that may already be deleted is idempotent (return 204 either way). Decrementing a counter is NOT idempotent — use a reservation model instead (reserve a specific reservation_id, not "decrement stock by 1").

The increment trap. A common bug: a payment step increments a running total. Replayed, it increments again. Fix: use a ledger model. Each payment event creates a ledger entry keyed by (saga_id, step). The balance is SUM(ledger entries). Replaying the event tries to insert a duplicate ledger entry; the unique constraint rejects it.

Testing idempotency. Every saga step should have a test: execute the step, then execute it again with the same input. Assert the side effects are identical. This is the single most important test in a saga codebase.

A saga spans multiple services, databases, and message brokers. Without purpose-built observability, debugging a failed checkout requires SSH into 4 services, grepping 4 log streams, and mentally correlating timestamps. At 1000 sagas per second, this is impossible.

Distributed tracing. Propagate a saga_id (or use it as the trace_id in OpenTelemetry) across every service call, event header, and log line. When a saga fails, a single query by saga_id returns the complete timeline: which steps succeeded, which failed, what the compensation outcome was, and how long each step took. Without this, you are blind.

Saga state dashboard. With an explicit state machine, build a real-time dashboard: (a) sagas by state — pie chart of COMPLETED / COMPENSATING / FAILED / in-progress, (b) sagas by age — histogram of time-in-flight for non-terminal sagas, (c) saga throughput — completions per second, (d) compensation rate — what percentage of sagas trigger compensations. Alert on: compensation rate > 5% (something upstream is broken), sagas in non-terminal state > 30 minutes (dead saga).

Dead saga detection. A saga is "dead" when it is stuck in a non-terminal state beyond its expected SLA. The timeout sweeper moves dead sagas to COMPENSATING, but the alert fires first so an engineer can investigate. Common causes: a service is down and not processing messages, a message was routed to the DLQ and nobody noticed, the orchestrator crashed and the new instance has not yet resumed (rare with Temporal, which handles this automatically).

DLQ monitoring. The dead-letter queue is a leading indicator of system health. A message in the DLQ means a saga step failed after max retries. Monitor: DLQ depth (should be zero in steady state), DLQ age (oldest message — indicates how long the issue has been unnoticed), DLQ event types (which step is failing?). Auto-alert on any DLQ message.

Structured logging. Every log line from a saga step includes: saga_id, step_name, step_number, saga_state, event_id, idempotency_key. Use structured JSON logging (not printf-style). Ship to a centralised log aggregator (Elasticsearch, Datadog, Loki). A query like saga_id = "abc123" returns the complete saga history across all services.

Metrics to own. saga_completed_total (counter, by saga_type), saga_failed_total (counter, by saga_type and failure_step), saga_duration_seconds (histogram, by saga_type), saga_compensation_total (counter, by saga_type and compensated_step), saga_inflight (gauge, by saga_type and current_state). These six metrics give complete operational visibility.