Architecture workshop · Backend & Distributed Systems

Design an idempotent notification platform

Turn a product brief into capacity estimates, an API contract, a delivery pipeline, and a failure strategy for email, push, and in-app notifications.

60–75 min Intermediate Paper, whiteboard, or diagramming tool

Lab outcome

A reviewable one-page design with explicit scale assumptions, idempotency boundaries, retry rules, and delivery-state ownership.

Before you start

Basic understanding of HTTP APIs and message queues.
A diagramming surface and a place to write assumptions.
Willingness to make and document trade-offs instead of searching for one perfect architecture.

Guided procedure

Complete the lab

Write the brief and non-goals

Design for 5 million users, 20 million notifications per day, and a 10× traffic burst. Support email, mobile push, and in-app delivery. Require user preferences and an auditable delivery status. Exclude campaign authoring and billing.

Checkpoint

• Functional requirements and non-goals are separate.
• At-least-once internal processing and duplicate-safe delivery are stated explicitly.

Calculate the first capacity numbers

20 million notifications per day is about 232 per second on average. A 10× burst is about 2,320 per second before fan-out. Record message size, retention, and channel fan-out assumptions rather than hiding them.

Capacity worksheettext

Average requests/sec = 20,000,000 / 86,400 ≈ 232
Burst requests/sec   = 232 × 10 ≈ 2,320

Assume:
- average request payload: ______ bytes
- average channels per notification: ______
- status-event retention: ______ days
- maximum acceptable enqueue latency: ______ ms

Checkpoint

• Every number has a unit.
• Unknown values are written as assumptions, not facts.

Define the idempotent API boundary

Create a request contract that lets a caller safely retry. Decide whether the caller supplies the idempotency key and how long the platform remembers it.

Example contracttext

POST /v1/notifications
Idempotency-Key: order-8472-shipped-v1

{
  "recipient_id": "user-123",
  "template": "order-shipped",
  "channels": ["push", "email"],
  "data": { "order_id": "8472" }
}

202 Accepted
{
  "notification_id": "ntf_...",
  "status": "accepted"
}

Checkpoint

• Repeating the same request cannot create a second logical notification.
• The response distinguishes accepted from delivered.

Draw the delivery pipeline

Include an API edge, authentication, idempotency store, durable log or queue, preference service, channel workers, external providers, delivery events, status store, retries, and a dead-letter path. Mark who owns each state transition.

Checkpoint

• The synchronous request path ends after durable acceptance.
• Slow email or push providers cannot block notification ingestion.
• Every retry carries a stable notification and channel-delivery identifier.

Run the ambiguous-timeout failure drill

Assume the email provider accepted a message but your worker timed out before receiving the response. Decide how you prevent a retry from sending a duplicate, how long you retry, and when the message moves to manual review or a dead-letter queue.

Failure tabletext

Failure: provider times out after possible acceptance
Detection: ______________________________________
Retry policy: ____________________________________
Provider idempotency key: ________________________
Final state after retry exhaustion: ______________
Operator evidence: _______________________________

Checkpoint

• The design does not equate timeout with failure.
• The retry policy has a limit and observable final state.

Verification

The API accepts safe client retries.
The queue or log absorbs the documented burst rate.
User preferences are evaluated before channel delivery.
Delivery attempts have durable, queryable states.
The ambiguous-timeout scenario has a duplicate-control strategy.

Cleanup

No runtime resources were created. Remove any real identifiers from the design before sharing it, then keep the sanitized diagram as your portfolio artifact.