Idempotency: Making Repeated Requests Safer

Networks can deliver the same logical request more than once.

A client times out and retries. A queue redelivers an unacknowledged message. A user double-clicks. A proxy repeats a request after a connection closes.

Idempotency makes repeated delivery produce one intended effect instead of duplicate consequences.

It is especially important for payments, orders, reservations, and any operation that cannot safely happen twice.

A concrete example: creating a payment

A client sends:

POST /payments
Idempotency-Key: checkout-8472-payment

The payment succeeds, but the response is lost. The client retries with the same key.

The server finds the original operation and returns its recorded result instead of creating a second charge.

Mathematical meaning

An operation is idempotent when applying it repeatedly has the same intended state effect as applying it once.

Conceptually:

f(f(x)) = f(x)

"Set account status to suspended" can be idempotent. "Add 10 credits" is not, because each repetition changes the balance again.

HTTP method expectations

HTTP defines methods such as PUT and DELETE as idempotent in their intended semantics.

• Repeating "replace this resource with this representation" should leave the same final state.
• Repeating "delete this resource" should keep it deleted.

Responses may differ, such as first deletion returning success and later deletion reporting absence, while the intended resource state remains the same.

POST can be made idempotent

Creation commonly uses POST, which is not inherently idempotent.

An idempotency key identifies one logical creation attempt. The server stores a relationship among:

• caller,
• key,
• request fingerprint,
• processing status,
• and final response.

This adds application-level semantics beyond the HTTP method.

Key generation

Keys should be:

• unique per logical operation,
• stable across retries,
• hard to collide accidentally,
• and generated before the first attempt.

A random UUID is common. A meaningful business identifier may work when it has the correct uniqueness scope.

Generating a new key for every retry defeats deduplication.

Scope

The same key may be valid only within:

• one account,
• one API endpoint,
• one merchant,
• one region,
• or one operation type.

Scope must be defined. If all customers share one global key namespace, accidental or malicious collision can cause one request to receive another customer's result.

Authentication context should be part of lookup.

Request fingerprint

If a key is reused with different request data, the server should not silently return an unrelated old result.

It can store a hash of normalized relevant request fields and reject mismatched reuse:

same key + different amount = conflict

Normalization must be stable so harmless formatting differences do not create false mismatch.

Atomic claim

Two copies can arrive at nearly the same moment.

The server needs an atomic way to claim the key:

• unique database constraint,
• conditional insert,
• transactional record,
• or compare-and-set operation.

A "check, then insert" sequence without atomicity can allow both requests to see no record and perform the side effect twice.

Processing states

An idempotency record may move through:

• started,
• completed,
• failed permanently,
• or uncertain.

If a duplicate arrives while the first is still processing, the server can wait, return a processing response, or ask the client to poll. Starting duplicate work is normally the wrong choice.

Storing responses

Many implementations store the original status and response body.

Returning the same response gives retries a stable result. Sensitive data should be minimized, encrypted where appropriate, and retained only as long as needed.

Large responses may be reconstructed from an operation record instead of stored verbatim.

Failure behavior

Should a failed attempt be cached?

It depends:

• a validation error for the same immutable request can be stable,
• a temporary unavailable error may deserve another attempt,
• an uncertain downstream result should not be treated as definitely failed.

The API contract must define which outcomes bind the key.

Retention window

Idempotency records cannot always live forever.

A server may retain them for hours or days based on:

• maximum client retry period,
• business risk,
• storage cost,
• privacy,
• and reconciliation needs.

After expiry, reusing the same key may create new work, so clients need the retention guarantee.

Natural idempotency

Sometimes the business model already has a unique identifier.

"Create shipment for order 8472" can enforce one active shipment record per order. A database uniqueness constraint prevents duplicate creation regardless of request key.

Natural constraints are often stronger than a temporary cache, though business rules must allow them.

Idempotent consumers

Message queues commonly provide at-least-once delivery, so a consumer may see the same message again.

The consumer can record processed event identifiers in the same transaction as its state change. If the identifier already exists, it skips the effect.

Recording completion separately after the effect leaves a crash window where duplicate work can occur.

The inbox pattern

An inbox table stores received message identifiers and processing status.

Within one local database transaction, a consumer:

1. inserts the message identifier,
2. applies the business change,
3. commits both.

A duplicate identifier violates the unique constraint and proves that the local effect was already committed.

Idempotency and exactly-once claims

Networks and brokers often deliver at least once or at most once.

End-to-end "exactly once" effects usually emerge from:

• duplicate delivery,
• idempotent processing,
• transactional state,
• and deduplication.

Marketing language about exactly-once transport should be examined at the full business boundary.

Side effects outside one transaction

A workflow may update a database and call an external provider.

One local transaction cannot atomically include both systems. Use operation records, provider idempotency keys, outbox patterns, status reconciliation, and compensating actions.

Idempotency has to extend across every boundary that can repeat.

Idempotency is not immutability

An idempotent API can still support updates.

"Set profile name to Mira" repeated twice has one final state. A later request with a new operation identifier can set it to Leela.

Idempotency groups repeated attempts of one logical operation; it does not prohibit future change.

Security concerns

Idempotency records must not leak another user's response.

Protect against:

• cross-account key reuse,
• guessable identifiers,
• stored sensitive bodies,
• unbounded key creation,
• and denial-of-service through retained records.

Rate limits and authenticated scope remain necessary.

Testing idempotency

Test:

• sequential duplicate requests,
• simultaneous duplicates,
• same key with changed payload,
• timeout after side effect,
• crash during processing,
• record expiry,
• and unauthorized key reuse.

Concurrency tests are critical because the dangerous case is often two first attempts racing.

Knowledge check

1. What final-state property defines an idempotent operation?
2. Why must a retry reuse the original idempotency key?
3. What race exists in a non-atomic check-then-insert?
4. Why should the server compare request fingerprints?
5. How does an inbox table help a message consumer?

The one idea to remember

Idempotency identifies repeated attempts as one logical operation and returns one durable outcome. Correct implementations define key scope, compare request content, claim keys atomically, preserve useful status, and extend duplicate protection across side-effect boundaries.