Webhooks: Retryable Event Notifications Over HTTP

Polling asks another service repeatedly:

Has anything changed yet?

A webhook reverses that relationship.

A webhook is an HTTP callback sent to a registered endpoint when an event occurs.

It provides timely notification without constant polling, but delivery crosses an unreliable network. Receivers must authenticate events, handle duplicates, and process retries safely.

A concrete example: payment succeeded

A customer completes payment at a provider.

The provider sends:

POST /webhooks/payments
Content-Type: application/json
 
{
  "eventId": "evt-8842",
  "type": "payment.succeeded",
  "paymentId": "pay-91",
  "orderId": "order-42"
}

The shop verifies the message and marks the order paid.

The customer's browser is not the trusted source of payment success. It can close, lose network access, or be manipulated.

Event versus command

A webhook usually reports a fact:

payment.succeeded

It should not depend on the receiver deciding whether the provider's payment succeeded.

Events are named in the past tense and carry stable identity.

A command such as charge this card is better represented as an authenticated API request from the caller to the provider.

Clear direction prevents confusion about ownership.

Registration

The provider needs an endpoint URL and usually event subscriptions.

Registration may be:

• Configured in a dashboard
• Created through an API
• Managed per account or tenant
• Verified with a challenge request

Protect registration because changing the destination can redirect sensitive events.

Notify owners when endpoints or secrets change.

Avoid allowing arbitrary internal URLs that could turn the provider into a server-side request forgery tool.

Authenticate the sender

HTTPS encrypts traffic and authenticates the endpoint to the sender.

The receiver still needs evidence that the request came from the provider.

Common methods include:

• HMAC signature
• Public-key digital signature
• Mutual TLS
• Provider-specific signed token

IP allowlists can add defense but are brittle and should not be the only proof.

Do not trust an event merely because its JSON looks correct.

Signature verification

A provider may sign:

timestamp + "." + rawRequestBody

The receiver:

1. Reads the signature header.
2. Reconstructs the signed bytes.
3. Computes or verifies the signature.
4. Compares safely.
5. Checks timestamp tolerance.

Verify before processing.

Use the provider's maintained library when available.

Small differences in whitespace or JSON parsing can change bytes, so preserve the raw request body until verification is complete.

Replay protection

An attacker who captures a valid signed request may resend it.

Defenses include:

• Timestamp freshness
• Unique event ID
• Stored processed-event record
• Expiring signature

Timestamp checks require reasonably synchronized clocks.

Event-ID deduplication protects both malicious replay and normal provider retries.

Do not use timestamp alone when a valid event could be replayed several times within the accepted window.

At-least-once delivery

Many webhook systems provide at-least-once delivery:

• The event should arrive.
• It may arrive more than once.

Exactly-once delivery cannot generally be guaranteed across independent systems and networks.

The receiver needs idempotency.

For payment.succeeded, applying the same eventId twice must not credit the order twice or send two shipments.

Duplicate delivery is expected behavior, not an exceptional bug.

Acknowledge quickly

Providers often expect a successful status within a short timeout.

A robust endpoint:

1. Verifies the signature.
2. Validates basic shape.
3. Stores the event or enqueues it durably.
4. Returns success.
5. Processes business work asynchronously.

Do not hold the connection while generating reports, sending email, or calling several services.

Slow acknowledgment causes retries, increasing duplicate load.

Return success only after the event is durably accepted.

Queue the work

A durable queue separates inbound delivery from processing.

Benefits include:

• Burst absorption
• Controlled concurrency
• Retry management
• Independent worker deployment
• Clear backlog monitoring

The queue can also deliver messages more than once, so workers remain idempotent.

If queue publishing and event recording must be atomic, use a database-backed inbox or equivalent transactional pattern.

Avoid acknowledging an event that exists only in process memory.

Ordering is not guaranteed

Events may arrive:

subscription.cancelled
subscription.updated

even if update occurred first.

Causes include:

• Parallel delivery
• Retry
• Network delay
• Several provider regions

Use:

• Event timestamp
• Resource version
• Sequence number
• Fetch-current-state fallback

Do not blindly set local status based on arrival order.

For stateful resources, compare versions or retrieve the authoritative current object.

Thin and fat events

A thin event contains identifiers:

{
  "type": "customer.updated",
  "customerId": "c-42"
}

The receiver calls the API for current data.

Benefits:

• Smaller payload
• Current state
• Less sensitive data in transit

Costs:

• Another API call
• Dependency on provider availability
• Event no longer contains historical state

A fat event includes more data, improving self-contained processing but increasing schema, privacy, and staleness concerns.

Choose according to workflow and audit needs.

Event schemas evolve

Webhook contracts need:

• Stable event type
• Unique ID
• Creation time
• Resource identity
• Schema version where needed
• Documented optional fields

Additive changes are safest when receivers ignore unknown fields.

Changing the meaning of an existing field is breaking.

Consumers may process events long after creation, so old payload versions can remain in retries and dead-letter queues.

Retries and dead letters

Providers commonly retry non-success responses with increasing delays.

Document:

• Which status codes trigger retry
• Schedule
• Maximum attempts
• Retention
• Manual replay

Permanent failures should move to a dead-letter process or visible failed-event list.

Operators need enough context to diagnose and safely replay after correction.

Unlimited rapid retries can overload an already failing receiver.

Endpoint failures

Return status based on whether retry can help:

• Temporary database outage: fail so provider retries.
• Invalid signature: reject permanently.
• Unsupported event type: often acknowledge and ignore, depending on contract.
• Malformed provider payload: record and escalate without endless retry.

Some providers retry every non-2xx response regardless of cause.

Receiver design should account for that behavior and avoid noisy retry loops.

Secret rotation

Webhook signing secrets should rotate.

A safe transition may accept signatures from:

• Current secret
• Previous secret for a limited overlap

Track which key verified each event.

Remove old keys after all senders and delayed deliveries have migrated.

Never print signing secrets in logs or store them in source code.

Multi-tenant routing

A platform may receive webhooks for many customer accounts.

The event must map to the correct tenant through trusted data:

• Provider account ID
• Registered endpoint identity
• Verified signed field

Do not trust an unsigned tenant ID from the payload.

Authorization and data isolation apply to background event processing just as they do to user requests.

One customer's event must never update another customer's records.

Observability

Track:

• Events received
• Signature failures
• Acknowledgment latency
• Duplicate count
• Queue depth
• Processing success and failure
• Oldest unprocessed event
• Retry and dead-letter count
• Processing delay from event creation

Correlate provider event ID with local workflow and resource IDs.

Logs should avoid full sensitive payloads unless protected and necessary.

Operationally, a webhook system is an event pipeline, not one controller method.

Reconciliation closes gaps

Even robust delivery can fail through:

• Misconfiguration
• Expired endpoint
• Software bug
• Extended outage
• Provider incident

Periodic reconciliation compares local state with the provider:

List payments changed since checkpoint.
Compare with processed events.
Repair missing records.

Webhooks provide timely notification. Reconciliation provides eventual confidence.

Critical financial or account workflows should use both.

Knowledge check

1. Why should payment status not rely only on the customer's browser?
2. Why must signature verification use the raw request body?
3. What does at-least-once delivery require from the receiver?
4. Why should a webhook endpoint acknowledge quickly after durable acceptance?
5. What problem does periodic reconciliation solve?

The one idea to remember

A webhook is an authenticated, retryable event notification over HTTP. Treat duplicates, reordering, delay, and missed delivery as normal possibilities, then use idempotency, durable queues, observability, and reconciliation to remain correct.