Callbacks, retries & recovery
Separate delivery from processing. Design bounded retries, durable acknowledgement and a useful dead-letter workflow.
A callback is a delivery attempt across a boundary. Treating it as a one-time function call hides the interesting failures: the receiver can commit work and lose its response, or acknowledge a request before the work survives a crash. A useful design says what acknowledgement means and what happens when the sender never observes it.
Our example delivers one content-session event to a host platform. The payload, timings and queue are synthetic. We assume the sender can retry, the receiver can persist an inbox entry, and a worker applies the business effect. Authentication and request signing are necessary integration concerns but are outside this simulator’s model.
Separate receipt from completion
The receiving endpoint has three possible milestones: bytes arrived, an event was durably accepted, and the intended effect was committed. A successful HTTP response can represent different milestones in different contracts. Name the milestone in both the endpoint documentation and the operational dashboard.
In this design, the endpoint validates an envelope and writes it to a durable inbox before acknowledging. A worker later processes the inbox. This makes the endpoint fast without pretending that memory is durable. If persistence fails, the endpoint must not acknowledge durable acceptance. If the worker later fails, recovery belongs to the inbox processing policy.
Some providers document duplicate delivery and do not guarantee ordering. Stripe is a concrete example, as described in its webhook documentation. That source supports designing for duplicates; it does not establish the retry schedule or guarantees of an unrelated provider.
Classify before retrying
Our example uses three failure classes. A malformed envelope is terminal until the producer changes it. A temporary storage outage may be retryable. An acknowledgement timeout is ambiguous because the receiver might already have committed. Combining all three into one retry loop wastes capacity and obscures the cause.
| Observation | Example response policy | Recovery owner |
|---|---|---|
| Unsupported schema version | Reject with a stable reason | Producer changes the event |
| Inbox temporarily unavailable | Do not acknowledge durable receipt | Delivery retry policy |
| Response lost after commit | Preserve identity on retry | Receiver deduplicates |
| Worker exhausted attempts | Retain event and failure context | Operator or repair workflow |
Status codes alone may not identify the class. A contract can define a retryable status and a bounded Retry-After response, while a particular failure still needs inspection. Never treat every client error as a reason to retry indefinitely, or every server error as proof that nothing happened.
Bound the retry schedule
A retry policy needs an attempt limit, elapsed-time budget, backoff rule and terminal destination. Decide whether “three retries” means three additional attempts; in this manual it does, so the maximum is four total attempts. This small vocabulary choice frequently produces mismatched tests.
The lab uses deterministic delays of 250, 500, 1000, 2000 and 4000 milliseconds. Those are virtual intervals, not recommendations for production. They make the event trace reproducible. Real deployments often need randomized delays to prevent many failed operations from returning together, plus coordination between retry layers.
AWS’s retry with backoff guidance describes transient recovery, increasing waits and the risk of retry amplification. Our concrete delay sequence and budgets are original teaching choices, not copied service defaults.
Think about amplification across layers. If a caller retries three times and its downstream worker also retries three times, a single intention can trigger more remote work than either team expected. Assign one layer responsibility for a given retry boundary, and count attempts using a shared operation identity.
Reproduce an ambiguous timeout
Open Callback Lab. Set delivery to at-least-once, duplicates to zero, retries to two, timeout to 200ms and idempotency off. The example handler commits an effect at 400ms. The sender’s deadline has already passed, so it schedules another attempt. All three attempts commit, and the delivery finally enters the dead-letter state.
This is intentionally uncomfortable: dead-letter does not mean no effect occurred. It means the sender exhausted its acknowledgement policy. Turn idempotency on and repeat. The first attempt commits; the next sees the recorded event and acknowledges in 50ms. The trace now contains one effect and a duplicate ignored.
{
"example": true,
"event_id": "example-event-001",
"delivery": 1,
"attempt": 2,
"type": "duplicate_ignored",
"callback_uri": "https://example.invalid/callback"
}
The lab is sequential. It waits until one synthetic handler has completed before scheduling the next attempt. A real sender can retry while the first handler still runs. That concurrency makes an atomic duplicate claim essential; it is not simulated by simply checking a browser-side set.
Make dead letters actionable
Retain enough information to identify the operation, schema version, attempt history and terminal reason. Avoid storing credentials or complete user profiles in the failure envelope. An operator should be able to distinguish a validation repair from a delayed acknowledgement without first reading sensitive content.
Replay should be deliberate. Keep the original business identity when retrying the same effect. Record a new delivery identity for the operator action. If a correction changes the meaning of the operation, create a new event under an explicit correction policy instead of silently reusing the old key.
There is a capacity trade-off. Long retry windows increase the chance of surviving a prolonged outage but also increase retained state and the time an obsolete event can return. A short window makes failure visible quickly but shifts more work to reconciliation. Set retention and replay limits together.
Test the boundary, not the happy path
Test failure before durable receipt, after receipt but before acknowledgement, and after acknowledgement but before worker completion. Assert which queue owns each unfinished item. Verify that the maximum number of delivery attempts is one plus the configured retry budget, and that once mode performs exactly one attempt even when retry controls contain larger values.
Send a duplicate after a successful effect and inspect the mutation counter, not just the endpoint status. Inject an invalid schema and verify that it cannot consume an unlimited retry budget. Finally, replay an exhausted event after its deduplication retention expires and confirm that the resulting risk is documented.
The next question is how to make that mutation counter dependable under races. Read idempotency for the transaction boundary, and observability for the difference between delivery health and useful outcomes.