Design the evidence before the failure
Build a six-scenario acceptance matrix with explicit oracles, bounded replay and privacy-minimal traces. Learn what a passing simulation cannot establish.
An injected error is useful only if the test can tell a correct recovery from a plausible-looking trace. Start with an oracle: the observable condition that must hold after the experiment. Then choose a failure that can challenge it. Otherwise a retry demo becomes a sequence of orange and green states with no argument for what the colors mean.
This chapter builds an original acceptance matrix for the local Scenario Workbench. Its finite receiver is deliberately inspectable: two numbered events, a scoped identity record, a bounded attempt budget and virtual time. It neither measures a provider nor certifies a production architecture.
Draw the failure boundary first
The most important pair of experiments differs by a single boundary. A pre-commit failure leaves no effect from that attempt. A lost acknowledgement occurs after the effect has been recorded. Both look like failure to a sender waiting for confirmation, but the correct recovery evidence is different.
For the first case, a later successful attempt may create the missing effect. For the second, a later matching delivery should read the existing outcome. Counting one effect per attempt would fail the second case. Counting a timeout as proof of no effect would misdescribe it before a retry even began.
The workbench places committed and failed in separate trace rows for a lost acknowledgement. This is not decorative detail. It makes the proposition “failure means nothing happened” directly falsifiable inside the model. The request lifecycle guide develops that uncertain-outcome boundary further.
An acceptance matrix with six different oracles
| Experiment | Deliberate fault | Required evidence | Incorrect conclusion to reject |
|---|---|---|---|
| Temporary failure | First attempt fails before commit | Bounded retry; eventually one effect per identity if budget permits | Every error is permanent |
| Lost acknowledgement | Commit succeeds; first ACK is lost | Existing effect survives; retry acknowledges without reapplying it | Timeout proves absence |
| Duplicate callbacks | A matching envelope is delivered again | Duplicate state appears; effect count does not increase | Every delivery is a new business event |
| Out-of-order arrival | Sequence 2 arrives before 1 | Early receipt retained; commit order is 1 then 2 | Arrival row number is business order |
| Identity conflict | Same ID, different payload revision | Quarantine state; no additional effect for the conflict | An ID match makes all content safe |
| Exhaustion | Sequence 1 always fails before commit | Attempt limit reached; sequence 2 remains visibly blocked | A finished animation means all work completed |
Each oracle is specific enough to fail. For example, the ordering test examines commit and acknowledgement sequences, not only the final number of rows. The exhaustion test requires the blocked item to remain in the summary. A UI that hides unresolved work would fail even if the pure function had returned it correctly.
Replay is about inputs as well as a seed
The workbench’s replay operation restores the last completed configuration: scenario, seed, attempt budget, base delay and cap. Editing a control after a run does not silently change the saved experiment. Exported JSON includes that configuration and a model version, so a trace can be understood outside the page.
The seed controls only the small jitter term in a retry delay. It does not choose whether the specified failure occurs. The delay formula is bounded by the selected ceiling, even when the ceiling is below the base. All time advances virtually: increasing a delay never leaves a real timer running for that interval.
AWS’s retry-with-backoff guidance discusses the load and idempotency consequences of retries. Our particular formula and its millisecond values are teaching choices. They are not an AWS configuration recommendation or a copied provider SLA. In a real acceptance matrix, record the service’s documented retry limits and the application’s own recovery budget separately.
Test a family, then explain a counterexample
A deterministic function makes it practical to enumerate small parameter families. The local tests vary six scenarios, four seeds, five attempt budgets, four base delays and three ceilings. They check 1,440 configurations for stable replay, bounded trace length, monotonic virtual time, capped delays, one effect per identity and accounting between deliveries and attempts.
Those checks supplement the six concrete cases; they do not replace them. A broken implementation could produce an empty trace for every input and still be deterministic. The out-of-order test therefore asserts actual business sequence, and the conflict test asserts that quarantine occurs. The lost-ACK test with a budget of one records a useful exception: the sender can exhaust its attempts while the effect exists. An invariant that prohibited that combination would encode the wrong model.
This is property-like enumeration over a finite set, not a proof over all possible distributed executions. Concurrent workers, crashes between durable writes, adversarial messages and network partitions are outside this receiver. A deployment test plan must address them where they are relevant.
Keep the evidence smaller than the incident
The exported example contains synthetic event IDs, transition names, numbers and bounded explanations. It has no request body from a person, credentials or endpoint. A real delivery trace should be designed just as deliberately, with an allowlist of fields required to distinguish the tested outcomes.
W3C Trace Context limits the purpose of its propagation fields and warns about sensitive information. OpenTelemetry’s sensitive-data guidance describes removal and filtering mechanisms. Those documents support minimizing what is propagated and exported; they do not set a universal retention period for this publication or your system.
Choose retention, access and redaction policies for the actual environment, and test the redaction path with synthetic hostile values. An export button that saves everything merely moves a privacy problem into a file. This lab avoids arbitrary imports and limits its own JSON export to 64 KiB because fixed, reproducible scenarios answer the teaching question without accepting untrusted payloads.
Close the loop in the interface
Run each scenario, replay after changing the visible seed, export it, and clear it. Verify that the trace explains the last executed inputs, replay restores them, the file matches the visible state and clearing removes all effects and blocked items. Try an invalid number: the previous result should remain identifiable rather than being silently replaced by an empty success.
A passing local run establishes these narrowly defined properties of this version. It does not establish availability, throughput, legal compliance or delivery guarantees elsewhere. That boundary makes the evidence usable: the next engineer can see exactly which claim was tested and what still needs a different experiment.
Source scope and review
Retry with backoff pattern
AWS Prescriptive Guidance — primary source. Publication date: undated. Accessed 2026-11-17. Bounded retries and growing delays can help with transient failures; retries can also amplify load and unsafe effects.
Trace Context
W3C — primary source. Publication date: 2021-11-23. Accessed 2026-11-17. Defines traceparent and tracestate for propagating distributed tracing context. Context is not authorization.
Handling sensitive data
OpenTelemetry — primary source. Publication date: undated. Accessed 2026-11-17. Describes removing or filtering sensitive attributes, including allowlist-based redaction processors.
The numbered stream, buffer, trace format and tests are original demonstrations. Source references support their stated boundaries; they do not certify this model as a deployed integration.