Service eventBounded metricsControlled tracesoperation · outcomeattempt · durationraw personal dataAllowlistMinimize before export
Sensitive attributes are removed before export; aggregate outcomes determine when to investigate. Synthetic design, not a vendor protocol. Scroll the diagram horizontally for detail.

When a session stalls, the first useful question is where the operation stopped making progress. A large log containing a user’s entire request often answers a different question: what sensitive information did the service happen to receive? Design telemetry around decisions, boundaries and outcomes so an investigation can proceed without collecting everything.

This guide builds an original observability plan for the synthetic platform used in the manual. It assumes separate client, platform and provider boundaries, a local operation record and an event delivery worker. No telemetry is collected by this website. The examples are design material, not live measurements or evidence of a deployed monitoring service.

Give each identity one job

A request ID identifies one request. An operation ID connects work that continues across retries. An event ID identifies a logical event. A delivery ID distinguishes attempts to deliver that event. A trace ID connects instrumented spans. Reusing one string for all five jobs makes a dashboard look correlated while hiding which thing actually repeated.

W3C Trace Context standardizes propagation through traceparent and tracestate. It provides an interoperability format for tracing context. It does not make incoming context trustworthy, grant authorization or guarantee that every downstream system participates in the trace.

At a trust boundary, validate tracing inputs and apply a deliberate propagation policy. Keep authorization decisions separate. If an external provider does not preserve your trace context, record a safe mapping between your operation reference and its documented response reference rather than inventing a continuous trace that was never observed.

Record decisions, not whole bodies

Start with a small allowlist: operation kind, internal operation reference, attempt number, state transition, coarse outcome category and duration. For a catalog importer, add candidate version and page count. For a callback worker, add whether an effect was committed or a duplicate was ignored.

{
  "example": true,
  "operation_id": "launch-example-017",
  "event_id": "example-event-001",
  "attempt": 2,
  "outcome": "duplicate_ignored",
  "duration_ms": 50,
  "route_template": "https://example.invalid/callback"
}

This example deliberately excludes credentials, raw launch URLs, personal profiles and full request headers. Even an internal identifier may become sensitive when joined to other data, so pseudonymous does not mean harmless. Define who can query it, how long it is retained and whether it is necessary for the specific investigation.

OpenTelemetry’s sensitive-data guidance describes attribute removal, filtering and allowlist-based redaction. In our design, minimization begins in application instrumentation and an export filter provides a second boundary. A collector filter cannot erase a secret that was already written to an unfiltered application log.

Distinguish attempts from effects

Count delivery attempts separately from committed effects. A spike in requests can mean successful recovery, harmful duplication or ordinary traffic growth. Without an outcome dimension, those cases can look identical. Likewise, an HTTP success rate can remain high while a worker falls far behind after durable acceptance.

Use a bounded set of outcome categories in metrics. Keep per-operation identifiers in controlled logs or traces instead of making every identifier a metric label. Otherwise, the number of time series grows with traffic and creates unnecessary operational cost. This is a proposed design constraint, not a claim about a particular vendor’s pricing model.

Signal Question answered Common blind spot
Age of oldest pending operation Is accepted work making progress? Fast failures may have low age
Committed effects by outcome Is useful work completing? Does not show sender certainty
Duplicate-ignored count Are repeats being suppressed? Needs traffic volume for context
Dead-letter growth Is recovery exhausting its budget? Effects may already exist

The callback lab makes that last distinction visible. With a short timeout and idempotency disabled, an envelope can enter dead-letter after committing multiple effects. Alert wording should say that acknowledgement recovery failed, not confidently state that processing never occurred.

Build an investigation sequence

Imagine a synthetic incident: the client shows pending, the platform has an unknown provider outcome, and delivery attempts increase. Begin with the operation record. Establish which durable state exists before interpreting log messages. Then identify the last confirmed transition and the first boundary with an uncertain outcome.

Check whether the same event identity spans the repeated attempts. If identity changes on each retry, deduplication may be unable to help. Compare the committed effect count with the attempt count. If a downstream system is involved, use its documented lookup or support reference; a local trace cannot prove a remote mutation did not happen.

Record the conclusion at the same confidence level as the evidence. “No acknowledgement observed” is supported by a missing response. “Provider created nothing” needs stronger evidence. This vocabulary makes incident handoffs more useful because the next engineer can see what still needs to be established.

Alert on an action you can take

Define an alert together with its operator action. An old pending-operation threshold might trigger reconciliation. A growing dead-letter queue might trigger inspection of the dominant terminal reason. A single malformed input may require counting and inspection without paging someone immediately.

Thresholds need workload context. The examples here do not supply production baselines or service-level objectives. Establish acceptable delay, measure normal variation, then test the alert against a controlled failure. A fixed “ten errors” rule can be noisy for a busy system and too slow for a small critical workload.

Sampling also changes evidence. A sampled trace set cannot establish that an unsampled failure never occurred. Pair sampled diagnostic detail with aggregate counters or durable operation records when deciding whether a whole population completed. Document which signal is complete and which is deliberately partial.

Test the telemetry boundary

Inject recognizable synthetic sensitive markers into a local request, then exercise validation failures, worker exceptions and timeouts. Search every resulting application log and exported telemetry payload. The test passes when prohibited fields are absent, not merely when a normal happy-path span looks clean.

Next run one successful effect, a duplicate ignored and a retry exhaustion. Assert that counters classify each outcome correctly. Send a malformed trace header and verify that the request remains subject to ordinary authorization and that logging the parse failure does not reproduce an unsafe raw value.

Finally, disable the telemetry exporter and confirm the business operation follows its documented failure policy. An observability outage should not accidentally become a hidden synchronous dependency. These tests complement idempotency: one protects the effect boundary, while the other makes that boundary inspectable without unnecessary exposure.