Feed pagessnapshot v12Candidate v12validate completelyPublished pointerv11 → v12Previous version v11collectpublishread v11 untilv12 is completeFailure on page 2?Keep the old pointer.
Readers stay on the previous complete snapshot until the staged version passes validation. Synthetic design, not a vendor protocol. Scroll the diagram horizontally for detail.

A content catalog looks like a list until an update fails halfway through. Then one part describes yesterday’s world, another describes today’s, and a missing page can masquerade as a deletion. Synchronization is a publication problem: collect a coherent candidate, validate it, then decide when readers should see it.

Our example platform maintains a local catalog of interactive puzzles. An upstream feed exposes stable content IDs, localized labels and a snapshot version. These are invented capabilities for a worked design. A real feed may provide weaker guarantees, and its synchronization strategy must acknowledge that difference rather than silently assuming a transaction across page requests.

Separate identity from presentation

Use a stable upstream content identifier plus a source namespace as the record identity. Titles are presentation, not keys. Renaming “Moon Maze” or translating it into another language should not create a second item. Likewise, two sources can legitimately publish the same short identifier without referring to the same content.

Keep capability and availability fields separate. A record saying that content supports a device category does not prove it is currently available to every user. A local catalog can describe what the platform knows while a launch-time check enforces the current operational contract. That separation prevents stale catalog metadata from becoming an authorization decision.

{
  "example": true,
  "snapshot": "catalog-example-v12",
  "items": [{
    "id": "puzzle-example-04",
    "version": 7,
    "labels": { "en": "Moon Maze", "fr": "Labyrinthe lunaire" },
    "state": "available",
    "asset_uri": "https://example.invalid/assets/puzzle-example-04.svg"
  }],
  "next_page_token": "example-page-two"
}

The tokens and fields above are synthetic. We display the asset URI as text and never request it. The JSON demonstrates record boundaries; it does not advertise an available game or a working content endpoint.

Walk the pages without guessing

Capture the snapshot identifier on the first page and require subsequent pages to belong to it. Treat continuation tokens as opaque. A token is not a page number to increment or a value to edit. Preserve the request filters associated with the pagination session.

Google’s AIP-158 pagination guidance describes opaque continuation tokens and notes that a short or even empty page may still have a continuation token. The lesson for our importer is concrete: stop at the feed’s documented end marker, not because the last batch contained fewer rows than expected.

Track previously seen tokens and impose a page budget. If a feed returns the same continuation token repeatedly, fail the candidate import instead of looping forever. This is a defensive rule in our importer design, not a behavior promised by AIP-158. Store the failed candidate’s page count and reason so an operator can reproduce it without exposing full payloads.

If the source cannot pin a snapshot, document the weaker consistency model. You might accept a bounded drift window, reconcile with a second pass, or ask for a change feed. Offset pagination over a mutating collection can skip or repeat entries; local deduplication can remove repeats but cannot reconstruct records that were never observed.

Stage before publishing

Write imported records under a candidate version that readers cannot see yet. Validate required fields, uniqueness, supported locales and legal state transitions. Keep the currently published pointer unchanged until the candidate is complete. A process crash during page three then leaves an incomplete candidate, not a half-updated public catalog.

In this design the publish operation switches one local pointer from the previous validated version to the new version. Readers pin that version for their request. Old versions remain available long enough for in-flight reads, then a separate cleanup policy removes them. This costs additional storage but gives a clear rollback boundary.

Candidate condition Publish decision Investigation
All pages collected and validated Switch the version pointer Record completion
Repeated continuation token Keep prior version Inspect the feed cursor
Duplicate stable ID with conflicting values Reject candidate Compare record versions
Large unexpected deletion set Require review under local policy Check completeness first

Model deletion explicitly

Absence only implies deletion when the source promises a complete snapshot and the importer proves it collected that snapshot. A timed-out page is not evidence that its records were removed. For incremental feeds, use explicit tombstones or another documented removal signal instead of inferring deletion from a missing record.

Out-of-order updates also need a policy. Our example stores a monotonically comparable record version and refuses to replace a newer record with an older one. If the source only offers timestamps with uncertain precision or clock behavior, that policy cannot be assumed. Record the uncertainty and choose a reconciliation mechanism that matches the actual contract.

Distinguish unavailable from deleted. An unavailable item may need to remain visible in history or saved user state. A deleted item may require a stable explanation rather than a broken link. These are product decisions carried by the schema, not cleanup details to decide after the importer ships.

Make locale fallback predictable

Choose a fallback order and make it testable: requested locale, configured default locale, then a neutral identifier. Avoid silently mixing fields from different versions of a record. If a French title exists but its description does not, decide whether a partially localized record is acceptable and signal the fallback where it matters.

Keep locale normalization explicit. A language tag is not simply a lowercase country name. For this example we only accept the two keys shown in the payload, so the importer does not pretend to implement a comprehensive internationalization engine. Production requirements may include script variants, regional preferences and accessibility alternatives for media.

Break the importer on purpose

Construct a local three-page fixture. Fail page two after page one has been staged. Assert that the published pointer still selects the previous complete catalog. Resume the import with the original snapshot identity and verify that repeated records do not duplicate local items.

Then serve an empty page with a continuation token, repeat a token, change the snapshot halfway through and send two conflicting versions of one ID. Each test should have a declared outcome. Add a missing locale and a tombstone to verify fallback and deletion behavior independently.

This guide designs an importer; it does not ship a live feed connector or vendor selection service. Request lifecycle explains why catalog visibility and launch success are separate facts. Observability suggests useful import signals: candidate age, validation failures, publication lag and the last successfully published version.