Content Sync Explained

Agility gives you three ways to read content: the Content Fetch API (REST), the GraphQL API, and Content Sync. The first two answer a question each time you ask. Content Sync copies your content into a store you own and keeps that copy up to date, so your app reads locally instead of calling Agility.

This guide explains when to use each one, how the sync token works, and how to drive incremental syncs from webhooks. For the endpoint reference, see Content Sync API. For the JavaScript SDK, see Content Sync JS SDK.

Sync, Fetch or GraphQL?

	Content Fetch API (REST)	GraphQL API	Content Sync
What you get	One item, list, page or sitemap per request	Exactly the fields you select, several queries in one request	Every content item and page that changed since your last sync
Where your app reads from	Agility, on each request (through the CDN)	Agility, on each request	Your own store (files, database, cache)
Filtering and sorting	filter, sort, take (max 250), skip	Same filter syntax, per list	Whatever your store supports
Counts towards API rate limits	Uncached requests do	Uncached requests do	Only the sync calls themselves
Best for	Server-rendered and statically built sites, most apps	Pulling many related lists and fields in one round trip	Offline use, very high read volume, copying content into another system

Some rules of thumb:

• Start with the Fetch API or GraphQL. For most websites, requesting content when you render (with your framework's cache in front, and webhooks to clear it) is simpler than keeping a copy.
• Choose GraphQL when a page needs fields from several lists at once and you want one request with only those fields.
• Choose Content Sync when you need content without calling Agility at read time: an offline-capable app, a search index or data warehouse, a system such as Redis that other services read from, or a static build that should not re-download everything each time.

You can mix them. A common pattern is Sync for a search index or cache, and the Fetch API for preview.

How sync works

The Sync API is part of the Content Fetch API. It has two endpoints, one for content items and one for pages:

GET https://api.aglty.io/{guid}/{fetch|preview}/{locale}/sync/items?syncToken={token}&pageSize={n}
GET https://api.aglty.io/{guid}/{fetch|preview}/{locale}/sync/pages?syncToken={token}&pageSize={n}

Both take your APIKey header like any other Fetch API request. pageSize defaults to 500. Each response is an object with the items (or pages) in that batch and the next token. The shape (values are placeholders):

{
  "items": [
    { "contentID": 0, "properties": { "state": 2, "referenceName": "..." }, "fields": { } }
  ],
  "syncToken": 0
}

Each item has the same shape as an item from the Fetch API, so the field shapes in Field Types and What the APIs Return apply.

Use the fetch API type and a fetch key to sync published content, or preview and a preview key to sync the latest saved content for a preview environment.

The sync token lifecycle

The sync token is a number that marks how far through the change history you have read. Think of it as a bookmark.

1. First sync. Call with syncToken=0. You get the first batch of everything, plus a new token.
2. Keep paging. Call again with the token you just received. Each call returns the next batch and a newer token.
3. Stop when you are caught up. When a call returns no items and the token no longer advances, you have everything. The @agility/content-sync SDK stops as soon as the returned token is not greater than the one it sent.
4. Store the token. Save the last token with your synced data (per locale, and separately for items and pages).
5. Next time, resume. Start from the stored token, not from 0. You only receive what changed since then.

Things to get right:

• Keep one token per locale and per endpoint. Items and pages have separate tokens, and each locale is synced separately.
• Save the token after you save the content. If your process stops between the two, you re-read a batch rather than skip one. Re-reading is safe because each item simply overwrites the copy you already have.
• Treat 0 as "rebuild". To start over (for example after you clear your store, or if your store and token get out of step), sync from 0 again.

Deleted and unpublished content

Sync returns removals as well as changes. An item that was deleted or unpublished comes back with properties.state set to 3 (Deleted). Remove it from your store when you see that state. The @agility/content-sync SDK does this for both content items and pages.

Incremental sync driven by webhooks

You don't need to poll. Let Agility tell you when something changed, then sync from your stored token:

1. Create a webhook in Settings > Webhooks for Content Publish Events (and Content Save Events if you sync preview content). See Webhooks.
2. When a delivery arrives, acknowledge it with a 2xx response straight away, then run a sync in the background from your stored token.
3. Don't trust the payload as the content. The webhook tells you that something changed (contentID, referenceName, state, languageCode); the sync call gives you the current data, including removals.
4. Use the payload's languageCode to sync only the locale that changed.
5. Collapse bursts. Publishing many items sends many webhooks. Queue them, and run one sync for the batch. The SDK runs a sync at most once per second per locale.

Webhook delivery is at-least-once, so the same event can arrive twice. Syncing from a stored token is naturally safe here: a second sync with nothing new returns nothing. For signatures, retries and the webhook-id header, see Verifying Signed Webhooks and Webhook Events and Payload Reference.

The @agility/content-sync SDK

The @agility/content-sync package (version 1.2.5 at the time of writing) runs this loop for you:

npm install @agility/content-sync

import agilitySync from "@agility/content-sync"

const syncClient = agilitySync.getSyncClient({
  guid: process.env.AGILITY_GUID,
  apiKey: process.env.AGILITY_API_FETCH_KEY,
  languages: ["en-us"],
  channels: ["website"],
  isPreview: false
})

await syncClient.runSync()

What runSync() does, per locale:

• Reads the stored tokens (starting at 0 the first time) and syncs content items, then pages, 100 at a time.
• Saves each item to the store, and removes items whose state is Deleted.
• Waits and retries if the API reports that it is busy, for up to 10 minutes.
• If anything changed, downloads the flat and nested sitemaps for each channel you list.
• Checks URL redirections and saves them when they have changed.
• Saves the new tokens.

By default the store is the local filesystem, under .agility-files. You can plug in your own store (a database, Redis, or anything else) by implementing the store interface. See Content Sync JS SDK for the interface and for reading content back with syncClient.store. Call clearSync() to empty the store and start from scratch.

Full re-sync and recovery

Sync from 0 again when:

• you change how you store or transform content,
• your store was lost or partly written,
• you switch the same store between published (fetch) and preview content.

With the SDK, call clearSync() and then runSync(). With your own code, clear your store and the stored tokens, then sync from 0. A full sync reads every item, so on a large instance run it outside peak hours.

Related

• Content Sync API
• Content Sync JS SDK
• Content Fetch API
• GraphQL API
• Webhooks
• Fetch API Status Codes and Caching