How Do I Implement Cursor-based Pagination?

Key takeaways

Cursor-based pagination retrieves large datasets one page at a time, using a token from the API to mark where the next request should continue.
A cursor doesn't guarantee consistency. Stable ordering, changing records, and the provider's pagination semantics decide whether a sync can skip or duplicate records.
Reliable pagination takes more than a loop. Retries, idempotent processing, and durable checkpoints let an integration recover from failures without losing data.
Pagination and incremental sync are different problems. A pagination cursor tracks progress through one result set but a sync checkpoint tracks which changes you've already synchronized.

Your customer needs to pull 100,000 customer records from a third-party API. The API returns 100 per request, so you implement a loop in their integration that fetches pages until it runs out. It works perfectly in a sandbox with 73 records.

But, in production, the dataset changes while it's being read.

With offset pagination, records can shift between requests. With cursor pagination, the result depends on what the provider's cursor represents and what consistency guarantees the API offers. Either way, a sync can easily produce duplicates or gaps in the data.

The cursor loop is simple: fetch a page, read the continuation value, and repeat. However, making that loop reliable when working with rate limits, network failures, and interrupted runs takes a bit more work.

What is cursor-based pagination?

Pagination retrieves a large collection in smaller batches. Instead of asking for "page 5" or "skip the first 200 records," the client sends a cursor or continuation token defined by the API.

Here's typical flow:

Send an initial request with a page size and no cursor.
Receive records and (if more are available) a continuation value.
Send that value with the next request, keeping filters and sort order consistent.
Repeat until the API tells you that the result set is complete.

For example:

1
GET /customers?limit=100&cursor=abc123

1234
{
  "data": [{ "id": "101", "name": "Example A" }],
  "pagination": { "next_cursor": "abc456", "has_more": true }
}

This response is illustrative since some APIs return a next cursor, some use a record ID such as Stripe's starting_after, and others provide a full next-page URL in a Link header. Relay-style GraphQL connections use edges and pageInfo; Prismatic's GraphQL API uses first and after, and its documented example returns nodes with pageInfo fields such as hasNextPage and endCursor. Whatever the details, follow the endpoint's contract. See Stripe's pagination docs, GitHub's REST pagination docs, and the Relay Cursor Connections specification.

If a provider states that its token as opaque, pass it back with zero changes. Don't parse or edit it since the provider may change its internal format. Remember, though, that some APIs explicitly define a cursor as a public record ID, so don't assume all cursors are opaque.

A cursor parameter also doesn't tell you how the server will do pagination. It might use keyset queries, a snapshot, or something else. Those details determine what the performance and consistency guarantees will be.

Cursor-based vs offset-based pagination

Offset pagination uses a numeric position, such as ?limit=100&offset=200. It can work well for small, mostly static result sets. But records can shift between requests: deleting a record near the beginning can cause a later offset to skip one, while an insertion can make a record appear twice. Deep offsets may also be costly because the database still has to compute skipped rows; see PostgreSQL's LIMIT and OFFSET documentation.

Keyset pagination continues from values in an ordered result set rather than a row count. With a matching index, the server can seek from the continuation point instead of computing and discarding a deep offset. However, a cursor token alone doesn't guarantee that implementation or performance.

	Offset-based	Keyset pagination with a cursor
Request	Page number or offset	Continuation token or sort-key values
Changing data	Offsets can shift	Avoids offset shifts; mutations can still affect results
Large datasets	Deep offsets may be expensive	Can be efficient with a matching index
Random page access	Usually supported	Usually sequential

Offsets remain primarily useful for lightweight views where user need to jump to a page. For large integration syncs, keyset pagination is often a better fit when the API supports it. But, neither approach guarantees a consistent snapshot.

Implementing a pagination loop in TypeScript

First, check the API docs for the page-size limit, cursor location, termination signal, cursor lifetime, sort order, and which parameters must remain constant. Don't assume a short or empty page means the result set is complete; use the documented continuation metadata.

Here's a loop for the illustrative response above. upsertCustomer() represents an application-specific, idempotent write.

123456789101112131415161718192021222324252627282930313233343536373839404142434445
interface Customer {
  id: string;
  name: string;
}

interface CustomerPage {
  data: Customer[];
  pagination: { next_cursor: string | null; has_more: boolean };
}

async function syncCustomers(): Promise<void> {
  let cursor: string | undefined;
  const seen = new Set<string>();

  while (true) {
    if (cursor !== undefined) {
      if (seen.has(cursor)) throw new Error("Repeated cursor");
      seen.add(cursor);
    }

    const url = new URL("https://api.example.com/customers");
    url.searchParams.set("limit", "100");
    if (cursor !== undefined) url.searchParams.set("cursor", cursor);

    const response = await fetch(url, {
      headers: { Authorization: `Bearer ${process.env.API_TOKEN}` },
    });
    if (!response.ok) throw new Error(`API request failed: ${response.status}`);

    // Validate the response at runtime in production.
    const page = (await response.json()) as CustomerPage;
    const next = page.pagination.next_cursor;

    if (page.pagination.has_more) {
      if (next === null || seen.has(next)) {
        throw new Error("Missing or repeated pagination cursor");
      }
    }

    for (const customer of page.data) await upsertCustomer(customer);

    if (!page.pagination.has_more) return;
    cursor = next!;
  }
}

The loop processes one page at a time instead of storing all the records in memory. It also checks the next cursor before processing, so a repeated cursor won't cause the same page's writes to run again. For very long traversals, bound the number of pages or use another cycle-detection strategy – instead of keeping every cursor indefinitely.

Where should the loop live?

For a result set that comfortably fits within memory and execution limits, a connector can fetch all pages and return one combined result. For a large or unpredictable result set, expose the cursor and page size as action inputs and return one page at a time. The integration can process each page before requesting the next.

Prismatic's custom connector pagination guidance describes both patterns. For very large syncs, process one page or a small batch per execution to keep memory use and run time predictable.

Handling failures, retries, and checkpoints

The loop is the easy part, in general. Beyond that, cursors can expire, APIs can rate-limit requests, and destination writes can fail partway through a page.

Retry without advancing the cursor

A 100,000-record sync with 100 records per page needs at least 1,000 requests. If a request fails with a 429, timeout, or transient 5xx, retry that same request with the same cursor. Advance only after you have received and processed the page. Honor Retry-After when provided; otherwise use bounded exponential backoff with jitter.

Save checkpoints after processing

A cursor in a local variable disappears when a process exits. For a long-running sync, store the last safely completed cursor durably. Here’s how:

Load the saved cursor.
Fetch that page and process every record.
Save the next cursor only after processing succeeds.
Repeat.

If processing fails halfway through a page, leave the checkpoint unchanged. The next run may re-fetch and reprocess some records, but it won't skip unfinished work. If processing succeeds but saving the checkpoint fails, that page will be processed again – so writes must be idempotent.

For example, upsert by the source record's unique ID rather than blindly inserting:

1234
INSERT INTO customers (external_id, name)
VALUES ('101', 'Example A')
ON CONFLICT (external_id)
DO UPDATE SET name = EXCLUDED.name;

An upsert won't make external side effects (such as sending an email or creating an invoice) idempotent. Use deduplication keys or a transactional outbox when needed. Also ensure that marking a sync as complete retires its old checkpoint.

Cursors may expire after minutes or hours, or be tied to the exact filters and sort order used. Find out how long they remain valid and define a recovery strategy: restart the traversal, resume from an application-level mark, or reconcile the destination with the source.

Account for concurrent changes and runs

A record updated during a traversal may move ahead of a cursor already passed, or appear twice, depending on the provider's ordering. Stable, unique ordering and snapshot support reduce risk, but the cursor alone doesn't guarantee either. Key destination writes by unique source IDs and plan for reconciliation when missed records matter.

Two workers can also start from the same checkpoint and process the same page. A slower worker may overwrite a newer checkpoint with an outdated one. For a single cursor chain, allow only one active run per customer and sync configuration, using a lock or lease. Conditional checkpoint updates provide another safeguard. Pages are sequential, so parallelize records within a page only if all work completes before advancing the checkpoint.

Pagination and incremental sync

A pagination cursor marks your place in one result set. An incremental-sync checkpoint (or watermark) notes how far you've synchronized changes from the source. Some APIs combine these concepts; others use a timestamp filter plus ordinary pagination.

For example, updated_after defines the set of changed records, while cursor moves through that set:

1
GET /customers?updated_after=2026-11-02T10:00:00Z&limit=100&cursor=abc123

Use a bounded sync window when the API supports it. Page through the window, and advance the durable watermark only after the whole window succeeds. Account for records sharing timestamps, low timestamp precision, delayed visibility, and deletions that may require provider events or tombstones. Overlapping windows plus deduplication can help with late-arriving changes.

Periodic reconciliation compares source and destination records to find missing, stale, or inconsistent data. Pagination moves records while reconciliation checks that the systems agree.

Test and monitor the integration

A loop that fetches three pages in a sandbox says little about production. Test the cases most likely to break progress:

A single page, multiple pages, and an empty page that still has a next cursor.
A repeated or missing cursor when the API says more data is available.
A 429, timeout, or 5xx response; the integration should retry without advancing the cursor.
A destination failure halfway through a page; a restart should safely reprocess records.
An expired cursor, concurrent run, or records changing during a traversal.

Mocked responses make these failures easy to reproduce. Then test against the provider to confirm cursor lifetime, ordering, termination behavior, and response shape.

In production, track pages fetched, records processed, retry attempts, run time, and checkpoint lag. Log the counts and failure reasons, but avoid logging full cursors if they contain sensitive data. A run can succeed request by request while falling behind, so monitor whether its completed checkpoint is keeping pace with the source.

Building cursor pagination into your own API

If you expose data to integration developers, document a predictable cursor contract.

Choose a stable, unique ordering. A non-unique field such as created_at needs a unique tiebreaker such as id. Define null ordering and index the fields used by the filters and sort.
Use keyset queries. For an ascending PostgreSQL ordering:

12345
SELECT id, name, created_at
FROM contacts
WHERE (created_at, id) > (:cursor_created_at, :cursor_id)
ORDER BY created_at ASC, id ASC
LIMIT :limit + 1;

Fetch one extra row. For a requested page size of 50, query for 51. If the extra row exists, return the first 50, set has_more to true, and build the next cursor from the last returned row. This avoids a separate count query.
Keep tokens opaque without implying secrecy. Base64url-encoded JSON is easy to decode. Use it only for non-sensitive state; use a random server-side token if contents must remain hidden. Sign or authenticate tokens to detect tampering, encrypt sensitive contents, and never treat a cursor as authorization.
Bind tokens to query parameters. Reject a cursor reused with different filters or sort order, for example by storing or verifying a canonicalized filter fingerprint.
Document the contract. Explain termination metadata, page-size limits, default ordering, cursor lifetime, and which parameters must remain constant. Prefer forward-only paging unless clients need backward navigation.

If users need "jump to page 40," cursor pagination may not be the right fit. Offsets can serve that interface; keyset cursors are better suited to efficient sequential traversal.

Where Prismatic fits

Prismatic supports pagination patterns in custom connectors and large data syncs. For modest result sets, a custom action can fetch all pages within its execution limits. For larger sets, return one page and its continuation value so the integration can process records incrementally. See Handling API Pagination.

For high-volume syncs, Prismatic supports batching. In code-first (code-native) integrations, batchFlowTrigger can return pagination state so the platform invokes it for subsequent batches. Custom connector triggers can also be configured for pagination and batching. These patterns help manage memory and execution time; they don't change the provider's cursor semantics or make destination writes consistent. See Large Data Syncs in Custom Triggers and Large Data Sync.

Prismatic's own GraphQL API uses first and after; its documented example returns nodes and pageInfo. The docs recommend explicitly sorting results to reduce the risk of missed or duplicated records. See Pagination with Prismatic's API.

The cursor is the start

Writing a loop that fetches 100 records at a time takes some effort, but understanding the provider's contract, processing records safely, and recovering when something fails is what makes it work reliably.

A cursor can avoid shifting-offset problems when the provider uses an appropriate keyset strategy, but it doesn't guarantee a consistent snapshot or safe recovery. Use stable ordering, retry without advancing progress, checkpoint only after successful processing, and make writes idempotent. For recurring syncs, add incremental watermarks and periodic reconciliation.

A properly-built integration knows when it hits a rate limit or an expired request (and responds accordingly).

Ready to build productized integrations without reinventing pagination? See how Prismatic supports pagination and large data syncs.

How Do I Implement Cursor-based Pagination that Survives Mid-sync Failures and Rate-limit Pauses?