Skip to main content

Handling API Pagination

Pagination is a common pattern in APIs that return large sets of data. Instead of returning all records at once, APIs often return a subset of records (a "page") along with information on how to retrieve the next page. This helps reduce the amount of data transferred in a single request and improves performance.

Pagination in custom components​

Every application implements pagination differently. Some applications require you to pass page number and number of records to return as URL search parameters (i.e. ?page=5&page_size=20). In that case, it's your job to keep track of which page you're on.

Others return a "cursor" with the response (either in the body or as a response header). You can include that cursor with your next request to get another page of results.

As you build an action for an API that paginates, ask this question: is it reasonable to pull down all records at one time?

If the API you're interacting with returns 100 records at a time, for example, and you know that customers never have more than a few hundred records of a particular type, it probably makes sense to pack pagination logic into your custom action. That way, your customers don't need to keep track of page numbers or cursors - your action simply returns all results. In this example, Airtable returns a JSON payload with an offset property and array of records that we accumulate in a do/while loop:

Handling pagination within an action
export interface AirtableRecord {
  id: string;
  createdTime: string;
  fields: Record<string, unknown>;
}

export interface AirtableRecordResponse {
  offset: string;
  records: AirtableRecord[];
}

const listRecords = action({
  display: {
    label: "List Records",
    description: "List all records inside of the given table",
  },
  inputs: {
    airtableConnection: connectionInput,
    baseId: baseIdInput,
    tableName: tableNameInput,
    view: viewInput,
  },
  perform: async (context, inputs) => {
    const client = createAirtableClient(inputs.airtableConnection);

    const records: AirtableRecord[] = [];
    let offset = "";

    do {
      const { data } = await client.get<AirtableRecordResponse>(
        `/v0/${inputs.baseId}/${inputs.tableName}`,
        {
          params: {
            view: inputs.view,
            offset,
          },
        },
      );
      records.push(...data.records);
      offset = data.offset;
    } while (offset);

    return { data: records };
  },
});

On the other hand, if you know that your customers have a significant number of records stored in a third-party application (e.g. they have millions of records in their Airtable base), it's more memory-efficient to fetch a page of records at a time, processing each page before fetching the next page. In that case, we recommend adding offset, cursor, page_number, etc., as inputs of your action, and ensure that your action returns those values for the next iteration.

To sync a large dataset, consider building a trigger that fetches one page at a time and lets Prismatic process records in batches. See large data syncs in custom triggers and large data syncs for code-native integrations.

If the API you're working with returns link headers, we recommend the parse-link-header package, which can be used to extract the next URL to use when paginating.