Handling API Pagination

Pagination in custom components

Every app implements pagination differently. Some apps require you to pass page number and number of records to return as URL search parameters (i.e. ?page=5&page_size=20). In that case, it's your job to keep track of which page you're on. Others return a "cursor" with the response (either in the body or as a response header). You can include that cursor with your next request to get another page of results.

As you build an action for an API that paginates, ask yourself this question: is it reasonable to pull down all records at one time?.

If the API you're interacting with returns 100 records at a time, for example, and you know that customers never have more than a few hundred records of a particular type, it probably makes sense to pack pagination logic into your custom action. That way, your customers don't need to keep track of page numbers or cursors - your action simply returns all results. In this example, Airtable returns a JSON payload with an offset property and array of records that we accumulate in a do/while loop:

Handling pagination within an action

export interface AirtableRecord {
  id: string;
  createdTime: string;
  fields: Record<string, unknown>;
}

export interface AirtableRecordResponse {
  offset: string;
  records: AirtableRecord[];
}

const listRecords = action({
  display: {
    label: "List Records",
    description: "List all records inside of the given table",
  },
  inputs: {
    airtableConnection: connectionInput,
    baseId: baseIdInput,
    tableName: tableNameInput,
    view: viewInput,
  },
  perform: async (context, params) => {
    const client = createAirtableClient(params.airtableConnection);

    const records: AirtableRecord[] = [];
    let offset = "";

    do {
      const { data } = await client.get<AirtableRecordResponse>(
        `/v0/${params.baseId}/${params.tableName}`,
        {
          params: {
            view: params.view,
            offset,
          },
        },
      );
      records.push(...data.records);
      offset = data.offset;
    } while (offset);

    return { data: records };
  },
});

On the other hand, if you know that your customers have a significant number of records stored in a third-party app (e.g. they have millions of records in their Airtable base), it's more memory-efficient to fetch a page of records at a time, processing each page before fetching the next page. In that case, we recommend adding offset, cursor, page_number, etc., as inputs of your action, and ensure that your action returns those values for the next iteration.

parsing link headers

If the API you're working with returns link headers, we recommend the parse-link-header package, which can be used to extract the next URL to use when paginating.