Recursive Flow Component
This component allows a flow to call itself with a cursor in order to process large amounts of data in serial.
Component key: recursive-flow
Description
The recursive flow component allows a flow to recursively call itself. It is particularly useful if you have a long-running workflow that needs to process logs of records.
A cursor is passed between executions to track what data has been processed, and what data is left to process.
For example, suppose you have 100,000 records in a third-party app to process. You know from experience that you can fetch 100 records at a time, and that each page of records takes about 5 seconds to fetch and process. Processing all of the data would take 5 seconds x 1000 pages (or about 83 minutes).
An execution can run for a maximum of 15 minutes, so you won't be able to process all pages in a single execution. Instead, you can process a hand-full of pages in one execution, and then invoke another execution with a cursor indicating where you left off.
In the 100,000 records / 5 seconds per page example, you could build your to flow process, say, 10 pages per execution.
After 10 pages (or about 50 seconds; well under the 15-minute limit), it could call itself with a cursor of 11.
The next execution would process pages 11 through 20, and send itself a cursor of 21.
This could continue until you detect that there are no more records to process.
A general flow that processes large sets of records from a paginated API in serial using the recursive flow trigger would look something like this:
Triggers
Recursive Trigger
Accept a request from an 'Invoke Recursive Trigger' step | key: recursiveTrigger
| Input | Default | Notes | Example |
|---|---|---|---|
Default Cursor Value string / Required defaultCursor | The default value to return as a cursor if no cursor was supplied in a trigger payload. For example, you could set this to '1970-01-01T00:00:00Z' (Unix Epoch) if your cursor is a creation timestamp, or '0' if your cursor is a page number. | 1970-01-01T00:00:00Z | |
Run on Deploy? boolean / Required runOnDeploy | false | Run this flow automatically when an instance is deployed. |
When invoked manually or via webhook without a cursor value, the recursive trigger begins an execution using the Default Cursor input as its current cursor value.
If the API you're integrating with uses page numbers, you can set the default cursor value to 0.
If you're using timestamps, Unix Epoch (1970-01-01T00:00:00Z) could work for an initial cursor.
When invoked by an Invoke Recursive Trigger action, a cursor is passed to this trigger via POST request and is used for the current execution.
Actions
Get Recursive Cursor
Get the current value of the recursive trigger cursor | key: getCursor
Invoke Recursive Trigger
Invoke the current flow with the current cursor | key: invokeRecursive
Set Recursive Cursor
Set the value of the recursive cursor for the next loop iteration or recursive trigger invocation | key: setCursor
| Input | Notes |
|---|---|
Cursor string / Required cursor | The programmatic cursor to set for the next loop iteration or recursive trigger invocation. |