Persisting Data between Executions

Persisting data between runs

Sometimes it's helpful to save data from one execution of an integration so it can be used in a subsequent execution. Prismatic provides components that allow you to save some state from one run for use in a future run.

Why is this important or helpful? Imagine you have an integration that pulls down and processes data from a data source. Your integration recently processed a record with ID "123", and the next time your integration runs you want to ensure it processes ID "124" and above. You can persist "123" using a Save Value action, and then the next time your integration runs it can use Get Value to know that "123" was the most recently processed record. You can then build your integration to process newer records than the one that was saved.

Let's look at two components that take advantage of persisting data:

The persist data component

Data can be persisted between runs using the Persist Data component. Data are stored in key-value pairs, and values can be strings, numbers, objects, or lists. You can choose to persist data with the Flow - actions - that lets you persist data scoped to the current flow. You can also use the Cross Flow - actions to persist data that can be shared between flows of an instance. Or, you can use the Integration - actions to persist data between instances of the same integration (so, multiple customers can share a data store).

You can store a key/value pair using the Save Value action, or you can use Persist Data's other actions to append to a persisted list. If you would like to save a timestamp instead, you can use the Save Current Time action to save the current time into a key of your choosing.

Later, in a subsequent run, you can fetch the value you saved using the Get Value action. If a key is not set, Get Value will return null.

You can remove data from an array, or remove a key/value pair altogether, using Persist Data's other actions.