Building Integrations

The Integration Designer#

After creating a new integration or selecting one from your list of integrations, you will find yourself in the integration designer. Here, you can build, test, and publish integrations.

The integration designer contains four important features:

  1. The configuration drawer lets you configure steps of your integration, set up config variables that are used by your integration, change runtime configuration, etc.
  2. The testing drawer lets you run tests of your integration, supply your integration with sample payloads, and see results of your integration test runs.
  3. The publication pane lets you publish a version of your integration, so it can be deployed to customers.
  4. The majority of the page is taken up by the integration editor pane. Here, you can add steps to your integration, create branches and loops, and generally arrange the flow of your integration.
Create your first few integrations using the integration designer

We recommend that you create your first few integrations through the integration designer to get a feel for how steps interconnect and data flow through an integration. If you prefer the integration designer in the web app, we encourage you to continue to use it. If you are a developer, you have the option to write your integration in YAML, which gives you the option to store your integrations in a source control system of your choosing.

Integration Steps#

Actions, like downloading a file from an SFTP server or posting a message to Slack, are added as steps of an integration. Steps are executed in order, and outputs from one step can be used as inputs for subsequent steps.

Steps are run in order from top to bottom, and you can add conditional logic to your integration with a branch, or run a series of steps on a data set in a loop. If one step throws an error, the integration stops running.

The Trigger Step#

The first step of your integration is the trigger step, which determines when instances of your integration will run. The integration triggers article details how triggers work, and how to invoke your integration.

Adding Steps to Integrations#

To add a step to an integration, click the

icon underneath the trigger or another action.

Select the action you would like to add to your integration. You can begin to type the name of the action you would like to add to filter the list of actions available.

Step Outputs#

When a step runs, it optionally outputs data that can be used as input for subsequent steps. For example, an SFTP List Files step will output an array of file names:

An SFTP Get File step will output the contents of a file that it fetched from an SFTP server (in this case, an image):

Outputs take one of three forms:

  • A primitive value, like a string, boolean value, number, or array of primitives. A subsequent step that references this output will receive the string, boolean, number, or array as input.

  • An object. An output might include multiple key-value pairs:

    { "key1": "value1", "key2": ["value2.0", "value2.1", "value2.2"] }

    You might see this after pulling JSON from an HTTP endpoint. Specific values from an object can be referenced as inputs using familiar JavaScript .dot and ["bracket"] notation. Using the above as an example, if you wanted to access value2.1 you could reference results.key2[1].

  • A binary file. Binary file outputs contain a combination of a file Buffer and in indicator of content type (like "image/png") in the form:

    {
    "data": Buffer,
    "contentType": String
    }

For More Information: Custom Component Outputs

Configuring Step Inputs#

Once you have added a step to your integration, you will likely need to configure some inputs for that step. Inputs might include a RESTful URL endpoint, an S3 bucket name, a Slack webhook to invoke, or even a binary file like an image or PDF to upload or process.

Some inputs are required, and denoted with a * symbol, while others are optional.

Inputs can take one of three forms:

  • A value is a simple string (perhaps a URL for an HTTP request).

  • A reference is a reference to the output of a previous step or trigger. For example, if a previous step pulls down a file from Amazon S3 and the step is named Fetch my file, then you can reference Fetch my file as in input for another step, and that subsequent step will be passed the file that Fetch my file returned.

    Outputs from one step can be referenced by a subsequent step by referencing the previous step's results field. So, if a previous step returned an object - for example, if an HTTP - GET action pulled down some JSON reading { "firstkey": "firstvalue", "secondkey": "secondvalue" } - you access that secondvalue property in a subsequent step's input by referencing that HTTP - GET step and choosing results.secondkey in your Reference search.

  • A config variable references one of the integration's config variables. For example, we can select a config variable, CMS API Endpoint, as an input for one of our steps. Config variables can be distinct for each customer, so each customer can be configured with a different CMS API Endpoint.

To enter values for the input of a step, first select if you want the input to be a value, reference, or config variable. Then, enter a value for the value, name of a previous step and search for the reference, or name of a config variable for the config variable.

For More Information: Custom Component Inputs

Changing Step Names#

By default steps are uniquely named after the action they invoke (so, they're named things like CSV to YAML, or Delete Object). To override that default name, click the step and click the default name to edit its name.

Like using descriptive variable names in a program, renaming steps allows you to give your steps descriptive names. For example, rather than a step being called Delete Object you can name it Delete Schematics File from Share. We recommend giving your steps descriptive names so your team members can read through integrations and understand their purpose more readily.

Note that once you change a step's name, you will likely need to update any reference to that step's outputs in code components. For information about referencing previous steps' outputs within a code component, click here

Reordering Steps#

Steps are executed in serial. To reorder steps, click and drag a step up or down.

Choosing Component Versions#

Components are versioned. You can choose a version of each component (custom or built-in) that works for your integration. "Pinning" component versions for your integration prevents accidental regressions if a new version of a component is published that contains breaking changes.

To choose what version of each component your integration uses, click the Component Versions button on the right-hand side of the integration designer.

You can choose to run the latest version of a component, or any previous version. Components running the latest available version will be marked in grey, while components running a version for which there is a newer version available will be marked in yellow.

To change the version of a component your integration uses, click the CHANGE VERSION button to the right of the component you want to change, and select a version from the dropdown.

For More Information: Component Versioning

Config Variables in Integrations#

As an integration builder you can use config variables to drive the logic of your integration.

Click the Config Variables button to open the Config Variables drawer. From there, you can define the configuration experience that will be used by your customer-facing teams. You can define names, descriptions, variable types, and optional default values of config variables that you will reference from your integration. You can use headers to organize your config variables, and descriptions to provide helpful hits to your customer-facing teams who will deploy your integrations.

When it comes time for your customer-facing teams to deploy your integration, they can enter or select configuration options and tailor the integration for a particular customer without the involvement of integration builders:

Config variables that you define in the config variable drawer can be used within your integration as inputs or steps, or through the Branch component to drive branching logic.

Config Variable Types#

There are several types of configuration variables:

  • String is a standard string of characters
  • Date follows the form mm/dd/yyyy
  • Timestamp follows the form mm/dd/yyy, HH:MM [AM/PM]
  • Picklist allows you to define a series of options that your integration deployment team members can choose from.
  • Code lets your integration deployment team to enter JSON, XML, or other formatted code blocks. This is handy if customers have unique formats for recurring reports, or other formatted documents that differ from customer to customer.
  • Credential is an API key, username/password, etc., that can be bound to one or more steps of your integration. Credential config variables authorization types (API key, basic user/pass, etc.) need to match a step's required authorization type - for example, and Amazon S3 component requires a credential config variable of type "API Key / Secret". Click the pencil icon to edit a credential config variable's authorization type. See credentials.
  • Boolean allows your integration deployment team to choose either true or false.

Once config variables are enumerated in this list, they are available as inputs to actions within your integration.

For More Information: Credentials

Synchronous / Asynchronous#

Integrations can be triggered synchronously or asynchronously. For more information, see the integration triggers article.

Integration Retry Configuration#

Professional or Enterprise Plan Required

To configure integration retry, you will need to have a professional or enterprise plan. Plan details are available on our pricing page.

You can configure your asynchronously-invoked instances to retry if they fail to run to completion. This is handy if your integration relies on a flaky third-party API - the third party API might be down briefly, but back up a few minutes later. You can configure your integration to try again in a few minutes when the API is back up, without needing to trigger unnecessary alert monitors for your team.

How to Configure an Integration to Retry

To configure an integration to retry its execution, click the Execution Configuration button on the right side of the integration designer and toggle the Retry option on.

Max Attempts indicates the maximum number of times (up to 5) that Prismatic will run the same instance invocation in the event of failure. If an instance has failed more than Max Attempts number of times, the run will be marked as execution failed and relevant alert monitors, if configured, will fire.

Minutes Between Attempts indicates the number of minutes that Prismatic should wait before trying to run an instance again. If, for example, you specify 4 minutes, and the first instance invocation failed at 10:24, then the instance will attempt to run again at 10:28, 10:32, 10:36, 10:40 and 10:44 if it repeatedly fails to complete.

Note: Due to the nature of scheduled AWS Lambda functions, scheduled retries are precise to the minute (as opposed to to the second). So, an attempt that fails at 10:24 with a Minutes Between Attempts set to 4 might trigger next at 10:28 or 10:29.

If you have Exponential Backoff selected, your instance will wait a longer and longer time between attempted runs, using an exponential factor of 2. For example, if your Minutes Between Attempts is set to 3 and Exponential Backoff is set, then retry attempts will fire after 3, 6, 12, 24, and 48 minutes.

Note: The maximum amount of time an instance will take before retrying is 24 hours. If backoff is computed to wait more than 24 hours, it'll fire after waiting 24 hours instead.

Finally, Retry Cancelation gives you the ability to cancel a set of retry attempts if a more recent invocation of the instance has occurred. For example, if your integration takes a payload as part of its invocation and then updates some third party system with that payload data, you may want to cancel older invocations if newer data comes in. Otherwise, you may end up updating some third party system with newer data, and then the retry would overwrite the newer data with older data afterwards.

To configure retry cancelation, select a unique request ID from the trigger payload. For example, you might pass in a header, x-my-unique-id: abc123 as part of your trigger payload. If another invocation with that header comes in that updates resource abc123, you might want to cancel currently queued retries. To do that, select your trigger's results.headers.results.headers.x-my-unique-id reference as your Unique Cancelation ID.

Cancelation IDs do not need to be headers. Instead, you can select a key from the payload body. For example, if your instance invocation looks like this:

curl -L -d '{"productId":"abc123","price":"250","description":"A box of widgets"}' \
--header "Content-Type: application/json" \
'https://hooks.prismatic.io/trigger/SW5zdGFuY2U6ODU4MDdiNDEtYjQ5Zi00NDIyLTgwNTctNjc0ZmI5MmQ4N2Ey'

You can key your unique cancelation ID off of results.body.data.productId.

For More Information: Instance Retry and Replay

Looping#

How to Loop Over Files in an Integration

For many integrations it's handy to be able to loop over an array of items. If your integration processes files on an SFTP server, for example, you might want to loop over an array of files on the server. If your integration sends alerts to users, you might want to loop over an array of users.

Prismatic provides the loop component to allow you to loop over an array of items. After adding a loop step to your integration, you can then add steps within the loop that will execute over and over again.

The loop components takes one input: a items. Items is an array - an array of numbers, strings, objects, etc. For example, one step might generate an array of files that your integration needs to process. Its output might look like this:

[
"path/to/file1.txt",
"path/to/file2.txt",
"path/to/file3.txt",
"path/to/file4.txt"
]

The loop component can then be configured to loop over those files by referencing the results of the list files step:

Subsequent steps can reference the loop step's currentItem and index parameters to get values like path/to/file3.txt and 2 respectively:

For More Information: The Loop Component, Loops in YAML, Looping Over Files Quickstart

Looping Over Lists of Objects#

The list of objects passed into a loop component can be as simple or complex as you like.

In this example, if we have a loop named Loop Over Users, and the loop was presented items in the form:

[
{
"name": "Bob Smith",
"email": "bob.smith@progix.io"
},
{
"name": "Sally Smith",
"email": "sally.smith@progix.io"
}
]

Then the loop will iterate twice - once for each object in the list, and we can write a code component that accesses the loop's currentItem and index values and sub-properties of currentItem like this:

module.exports = async (
{ logger },
{ loopOverUsers: { currentItem, index } }
) => {
logger.info(`User #${index + 1}: ${currentItem.name} - ${currentItem.email}`);
};

That will log lines like User #1: Bob Smith - bob.smith@prismatic.io.

Return Values of Loops#

A loop will collect the results of the last step within the loop, and will save those results as an array. For example, if the loop is presented the list of JSON-formatted user objects above, and the last step in the loop is a code component reading:

module.exports = async(context, loopOverUsers: { currentItem }) => {
return {data: `Processed ${currentItem.email}`}
}

Then the result of the loop will yield:

["Processed bob.smith@progix.io", "Processed sally.smith@progix.io"]

Branching#

The branch component allows you to add branching logic to your integration. Think of branches as logical paths that your integration can take. Given some information about config variables or step results, your integration can follow one of many paths.

Branch actions are handy when you need to conditionally execute some steps. Here are a couple of examples of things you can accomplish with branching:

Example 1: If each of your customers have a different preferred file store, your integration can branch into "save to Amazon S3", "save to Azure Files" or "save to DropBox" branches depending on a customer-specific config variable value.

Example 2: Your customers want to be alerted when their rocket fuel level is below a certain threshold. You can branch into "send an alert" and "fuel level is okay" branches depending on results of a "check rocket fuel level" step.

Example 3: You want to upsert data into system that doesn't support upserting. You can check if a record exists, and branch into "add a new record" or "update the existing record" branches depending on if the record exists.

For More Information: The Branch Component, Branching in YAML

Branching on a Value#

Adding a Branch on Value action to your integration allows you to create a set of branches based on the value of some particular variable. It's very similar to the switch/case construct present in many programming languages.

How to Branch on a Single Value in an Integration

Consider example 1 above. Suppose your customers each have a config variable named fileStore that might be equal to aws, azure or dropbox, depending on which file storage service they like to use. You can use the config variable fileStore as the input value, and create three distinct branches for AWS, Azure, and Dropbox. From there, you can add the appropriate actions for saving a file to Amazon S3, Azure Files, and DropBox respectively:

For More Information: Branch on Value Action

Branching on an Expression#

The Branch on an Expression allows you to create branches within your integration based on more complex inputs. You can compare values, like config variables, step results, or static values, and follow a branch based on the results of the comparisons.

How to Branch on an Expression in an Integration

Consider example 2 above. You have a step that checks rocket fuel level for a customer, and you want to alert users in different ways if their fuel levels are low. You can express this problem with some pseudocode:

if fuelLevel < 50:
sendAnSMS()
else if fuelLevel < 100:
sendAnEmail()
else:
doNothing()

To express this pseudocode in an integration, add a step that looks up rocket fuel level. Then, add a Branch on an Expression action to your integration.

Create one branch named Fuel Critical, and under Condition Inputs check that results of the fuel level check step is less than 50. Then, create another branch named Fuel warning and check that results of the fuel level check step is less than 100.

This will generate a branching step that will execute the branch Send Alert SMS if fuel levels are less than 50, Send Warning Email if fuel levels are less than 100, or will follow the Else branch if fuel levels are 100 or above.

You can compare config variables, results from previous steps, or static values to one another using the following comparison functions:

  • equals: check that the two fields are equal to one another.
  • does not equal: check that the two fields are not equal to one another.
  • is greater than: check that the left field is greater than the right field.
  • is greater than or equal to: same as is greater than, but the values can also be equivalent.
  • is less than: check that the left field is less than the right field.
  • is less than or equal to: same as is less than, but the values can also be equivalent.
  • contained in: check that the right field contains the left field. The right field must be an array. For example, you can check if "b" is contained in ["a","b","c"].
  • not contained in: check that the right field (an array) does not contain the left field.

Multiple expressions can be grouped together with And or Or clauses, which execute like programming and and or clauses:

if ((foo > 500 and bar <= 20) or ("b" in ["a","b","c"]))

For More Information: Branch on Expression Action

Converging Branches#

Regardless of which branch is followed, branches always converge to a single point. Once a branch has executed, the integration will continue with the next step listed below the branch convergence.

This presents a problem: how do steps below the convergence reference steps in branches that may or may not have executed (depending on which branch was followed)? In your integration you may want to say "if branch foo was executed, get the results from step A, and if branch bar was executed, get the results instead from step B." Prismatic provides the Select Executed Step Result to handle that scenario.

Imagine that you have two branches - one for incoming invoices, and one for outgoing invoices, with different logic contained in each. Regardless of which branch was executed, you'd like to insert the resulting data into an ERP. You can leverage the Select Executed Step Result action to say "get me the incoming or outgoing invoice - whichever one was executed."

This action iterates over the list of step results that you specify, and returns the first one that has a non-null value (which indicates that it ran).

Within the component configuration drawer, select the step(s) whose results you would like, and the Select Executed Step Result step will yield the result of whichever one was executed.

Persisting Data Between Runs#

Sometimes it's helpful to save data from one execution of an integration so it can be used in a subsequent execution. Prismatic provides components that allow you to save some state from one run for use in a future run.

Why is this important or helpful? Imagine you have an integration that pulls down and processes data from a data source. Your integration recently processed a record with ID "123", and the next time your integration runs you want to ensure it processes ID "124" and above. You can persist "123" using a Save Value action, and then the next time your integration runs it can use Get Value to know that "123" was the most recently processed record. You can then build your integration to process newer records than the one that was saved.

Let's look at two components that take advantage of persisting data:

The Persist Data Component#

Data can be persisted between runs using the Persist Data component. Data are stored in key-value pairs, and values can be strings, numbers, objects, or lists.

You can store a key/value pair using the Save Value action, or you can use Persist Data's other actions to append to a persisted list. If you would like to save a timestamp instead, you can use the Save Current Time action to save the current time into a key of your choosing.

Later, in a subsequent run, you can fetch the value you saved using the Get Value action. If a key is not set, Get Value will return null.

You can remove data from an array, or remove a key/value pair altogether, using Persist Data's other actions.

The Process Data Component#

For some integrations, it's handy to know what data from an array you've already processed, and what data you haven't. For example, your integration might pull down an array of orders from a data store to be converted into invoices. Your order array might look like this:

[
{
"orderid": 123,
"items": {
"widgets": 5,
"whatsits": 7
}
},
{
"orderid": 122,
"items": {
"whoseits": 2
}
}
]

The Process Data's DeDuplicate action allows you to pass in such an array objects in descending order, along with a unique identifier (like orderid). The action uses data persistence between runs to track the most recently processed item (in this example, "orderid": 123). The next time this integration is run and an array of orders are passed into the DeDuplicate step, the step will return all objects in the array that appear before the object with "orderid": 123. So, if there's an order ID 124, 125, etc., it'll return those. That way, the subsequent execution will ignore order ID 123 and before, and instead process only more recent orders.

For More Information: Persisting Data in Custom Components