Skip to main content

Defining Integrations as Code

Overview#

There are many ways to create Prismatic integrations. You can assemble one through the web app, or by defining your integration as code in a YAML file and uploading it via the Prismatic CLI tool, Terraform Provider, or API.

We recommend that you assemble your first few integrations through the web app so you can acquaint yourself with triggers, action steps, inputs and outputs, config variables and credentials, but once you feel comfortable with these concepts we encourage you to explore creating integrations programmatically.

Anything you can add to an integration in the web app, you can add through YAML as well. Using YAML syntax, you can define an entire integration, including its trigger and action steps, and how data flows between the steps.

Advantages of Writing Integrations as Code#

If you are a developer, regardless of if you fall on the Vim, Emacs, or VSCode side of the editor war, you are likely more productive in a text editor than you are in a web browser. You can grep over a codebase and make changes to your integrations more quickly in an editor than you can in a browser. In addition to yielding greater productivity, writing integrations as code gives you:

  • A faster feedback loop. Make significant changes to multiple steps of an integration and test the integration from your terminal - no clicking required!
  • The ability to store integrations in your source repository. Your integration can go through whatever spec writing, development, code review, and deployment cycle you currently use. You can develop your integration in tandem with your core product and its APIs.
  • The option to update integrations as part of your CI/CD Pipeline. Is your integration dependent on updates to your APIs? You can plug your YAML-defined integrations into your CI/CD pipeline, so integrations are deployed alongside the rest of your code base.

Integration YAML File Anatomy#

A YAML-defined integration contains the following sections: definitionVersion, componentVersions, name, description, category, flows, endpointType, preprocessFlowName, externalCustomerIdField, flowNameField and requiredConfigVars. flows further contains name, description, isSynchronous, retryConfig and steps.

Integration YAML File Anatomy
---definitionVersion: Integer
componentVersions:  component-key-1: Integer  component-key-2: Integer
category: (optional) String
name: String
description: (optional) String
endpointType: (optional) [flow_specific, instance_specific, shared_instance]
preprocessFlowName: (optional) String
externalCustomerIdField: (optional) String
flowNameField: (optional) String
flows:  - name: String    description: String    isSynchronous: Boolean
    retryConfig: (optional)      delayMinutes: Integer      maxAttempts: [1,2,3,4,5]      uniqueRequestIdField:        type: reference        value: String      usesExponentialBackoff: Boolean
    steps:      - isTrigger: true # Start with a trigger        name: String        description: (optional) String        schedule: (optional)          type: [value, configVar]          value: Cron string or config variable name          meta: (optional)          scheduleType: [none, custom, minute, hour, day, week]          timeZone: String        action:          componentKey: String          key: String      - name: String        description: (optional) String        action:          key: String          componentKey: String        credential: (optional) String        inputs: (optional)          InputKeyOne:            type: [value, reference, configVar, template]            value: String          InputKeyTwo:            type: [value, reference, configVar, template]            value: String
requiredConfigVars: (optional)  - dataType: [string, date, timestamp, picklist, code, credential, boolean]    key: String    defaultValue: (optional) String    hasDivider: (optional) Boolean    header: (optional) String    credentialTypes: (optional - used when dataType is credential)      - [basic, apiKey, oauth2, oauth2ClientCredentials, apiKeySecret, privateKey]    orgOnly: Boolean
  • definitionVersion: YAML syntax changes over time. This declares which syntax this file uses (we currently use YAML syntax version 5).
  • componentVersions is a list of versions of components to use.
  • category is an optional string you can set to categorize the type of integration you developed. This helps when filtering lists of integrations.
  • name is a brief unique identifier, and the name of the integration. For example, "Queue 3D Print Job".
  • description is an optional longer explanation of what the integration does. Your team members in the web app can search for integrations by name or description. For example, "This integration converts a completed .rocket file to AutoCAD format and queues a 3D print job."
  • requiredConfigVars describes the config variables, their keys (names) and default values that your integration uses.
  • flows is an array of flows. Each flow has several properties:
    • name is the name of the flow
    • description is an option description for the flow
    • isSynchronous is a boolean (true/false) that determines if the integration should be run synchronously or asynchronously
    • retryConfig is a set of values that, when present, dictate how an integration should retry failed runs. See Integration Retry Configuration.
    • steps is an ordered list of actions your integration will execute. Each step takes a required name, an action to execute, and optional description and step inputs, described in more detail below. Your first step should always be a trigger, indicated by isTrigger: true.

Additionally, these fields describe how webhook URLs should be configured:

  • endpointType defines if flow-specific (default), instance-specific, or shared-instance webhook URLs should be created.
  • preprocessFlowName declares which flow, if any, should be run prior to dispatching an flow invocation to a specific customer/flow.
  • externalCustomerIdField determines where to look for a customer's external ID in a webhook request if endpointType is set to shared_instance.
  • flowNameField determines where to look for the flow's name to execute if endpointType is set to shared_instance or instance-specific.

Integration Triggers in YAML#

An integration trigger is the first step of an integration, and defines when an instance of an integration should run. The integration can be triggered by a webhook or on a schedule similar to a cron job, and you can select which component's trigger to use.

A standard action block for a trigger reads:

- action:    componentKey: salesforce    key: webhook  description: ""  inputs: {}  isTrigger: true  name: Integration Trigger

A trigger can take an optional description, which is a longer, more detailed description of how the trigger is invoked.

If a schedule is specified, the trigger will run on a schedule. If schedule is omitted, a webhook trigger will be created.

For More Information: Integration Triggers

Specifying a Scheduled Trigger in YAML#

You can run integrations on a schedule using cron notation. For example, if you want instances of an integration to run at 03:30 PM CST daily, you can configure a trigger to run at that time:

- action:    componentKey: schedule-triggers    key: schedule  description: ""  inputs: {}  isTrigger: true  name: Integration Trigger  schedule:    value: 30 15 */1 * *    type: value    meta:      scheduleType: day      timeZone: America/Chicago

Note that the "Daily at 3:30 PM" is translated into a cron string above. The meta block contains information for how the integration designer should present the UI, which is useful if you choose to explore the integration in the web app.

Like inputs, the schedule can be configured to reference a config var. This is helpful if different customers want to their instances run at different times or intervals:

requiredConfigVars:  - dataType: schedule    defaultValue: 30 15 */1 * *    hasDivider: false    key: My Cron Schedule    orgOnly: false    scheduleType: custom
steps:  - action:      componentKey: schedule-triggers      key: schedule    description: ""    inputs: {}    isTrigger: true    name: Integration Trigger    schedule:      meta:        scheduleType: custom        timeZone: ""      type: configVar      value: My Cron Schedule

For More Information: Schedule Triggers

Accessing Webhook Trigger Outputs in YAML#

If your trigger is invoked by a webhook, the body and header information of that trigger invocation is accessible using a reference input and referencing triggerName.results.{body,headers}.

Note that trigger names follow step naming conventions, and step names are converted to camelCase when referenced in YAML.

To see what a named trigger's referenceable name is in camel case, type it in here:

Suppose a webhook invocation contained a JSON payload body of {"customer":{"id":123}} and you have a custom component that you would like to pass the 123 value into as an input. You can access that portion of the output of the trigger as input like so:

steps:  - name: My Trigger    description: Kicks off the integration    action:      componentKey: webhook-triggers      key: webhook    inputs: {}    isTrigger: true
  - name: Do something with the customer ID    action:      key: my_custom_component      componentKey: myCustomAction    inputs:      myCustomerId:        type: reference        value: myTrigger.results.body.data.customer.id
note

If keys contain spaces or other special characters, you can reference those using ["bracket"] notation. For example,

value: myTrigger.results.body.data["My key!"]

For More Information: Webhook Triggers

Config Variables in YAML#

Config variables are defined in the requiredConfigVars portion of integration YAML. You can define config variables of type string, date, timestamp, picklist, code, credential or boolean.

You can add headers or dividers to affect how the UI looks when someone from your team deploys the integration to a customer. Here are a few example config variables of various types, a few of which have headers above them:

requiredConfigVars:  - header: Slack Info    hasDivider: true    key: Slack Webhook    dataType: string    defaultValue: "https://hooks.slack.com/services/a/b/c/"    orgOnly: false  - header: Fuel Info    key: Fuel Units    description: Units of measurement of rocket fuel    dataType: picklist    defaultValue: Gallons    pickList:      - Gallons      - Pounds      - Liters    orgOnly: false  - key: Convert Units    defaultValue: "true"    dataType: boolean    description: Convert units before processing    orgOnly: false  - key: Rocket Fuel API Key    dataType: credential    credentialTypes:      - basic      - apiKey    orgOnly: false

Variables with orgOnly set to true are only configurable by your customer-facing team members (and not by your customers).

Steps can reference these config variables as inputs by referencing their keys (names) - like Slack Webhook or Convert Units.

For More Information: Config Variables in Integrations

Integration Action Steps in YAML#

Integrations are comprised of multiple steps. A step requires a name (referenceable name), and an action to execute. Optionally, a step can take a description, and can accept inputs that might be strings, config variables, references to outputs from a previous step, or templated expressions concatenating strings, step results and config variables together.

If a step requires a credential, it can reference a credential config variable by key (name).

The step's action takes a key of an action, and the action's componentKey. The componentKey is required, since action keys are not necessarily unique (Amazon S3 and Azure file might both have a gitObject action).

Action keys and component keys can be found in our component docs or by using the Prismatic CLI components:actions:list subcommand.

For example, if you want to add a step that lists all objects of an S3 bucket, you can run this to get the action's key and componentKey:

prism components:actions:list --columns key,componentKey,label,description --filter 'componentkey=^aws-s3$'
 Key          Componentkey Label         Description ──────────── ──────────── ───────────── ───────────────────────────────────────────────── copyObject   aws-s3       Copy Object   Copy an object in S3 from one location to another deleteObject aws-s3       Delete Object Delete an Object within an S3 Bucket getObject    aws-s3       Get Object    Get the contents of an object listObjects  aws-s3       List Objects  List Objects in a Bucket putObject    aws-s3       Put Object    Write an object to S3

You can then use those keys in a step:

steps:  - name: Get rocket schematic files    description: Get a list of all rocket schematics in our S3 bucket    credential: My AWS Credentials Config Variable    action:      key: listObjects      componentKey: aws-s3    inputs:      awsRegion:        type: value        value: us-east-1      bucket:        type: configVar        value: My Bucket Name

Referencing Step Outputs as Inputs in YAML#

A step can reference another step's output as its own input. For example, you may want to download a file from Dropbox and upload it to S3. Your steps, then, would look like this:

steps:  - name: Download from Dropbox - A PNG    description: Download a .png file from DropBox    credential: My Dropbox Credentials    action:      key: downloadFile      componentKey: dropbox    inputs:      path:        type: value        value: /path/to/my/file.png  - name: Upload the file to S3    description: Upload the downloaded PNG file to Amazon S3    credential: My AWS Credentials    action:      key: putObject      componentKey: aws-s3    inputs:      awsRegion:        type: configVar        value: My AWS Region      bucket:        type: configVar        value: My Bucket Name      fileContents:        type: reference        value: downloadFromDropboxAPng.results      objectKey:        type: value        value: path/to/output.png

Note the fileContents reference above. The step "Download from Dropbox - A PNG" was referenced by the value downloadFromDropboxAPng. The step name was converted to camelCase so that sub properties of the output could be referenced if need be.

To convert a step name to a referenceable name for the code component or YAML, remove all special characters and camelCase the remaining string. You can test step name -> reference name conversions here:

Drilling Down Into Trigger and Step Outputs in YAML#

It's useful to be able to pass in a portion of a step result rather than the whole thing into an input of a subsequent step.

For example, suppose you've pulled down a list of TODO tasks with an HTTP - GET action named Pull down some JSON, and for whatever reason you need the third item's title for the input of a step.

You can access that value for an input in one of two ways:

inputs:  someInput:    type: reference    value: pullDownSomeJson.results[2].title

or

inputs:  someInput:    type: reference    value: pullDownSomeJson["results"][2]["title"]

Note that the dot .key syntax works well when keys have no spaces or special characters. If your keys have spaces or special characters, you can use bracket ["key"] syntax to drill down into properties of previous steps' results.

Using Template Expressions as Inputs#

You can concatenate multiple config variables, step results and static strings into an input using a template expression.

To do that, create an input of type template and follow the expression syntax detailed here:

- description: ""  action:    componentKey: http    key: httpGet  name: GET Request  inputs:    url:      type: template      value: "{{#API Endpoint}}/product?id={{$getProductId.results}}"

Loops in YAML#

As noted on the integrations article, you can loop over steps within an integration. This is useful if you want to perform a series of actions on an array of things (an array of files, an array of users, etc).

In YAML, a loop is a component like any other, and the loop component itself takes a series of sub-steps.

For example, this YAML definition loops over users "Bob", "Sally", and "Susan":

steps:  - name: List Users    action:      componentKey: code      key: runCode    inputs:      code:        type: value        value: |-          module.exports = async (context, stepResults) => {            return { data: ["Bob","Sally","Susan"] };          };  - name: Loop Over Users    action:      componentKey: loop      key: loopOverItems    inputs:      items:        type: reference        value: listUsers.results    steps:      - name: Print a greeting        action:          componentKey: code          key: runCode        inputs:          code:            type: value            value: |-              module.exports = async (context, stepResults) => {                return { data: `Hey ${stepResults.loopOverUsers.currentItem}` };              };

Each iteration of the code component within the loop returns "Hey ${NAME}", and the result of the loop component will read:

["Hey, Bob", "Hey, Sally", "Hey, Susan"]

For More Information: Loops in Integrations, Looping Over Files Quickstart

Branching in YAML#

An integration can contain logical branches, which are paths the integration can take based on some config variables or step results. More information about branching is on the integrations article.

The branch component contains two actions: branchOnExpression and branchOnValue.

Branching On Value in YAML#

In this example, we use the branchOnValue action to follow one of three logical branches based on the value of a config variable:

- name: Branch on Value  action:    componentKey: branch    key: branchOnValue  inputs:    inputValue:      type: configVar      value: notificationMethod    branchValueMappings:      type: complex      value:        - name: Send SMS          type: value          value: sms        - name: Send Email          type: value          value: email  branches:    - name: Send SMS      steps: []    - name: Send Email      steps: []    - name: Else      steps: []

In this example the inputValue pulled in is a config variable named notificationMethod. That inputValue is then checked against two values, sms and email, and if inputValue is equal to sms, the integration follows the Send SMS branch (same for the Send Email branch).

If inputValue is not equal to sms or email, the integration follows the Else branch.

In this example the three branches do not contain any steps. You can add steps to each branch like you would add steps normally: create a list of steps under the steps: section in each branch.

Branching On Expression in YAML#

In this example, assume that another step named Get User Data has pulled down data that looks like this:

{  "username": "bob",  "age": 40,  "address": {    "street": "123 Fake St",    "state": "SD",    "city": "Sioux Falls"  }}

If we pull down data about a user from South Dakota who is over 20 years old, we want our integration to branch one way. Otherwise branch the other way.

We can set up our branch on expression action like this:

- name: Branch on User Data  action:    componentKey: branch    key: branchOnExpression  inputs:    conditions:      type: complex      value:        - name: Interesting User          type: complex          value:            - and            - - greaterThan              - type: reference                value: getUserInfo.results.age              - type: value                value: "20"            - - equal              - type: reference                value: getUserInfo.results.address.state              - type: value                value: SD  branches:    - name: Interesting User      steps:        - action:            componentKey: slack            key: postSlackMessage          inputs:            message:              type: reference              value: getUserInfo.results.username            webhookUrl:              type: value              value: https://hooks.slack.com/services/FOO/BAR/BAZ          name: Slack Message    - name: Else      steps:        - action:            componentKey: stop-execution            key: stopExecution          name: Stop Execution

Here, we used a complex type to do an expression comparison. We use the and operator to check that both the next expression and the expression after it are true. The next two expressions use greaterThan and equal operators to compare the values that they're presented.

The operators that are available are:

OperatorDescriptionArguments
andReturn true if all of the arguments return trueA series of conditionals, that may themselves be and, or, greaterThan, equal, etc.
orReturn true if any of the arguments return trueA series of conditionals, that may themselves be and, or, greaterThan, equal, etc.
equalReturn true if argument one is equal to argument twoTwo values which are static values, config variables, or references to step outputs
notEqualReturn true if argument one is not equal to argument twoTwo values which are static values, config variables, or references to step outputs
greaterThanReturn true if argument one is strictly greater than argument twoTwo values which are static values, config variables, or references to step outputs
greaterThanOrEqualReturn true if argument one is greater than or equal to argument twoTwo values which are static values, config variables, or references to step outputs
lessThanReturn true if argument one is strictly less than argument twoTwo values which are static values, config variables, or references to step outputs
lessThanOrEqualReturn true if argument one is less than or equal to argument twoTwo values which are static values, config variables, or references to step outputs
inReturn true if argument two contains argument oneTwo values which are static values, config variables, or references to step outputs. Value two must be an array.
notInReturn true if argument two does not contain argument oneTwo values which are static values, config variables, or references to step outputs. Value two must be an array.

Component Versions in YAML#

Components are versioned, and the components that you add to an integration are pinned to specific versions. To set and adjust the versions of components you use, edit the componentVersions section of your integration's YAML file, adding key-value pairs containing a component key and version number. A sample componentVersions block might look like this:

componentVersions:  branch: 20  change-data-format: 17  code: 30  loop: 14  slack: 24  text-manipulation: 16

Multi-line Values in YAML#

YAML syntax supports multiline strings, so Prismatic does, too. Multiline strings are especially helpful if you use a code component. Your code component code can be written entirely in-line:

steps:  - name: Return Hello World    description: Return the string "Hello World"    action:      key: runCode      componentKey: code    inputs:      code:        type: value        value: |          module.exports = async (context, stepResults) => {            return {              data: "Hello, World",            };          };

Importing a YAML-Defined Integration#

To import a YAML-defined integration, use the Prismatic CLI integrations:import subcommand:

prism integrations:import --path path/to/your-integration.yaml

Exporting an Integration as YAML#

An existing integration that was built via YAML or in the integration designer can be exported to a YAML file using the Prismatic CLI integrations:export subcommand:

prism integrations:export EXAMPLEyYXRpb246YzVhZWI2OWMtZWJlNy00YWQ4LTg3YjktMzhkNWJlNGMwMTIx > my-integration.yaml

Example Integration in YAML#

The integration from the getting started tutorial pulled down "TODO" items from a list, and posted a message to Slack. That entire integration can be defined in a short YAML file:

---definitionVersion: 5
name: My First Integration
description: My first foray into Prismatic
category: Prismatic Tutorials
requiredConfigVars:  - dataType: string    defaultValue: https://hooks.slack.com/services/REPLACE/THIS/URL    hasDivider: false    key: Slack Webhook URL    orgOnly: false  - dataType: string    defaultValue: "https://jsonplaceholder.typicode.com/todos"    hasDivider: false    key: Todo API URL    orgOnly: false
componentVersions:  code: 48  http: 53  slack: 40
flows:  - name: Flow 1    description: ""    isSynchronous: false
    steps:      - name: Integration Trigger        action:          componentKey: schedule-triggers          key: schedule        inputs: {}        isTrigger: true        schedule:          meta:            scheduleType: day            timeZone: America/Chicago          type: value          value: 30 15 */1 * *      - name: Get TODO List        action:          componentKey: http          key: httpGet        inputs:          responseType:            type: value            value: json          url:            type: configVar            value: Todo API URL      - name: Create Slack Message        action:          componentKey: code          key: runCode        inputs:          code:            type: value            value: |              module.exports = async (context, stepResults) => {                var todoItems = stepResults.getTodoList.results;                var totalItemCount = todoItems.length;                var completedItemCount = todoItems.filter((x) => x.completed).length;                return {                  data: `${completedItemCount} of your ${totalItemCount} tasks are marked complete.`,                };              };      - name: Send Slack Message        action:          componentKey: slack          key: postSlackMessage        inputs:          message:            type: reference            value: createSlackMessage.results          webhookUrl:            type: configVar            value: Slack Webhook URL

Opening up the web app, we see that this YAML-defined integration created the appropriate steps, config variables and inputs.

Further Reading#

After you feel comfortable defining basic integrations with YAML, we recommend you check out our Writing an Integration in YAML with Trigger Payloads, and Inputs quickstart, where we examine how to handle webhook trigger payloads, code components, and knitting inputs and outputs throughout a YAML-defined integration.