Using Shared Webhooks and Preprocess Flows

In this tutorial we'll cover how to create a single webhook endpoint that can be invoked by multiple customers.

For our scenario, imagine we want to integrate with a third-party ERP "Acme ERP". The Acme ERP tracks things like inventory and orders. Configuration of the ERP is limited, and only allows you to specify a single webhook URL for all of your customers to post inventory and order updates.

We need to create a single webhook URL that can accept the webhook payload that Acme ERP sends. That endpoint will need to inspect the payload, figure out what customer it pertains to and what sort of payload it is (an "inventory update" payload, or an "order creation" payload), and then it'll need to route that information to the correct flow in an instance deployed to the correct customer in Prismatic.

Our Acme ERP Integration

The Acme ERP integration that we deploy to customers has two flows. The first flow takes a collection of inventory updates ("remove 5 bowler hats", "add 20 pencils", etc.), and ensures that those updates are reflected in Progix:

The second flow takes an order created in Acme ERP, processes the data in the order, and submits an order request to Progix:

All of our customers are going to use the same webhook endpoint, so we'll need to configure our integration to create just one URL for all instances of this integration that are deployed. To do that, let's open up the Endpoint Configuration drawer using the button on the right-hand side of the integration designer. We'll toggle Endpoint Type to Shared.

We'll leave Preprocess Flow, Flow Name and External Customer Id alone for now, and come back to them in a moment.

Read more about webhook endpoint configuration on the Integration Triggers article.

The webhook payloads

When a customer in Acme ERP updates their inventory, Acme ERP sends a payload in this format to the shared webhook:

{
  "customerId": "A06DFFAC",
  "type": "inventory_update",
  "data": [
    {
      "txid": "E865298B-A3EC-4D17-B410-5FDFC8861BA7",
      "item": "bowler hat",
      "quantity": 5,
      "state": "removed"
    },
    {
      "txid": "4A38DE96-EA6B-4AB9-B9E2-3B033CA997CC",
      "item": "pencils",
      "quantity": 20,
      "state": "added"
    }
  ]
}

When a customer in Acme ERP creates a new order, Acme ERP sends a payload in this format to the shared webhook:

{
  "customerId": "C3E72B0C",
  "type": "create_order",
  "data": {
    "orderid": "75F5AEC5-D482-4386-8878-219F92185DEC",
    "date": "2021-09-08",
    "shipped": false,
    "addr_1": "177A Bleecker St.",
    "addr_2": "New York, NY 10012",
    "total": 122.57,
    "paid": true
  }
}

These two webhook payloads share some similarities: they both contain a customerId for your customers in Acme ERP, and they both have a type indicating the type of the webhook payload. We're going to use those two values to determine which customer in Prismatic to send the webhook to, and which flow to invoke for that customer.

To determine which customer to dispatch the webhook request to, we'll need a way to map an Acme ERP customerId to the external ID that we use in Prismatic. We'll also need need to map type to a flow name. To do those things, we'll create another flow that will execute when a webhook is first received, but before it's been dispatched to a customer's instance.

The preprocess flow

A "preprocess" flow allows us to process data that comes in to a shared webhook URL, and dispatch the data to a specific customer's instance and flow accordingly. The preprocess is called synchronously, and returns whatever the last step of the flow returns. Our preprocess flow will contain two steps: one step will look up the customer's Progix external ID given the Acme ERP customer ID, and the second step will map type to a flow name, and return both customer ID and flow name:

The customerId from the Acme ERP webhook request (accessible from integrationTrigger.results.body.data.customerId) can be passed in to a Progix external ID lookup action. The code component, then, can grab the external ID that is returned and can map type to a flow name:

const flowNamesMap = {
  inventory_update: "Update Inventory",
  create_order: "Create Order",
};

module.exports = async ({ logger, configVars }, stepResults) => {
  const customerExternalId = stepResults.getCustomerByAcmeId.results.id;
  const webhookType = stepResults.trigger.results.body.data.type;
  const flowName = flowNamesMap[webhookType];
  return {
    data: { customerExternalId, flowName },
  };
};

If we run our flow with a sample payload from above using the Run button, we can see that our last step returns an customerExternalId and flowName.

With a working flow, now we just need to indicate in Prismatic that this flow is a preprocess flow. To do that, let's open up the Endpoint Configuration once again, and select our "Preprocess" flow as the Preprocess Flow (you can name your preprocess flow whatever you want).

Read more about preprocess flows on the Integration Triggers article.

Routing the webhook to the correct customer and flow

Now that our preprocess flow has yielded a customer's customerExternalId and flowName, we now need to configure our integration to route the webhook invocation to the proper customer and flow. All we need to do is instruct our integration where to look to get the customerExternalId and flowName to key off of.

Let's open up the Endpoint Configuration drawer once more. This time, we'll open the Flow Name input, which works a lot like a step input. The results of the preprocess flow test run are available here as an object named results. We can select results.flowName that our preprocess flow returned from the result picker. We'll do the same for the External Customer ID input - this time we'll select results.customerExternalId:

That's it. All instances we deploy of this integration will share a single webhook endpoint, and webhook invocations will be routed to the proper customer and flow based on the information contained in the body of the webhook request.

Follow-up questions

Can I test endpoint config in the designer?

Yes! Your requests will always be routed to your integration designer (and not actual instances), but you can open the test runner drawer and open Test Configuration. Then, select Endpoint payload and paste in a sample payload.

Once you've done that, click the globe icon to the right of Run to run a test of endpoint configuration.

Was the preprocess flow necessary here?

Possibly. If the webhook trigger had contained the flow name and customer external ID that we needed, we could have omitted the preprocess flow. When no preprocess flow is assigned, the results object that External Customer ID and Flow Name can reference contains the header and payload information from the webhook invocation. We could have mapped the Prismatic customers' external ID from the webhook's results.body.data.externalId. That would require AcmeERP to be cognizant of the external IDs we assign in to customers in Prismatic, which may or may not be possible in the third-party system.

For Flow Name, we could have named our flows inventory_update rather than Update Inventory and create_order rather than Create Order, and we could configured Flow Name to reference the webhook request's results.body.data.type instead. That would look less pretty in the integration designer, so again, there are trade-offs there.

Could the customer ID have been passed in as a header instead of in the body?

Yes. The customer ID could have just as easily been a header as a value in the body. In that case, the lookup action could have referenced stepResults.integrationTrigger.results.headers.customerId rather than stepResults.integrationTrigger.results.body.customerId.

Do the same concepts apply to an instance-specific webhook?

Yes. Instance-specific webhooks give you a unique webhook per instance you deploy (so, each customer gets its own webhook endpoint). In that situation, you wouldn't need to worry about Customer External ID, but you would still want a way to route to a particular flow, so the flow-name-mapping portion of this tutorial still applies.