When Should You Use Scheduled Integrations?

In a recent post, we introduced the common integration patterns we see used for B2B SaaS. Then, we looked at event-driven integrations, perhaps the most common type we see teams building on Prismatic. In this post, we'll examine the scheduled integration pattern, including an in-depth look at an example integration.

What is a scheduled integration?

When the system you need to transfer data from does not have webhooks or another pub/sub system to which you can subscribe, you'll often need to use a scheduled integration. Instead of being triggered when the integration receives a notification, a scheduled integration is triggered at a specific time or time interval. For example, a scheduled integration may run every five minutes or every Friday at 4:00 PM.

By definition, a scheduled integration isn't real-time. The data may be only a few minutes old for a scheduled integration that recurs frequently, but it will not be as fresh as one can get from an event-driven integration. However, scheduled integrations are probably the correct answer when there isn't an API, or the API only supports queries.

The main difference between a scheduled integration and an event-driven one is that the source system in an event-driven integration tells the integration, "I have something," but a scheduled integration asks the source system, "What do you have for me?" That is, the integration is using a pull rather than a push to transfer the data.

How often should scheduled integrations run?

The key with scheduled integrations is to know when the data or file is likely to be available in the source system. If the source system is only updating something once a day, there is no point to having the integration run every N minutes. Instead, if you know roughly (or exactly) when the source system is updated each day, you can set the integration to run shortly after that.

Of course, if you are importing data that is being updated constantly throughout the day, it may well make sense for you to set up the integration to run every few minutes.

Types of scheduled integrations

We generally see scheduled integrations used for the following scenarios:

• Importing data from classic APIs (no webhooks or other pub/sub system)
• Importing files from file systems or data platforms
• Exporting files to file systems or data platforms

We'll look at each of these in detail in the following sections and then an example of importing files from file systems or data platforms.

Looking at scheduled data import integrations

You'll want to use a scheduled integration to import data when you have a third-party API that does not have webhooks or another pub/sub system. If a pre-built Prismatic component for that API is available, you will use that. If a pre-built Prismatic component for this third-party API isn't available, you'll need to build a custom component that wraps the various API endpoints containing the data the integration will query. The API then processes the query and returns the requested payload.

In the event-driven import post, we talked about how you'll want to build a custom component for your app that can accept any data you need to import. The good news is that once you've built that component, you can also use it for scheduled import integrations. Your custom component doesn't care how the data was sourced; it takes the payload from Prismatic and provides it to your API.

Looking at scheduled file import integrations

Some third-party apps may be configured with a file export instead of an API, which is more common with non-SaaS or legacy systems. In this case, the app may write out data as PDF, XML, CSV, or another file type to an external data source such as Dropbox, SFTP, MS SQL, or a queue. To import data from these files, you'll build an integration that checks the location(s) according to some regular schedule to determine if there are new files to process.

When there is new data within the location (for example, new XML files in an SFTP directory), the third-party custom component you've built for your integration will loop over each of the files, process the files, and then move or delete the processed files. As with other scheduled integrations, this data transfer is not near-real-time but can be current within a few minutes of the file(s) being placed in the external data source.

Looking at scheduled file export integrations

We tend to see this type of integration when you need to generate a regular report for your customers. You'll set up an integration where you are either querying your own app's API or a third-party API for specific data.

Once the API returns the requested data, your integration generates a file with the data in the necessary format: PDF, XML, CSV, etc. Then, this file is packaged up and sent off to its destination (which could range from email to Dropbox or another file-hosting service).

You should be able to use a built-in Prismatic component to handle the part where the file is sent via email, placed into Dropbox, etc. Some of the common data platforms for which Prismatic has built-in components are Dropbox, Amazon S3, Google Drive, and Azure Files.

Example of a scheduled integration

We've looked at the various types of scheduled integrations. Now let's look at one scheduled integration in detail. This example is a scheduled file import integration that pulls files from Dropbox and places the data from those files into the Acme app (which we are using as a stand-in for your app). It doesn't matter what process or person originally placed the files into Dropbox (since that is not part of the integration).

Here are the steps:

1. Once the integration has been activated, the Schedule Trigger runs every five minutes.
2. Using functionality provided via the built-in Dropbox component, this step gets a list of all the files in the Dropbox folder.
3. Then, the system runs a loop for each file in the folder. If there are five files, the loop runs five times, etc.
4. The integration downloads a file from the Dropbox folder (Download Order Summary).
5. The file is deserialized from its original XML structure into into a JavaScript object whose properties can be referenced by subsequent steps.
6. The integration determines (according to header data from the file, for example) if this is a new order or an updated order.
7. Based on whether this is a new or updated order, the integration uses functionality from the Acme custom component to create or update an order in the Acme app (the stand-in we are using for your app), sending the data parsed from the XML file.
8. Using functionality provided by the built-in Dropbox component, the integration removes the file from the Dropbox folder (Delete Order Summary).

Once all the files have been looped over, processed, and removed, the integration ceases to run until the next scheduled time. There is no need to have a tear down step (as with the event-driven integration) because the integration stops when it runs out of files in the Dropbox folder.

That's it. You could easily expand this integration to import invoices in addition to orders, for instance, but the overall approach wouldn't change much.

Conclusion

While newer apps generally use an event-based API, there are thousands of legacy apps that either have query-based APIs or exchange data as files. Scheduled integrations allow you integrate your app with these apps in the most effective manner possible.

Prismatic makes it easy to integrate your app with any system your customers use — from the latest SaaS app to the oldest legacy software. Schedule a demo to learn more. We'd love to show you how it can help your team!