Integrations are hard. By definition, an integration connects two or more systems to enable data exchange. These systems are often using disparate databases, auth, and APIs. And it's common for these systems to be under development also, which means you are trying to hit a moving target.
That said, using well-defined requirements to build integrations can speed time to market for your product's integrations, reduce development, deployment, and support costs, and ensure that end users are pleased with the results.
What are integration requirements?
Integration requirements describe what an integration does and how it is expected to function. A collection of these is called a specification.
We sometimes talk about high-level versus detailed requirements. Here's an example of a high-level integration requirement: "The integration syncs order data between the CRM and Accounting systems."
If we were to start developing an integration based on this requirement, we would end up in an unknown space very quickly. High-level requirements help us define the box, but we need detailed requirements to fill the box. When we refer to requirements for the balance of this post, we are talking about detailed requirements.
Why are requirements critical for integrations?
Requirements for integrations are often considered unimportant compared to requirements for other software. This thinking about integrations stems from a simplistic understanding: "It just copies data from one system to another. How hard can it be?"
Integrations are much like icebergs. Most of the integration is not visible. So, if we only create requirements for what we can easily see, we'll build something ill-suited to the underlying business need.
Integration requirements are essential to establish an agreed-upon integration scope for all the stakeholders, from engineering to end-users. Yes, even if your devs take an agile approach to integration development, solid requirements make the difference between success and struggle.
Common questions for integration requirements gathering
The nice thing about gathering requirements for integrations is that we know, at the highest level, what integrations do. This knowledge allows us to narrow the scope of requirements questions from the beginning.
Here are standard questions to solicit integration requirements:
- What will cause the integration to run, and how often will it run?
- Will this be a one-way or two-way integration (or something else)?
- What auth method will each system use?
- What API will each system use?
- What data is to be transferred?
- What transport languages will be used?
- What transfer protocols will be used?
- Will any data need to be modified within the integration?
Let's look at these questions in detail and see how they help us build a complete picture of integration requirements.
What will cause the integration to run, and how often will it run?
Integrations generally have some type of trigger that causes them to run. These triggers may be event-based (for example, a change in an order record in the CRM) or scheduled (for example, every Thursday at 7:15 AM local time). They may also have manual triggers (if something needs to be forced to run or if the need for data transfer can't be planned programmatically).
Will this be a one-way or two-way integration (or something else)?
A one-way integration sends data from one system to another. A two-way integration sends and receives data. In addition, a single integration could involve three or more systems, each of which could send and/or receive data.
Since one-way integrations tend to be simpler than N-way integrations, knowing the direction of the integration is a significant factor in determining the overall integration effort.
What auth method will each system use?
Auth (or authentication) is needed to ensure integrations run securely. Knowing the auth method for each system lets us determine a crucial part of what we'll need to configure for the integration. Systems generally use basic auth, API keys, or OAuth (though other auth types exist). Along with the type of auth, there is the question of whether auth is defined per user or system.
What API will each system use?
Not all integrations need APIs, but most of them will. Knowing the types of APIs being used gives developers a good idea of the rules and restrictions they'll be working with. REST APIs are used more often than any other type of API, but webhooks (sometimes called reverse APIs) are increasingly common. And some integrations may have an API on one end and a file import/export or something else on the other.
What data is to be transferred?
Data transfer is the core of an integration, so getting this right is critical. We often speak loosely about data, using terms like "order data" (going back to our high-level requirement example). Does it mean records from the order table in the sending or receiving system's database (or both)? Does the order data include data from any other tables? If there is conflicting data in the two systems, which one overrides the other?
In many cases, end users look at data on a screen and don't realize that the screen is displaying data from half a dozen database tables. When you specify the data to be transferred, doing so field by field is essential. If "order data" is comprised of 35 fields, explicitly list all those fields.
What transport languages will be used?
People use human languages to communicate. Integrations use transport languages to encode the data which is sent between them. XML and JSON are the most common transport languages (also called data exchange formats or data interchange formats), but there are others.
Sending and receiving systems may use different transport languages. In that case, transport language decoding and encoding must be part of the integration.
What transfer protocols will be used?
In addition to being encoded in specific languages, integrations also use transfer protocols (a network protocol that supports data transfer between systems). While the APIs often determine these protocols, some systems may provide a choice of protocols.
HTTP (and HTTPS) are the most used transfer protocols for integrations running over the web. Integrations that are transferring files might rely on FTP, FTPS, or SFTP.
Will any data need to be modified within the integration?
We touched on translating/mapping between transport languages above, but that's only the start. You may also need to manipulate text strings, perform math calculations, and map data elements from dissimilar schemas. While these are the most common ways to modify data within an integration, this is not an exhaustive list.
To determine needed modifications to integration data, you must compare input data requirements with output data requirements to see what actions the integration must take to transform the input data to match the output data.
Tips for integration requirements gathering
If you've received detailed answers to all the questions in the preceding section, you may not have a finished integration requirements specification, but you should have a good draft.
In addition to those specific questions, here are some tips to inform your overall process and help you turn that draft into a solid specification:
- Talk to end users
- Talk to more users
- Seek out corner cases
- Engage with your tech partner
Talk to end users
Every organization handles requirements gathering differently and expects different roles to contribute. That's fine if you can ensure that you are soliciting requirements information from everyone. And by everyone, we mean everyone who has something to contribute.
To ensure you capture a sufficiently detailed list of requirements, we recommend talking with end users who work with any part of the systems or process with which the integration will interact. If there are users who copy data from System A and place it into System B (and the integration is supposed to replace that manual step), talk with them. Get them to show you precisely what they are doing today and share their expectations of how the integration will change their workflow.
Talk to more users
Remember to gather requirements from multiple customers, ensuring you capture the differences between those customers. 95% of the functionality may be identical between customers. But the 5% that is different is critical. Maybe some customers need to include one or more JPG files in the data transfer, while others don't need JPG files at all. Perhaps some customers only need 20 of 25 available data elements, but other customers need all 25.
To productize an integration, you'll need to document the differences (the 5%), decide what is essential to all your customers, and then build a single integration. Inputs and other values that should differ per customer can be set as configuration options, allowing you to build once and deploy many times.
Seek out corner cases
As part of fleshing out the high-level requirements to get the level of detail you need for the specification, you'll need to identify the corner cases – scenarios that may not be immediately obvious but can have an oversized impact on the integration.
Examples of corner cases may include things like:
- Middle name is optional in our product, but required in the system we are sending data to. What are we planning to do for records with no middle name value?
- The other system includes voided transactions. Can we process those into our product?
- Are there other record types that don't fit the standard pattern?
All of these questions come down to not making assumptions, but figuring out what the possible scenarios can be and how you will ensure that your integration can handle them. Sorting these out in the requirements phase will save your team a lot of time and rework during development.
Engage with your tech partner
To integrate your SaaS product with a tech partner's app, confirm that you have access to the corresponding API and app (and all available documentation). If you need an NDA with that partner before you and your team can dig into the details, get that done as quickly as possible.
When working with a tech partner, it is common to find that the integration is not as important to them as it is to you. As a result, they lack a sense of urgency and don't see answering your questions or meeting with your team as priorities. You must determine how to make gathering requirements a priority for them or at least come to a shared understanding concerning how quickly and thoroughly they will answer questions so you can plan accordingly.
Requirements are foundational
There is more to building good integrations than gathering requirements. But you need solid requirements to build first-class integrations.
Asking questions to gather integration requirements can fill that box we mentioned earlier. But what do you do at that point? Do you hand the box over to your development team? Or do you implement an embedded iPaaS to supercharge your integration development, deployment, and support efforts?
An embedded iPaaS provides:
- A means to react quickly to new integration requests from prospects and customers
- A low-code designer with pre-built components
- The ability to write code when needed
- Collaboration with your current dev process and tools
- An embedded integration designer for end users to build their own integrations with your product
- Infrastructure purpose-built to address security, availability, and reliability
- For technical non-devs to build, demo, test, deploy, and manage large numbers of integrations
If an embedded iPaaS is something you should consider for your SaaS integrations, contact us or schedule a demo, and we'll be glad to show you how Prismatic can power your integration strategy.
