As I work with our customers (SaaS teams who need to build integrations to their customers' other apps), I'm constantly reminded that real-world integrations are often not straightforward. While many integrations use APIs, others don't have an API, have an API on one end of the integration but not the other, or have an API buried in the middle. I want to outline a few of these "no API" integration scenarios and discuss how to address them.
Remember that even though we are talking about integrations with an API-less architecture, we still need authentication and security. Data passing from one app to another must be secured and accessible to authorized users only.
While we are referring to these as API-less integrations, it's more accurate to refer to them as API-less integration connections since the same integration that connects directly to a database on one end might well connect with a standard type of API on the other. And sometimes we'll see an API from a third party used as middleware (such as with a message broker). Building these integrations can often be more challenging because the API functionality we often rely on for integrations isn't there. As a result, we use additional tools or write code to bridge those functional gaps.
Older systems (sometimes called legacy systems) are most likely not to have APIs. This is true for various reasons, but chief among them is that some systems have been around so long that they pre-date modern HTTP conventions.
Some market verticals, such as banking and insurance, are more likely to have older systems in place, partly because those industries are heavily regulated. Conforming to these regulations adds non-trivial amounts of work to swapping out systems that have been running for decades.
However, the need to integrate systems with no APIs may come up in any industry. Surprising as it may be, even modern systems are sometimes built without integration APIs.
Types of API-less integrations
There are still ways to create integrations when you don't have standard APIs. Let's look at the four API-less integration scenarios that I see our customers working through most often:
- File transfer
- Message queue and message broker
- Direct database connection
- Custom code connection
File transfer integrations
Not all file transfer integrations use an API-less architecture. Collaboration tools such as Dropbox and Box, and file-sharing systems such as Azure Files, all use APIs for file transfers. Dropbox uses a JSON-RPC API, while Box and Azure Files each depend on a REST API.
It's common to have scenarios where a system that doesn't have an API does have the functionality to export data in a flat-file format such as CSV (comma-separated values). In many cases, these exports are made directly from the underlying database, or a thin wrapper around the database.
The source app exports the file to either a file system or an FTP or SFTP server. From here, a file transfer integration may work in a few different ways:
- One way is to have the integration pick up the file from where it's sitting and move it to another location. For example, the integration might move the file from one SFTP server to another.
- More commonly, the integration picks up the file from its location, converts it (the file) into another format, and inserts that data into an HTTP request before the request is sent to an API for the destination app.
- We also see the reverse, where data is exported with an API, converted into a CSV, JSON or XML file, and the resulting file placed on an SFTP server for the destination app to import.
In the first scenario, there truly is no API involved. In the second scenario, there is an API on the receiving end of the integration. In the third scenario, there is an API on the sending end of the integration. In any case, a file storage system is used as an intermediary to share data between two apps.
An example of a file transfer integration might be a manufacturing company with a legacy inventory system. The system runs a nightly batch job to create a CSV file that shows all products which are at or below predefined inventory levels. The CSV file is then placed into an SFTP location. The file is picked up from the SFTP system, converted into XML, and submitted to the SaaS accounting system's API. Based on that import, the accounting system creates purchase orders to restock needed inventory.
Message queue and message broker integrations
If either side of an integration lacks an API, data can be shared between two apps via a message broker or message queue. As the name suggests, message queues like Amazon SQS, MQTT or RabbitMQ queue up messages between applications. One app can write messages to the queue and another app can then fetch messages from the queue. Message brokers extend the idea of a queue further – systems like Amazon SNS and Apache Kafka allow an app to write messages to a queue, and the broker can then inspect the message and forward it to interested parties.
Here's how a message queue app like Amazon SQS works. An app sends a message to Amazon SQS. Another app can then request messages from Amazon SQS. After processing those messages, the receiving app can mark the messages as "read" or "processed." Amazon SQS is a pull system – the receiving app needs to request messages from the queue.
Compare that to a message broker like Amazon SNS. SNS is a push system (since the receiving system gets the message as soon as it's published). SNS can push messages to subscribers via HTTP request, SMS, email and more. Message brokers are useful when when a source system finds it easier to publish data to a location where other systems can read it rather than supporting thousands of queries in a relatively short time for the same data. Multiple apps can subscribe to a single topic (SNS calls a queue a topic).
A example for this might be a central bank that needs to constantly update interest rates. Banks and other businesses can subscribe to an interest rate topic and be notified immediately when rates change, allowing them to make business decisions based on those changes.
Apache Kafka, unlike Amazon SNS, can work as both a queue and a broker. Kafka's producer API allows an app to send messages to Kafka. Its consumer API allows another app to read messages from the queue. Additionally, it offers a stream API and can act as a message broker and forward messages to a consumer in real time.
Integrations using a message queue or a message broker tend to handle numerous small messages compared to file transfer integrations or standard APIs (REST, SOAP, etc.)
Direct database connection integrations
Because of the security issues inherent in connecting directly to a database from outside the local network, these types of integrations are becoming less common than they once were. However, there are still scenarios where such connections are the only way to pass data between systems.
Direct database connections should always use credentials to ensure no one can connect to the database without proper authentication. Network ACL rules are also often put into place to limit connections by source IP address. In addition, it's best practice (if you are opening up the database to such a connection) not to allow any user full access to the database but rather to build database views as necessary to prevent users from accessing restricted data.
Because database connections are generally made with SQL Server, MySQL, PostgreSQL, or another relational database, they rely on SQL for the queries. In one regard, this is good, because devs are generally very comfortable with SQL queries. However, if the database is not properly locked down (from authentication to views), the power of SQL queries makes it very easy for the system to be abused.
Whether the direct database connection is for import to the database or export from it (or both), be aware that such connections run the risk of overwhelming database resources – assuming that the database is also being used as the back-end for a production system. For long-running queries, it's sometimes better to create read replicas of the database specifically for integrations.
An example might be a call center that has an older recording system, but needs to tie those recordings to a SaaS ticketing system. Devs set up a direct database connection where the SaaS ticketing system queries the recording system for metadata about the recording of each call, so it can update the ticket details accordingly.
Custom code connection integrations
When does code become an API? This question sometimes comes into play with custom code connection integrations. By this, we mean apps or systems that don't have a standard API but can still be accessed via custom code. If we consider the relationship between an API and an app as being that of a door into a building, we should consider the custom code connection in terms of a mail slot or drop-off box.
Of course, devs can write custom code to both export and import data from an app, so our analogy breaks down rather quickly, but we hope it makes sense insofar as it goes. Custom code enables integrations for apps that don't have an API or another means of importing or exporting data. Often, were this custom code to be expanded and formalized, the app would end up with an API.
But in many cases, the code is written to meet a specific, narrow set of business requirements. For example, we might have an older procurement system that lacks an API or an easy means of exporting shipping data, but we need the shipping data in the CRM. As a result, devs have written custom code to export the shipping data for each order from the procurement system and send it to the CRM.
In some scenarios, the export may be a flat file. In other scenarios, the data may be encoded in the format (such as XML or JSON) required by the receiving system. No matter the specifics, the approach to building the integration rests on the custom code. If the custom code uses functionality largely congruent with a standard type of API (REST, SOAP, XML-RPC, or GraphQL), then we'll connect with the custom code according to that pattern.
However, the nature of custom code is that it may be done in a way that doesn't conform to any standard API pattern. If that's the case, we may need to write additional custom code for the integration to either accept data from or provide data to the custom code.
Integrations without APIs are messy but entirely doable
While REST, SOAP, RPC, and GraphQL can make building integrations a more pleasant experience, we know that for every legitimate API for integrating B2B SaaS apps, there is a handful of not-quite-API or not-really-API scenarios that also need integrations.
We've spent some time discussing four different categories of API-less integrations in the hope that you'll be able to understand them better than before. However, even with that understanding, the reality is that building API-less integrations is extremely messy and time-consuming. Building integrations from scratch can interfere with development on your core app, even without factoring in all the extra work that API-less integrations can take. Building API-less integrations can be incredibly frustrating for your devs and very, very expensive.
But this is where an embedded iPaaS can make a huge, positive difference. Prismatic's embedded iPaaS includes the connectors you need to handle file transfer integrations, message queue and message broker integrations, and direct database connection integrations. In addition, devs can use TypeScript to build the code for any custom connector necessary for other API-less scenarios that don't fit into one of the categories I've already mentioned. Our embedded iPaaS also includes everything else you need to deploy, run, and manage native integrations for your customers. Create the integrations your customers need, whether your integration scenario has lots of APIs or none at all.
Schedule a demo. We can dig into the details of your not-so-standard integration scenario together.
