[ding 🔔] goes your email notification. You've just been assigned a new ticket. The subject line reads "Integrate our app with Acme CRM." The ticket's description is pretty sparse – just a few bullet-points saying things like:
- Sync users' appointments in Acme with our calendar app
- Track the number of email and phone touch-points users have in Acme on our dashboard
You expected this. A lot of the businesses that use your software also use Acme CRM, and you've been hearing from your sales and support teams that customers keep asking for their Acme data to sync with your app.
So.... it's up to you to build the integration with Acme. Where do you even begin?
I know I'm always inclined to fire up my IDE and dive right in to code, but that always results in misunderstanding expectations, missing details, and having to redo work.
Let's talk about the stuff you should do before you start coding. Let's think through the questions you should ask, requirements you should clarify, and tools you can reach for to get a feel for how the integration is going to work.
Gather requirements for your integration project
In an ideal world, the person requesting the integration would provide you with a specification that details what events trigger the integration to run, what type of data gets sent over, and how the data should look when it arrives at its destination. The world's rarely ideal, though, so you'll probably have to go digging for clearer integration requirements.
It's incredibly helpful to have a series of "When foo then bar" statements. For example, "When a user in Acme CRM sets an appointment with a customer, a calendar event is created in our app that contains the customer's name, phone number, email address, and time of the appointment." Bonus points if you can tie each step to relevant API documentation!
As you think through the "when foo then bar" statements, there are several companion questions you should be asking with regards to how data should flow from one system to another:
- What triggers an integration to run? Does Acme support events like webhooks (where they send data whenever it's been changed) or are you expected to fetch data on a schedule?
- Is this a one-way or two-way integration (that is, will Acme just send data to your app, or does your app also need to send data back to Acme)?
- What's the format of the data the source system is sending? JSON? XML? CSV?
- What information is included in the payload that's sent your way?
- What format of data is the destination system expecting? You may need to fetch more information from other third-parties or Acme's other API endpoints to satisfy the destination's requirements.
After you have a sense of how data will flow, ask yourself what things could go wrong, and clarify how your integration needs to behave in these common situations:
- How should you deal with data "collisions" where the same data gets sent over twice? Should you ignore the second request, create a second record, or update ("upsert") the existing data?
- What should happen if incomplete or "bad" data is shipped (a calendar event that's missing date, for example)? Do you need a “dead letter queue” to store data that causes your integration to error out so your devs or support folks can examine the bad data later?
- What happens if either API is down? Is your integration expected to hold on to data and retry later?
Next, figure out how configurable you need this integration to be. (Occasionally, I've needed to build "one-off" integrations for a single customer. Far more often, I've needed to build reusable integrations that can behave differently from customer to customer and handle each customer's credentials for the third-party app.)
- How do users authenticate with each system? Do users have their own credentials to Acme, or some shared key? Where are you going to store those secrets securely?
- What things are going to be different between customers you deploy this integration to? Do customers have different API endpoints, or data mapping requirements?
- What all needs to be configurable to make it so you can deploy the same integration to multiple customers?
Finally, figure out deployment and support concerns. If your company has several app integrations already, you may already have answers to these questions:
- How will your customers activate and configure the integration? Do you need to build UX for this?
- What are the expectations for logging, monitoring and alerting? Where should the integration's logs be sent, and under what conditions should someone be alerted if the integration runs into a problem?
- Where is the integration itself going to live? Will you need to build custom infrastructure to host this integration?
As you can see, a quick bullet point spec like "Import users' appointments into the our calendar" seems simple at first, but there's a ton of things to think through if you want to build, deploy and support your integration thoroughly.
Get familiar with how the third-party app works
Once you've thought through all of the questions above and have a solid understanding of the task at hand, it's time for you to familiarize yourself with the third-party app(s) involved. I recommend logging in to Acme CRM like an end user would. Figure out what exactly it means to "set an appointment" or "log a phone call with a customer" in the context of Acme. This should give you a sense of how different objects that the application stores are related.
If you don't have an Acme CRM account, ask for one. It's mutually beneficial for both your company and Acme to have an integration between your systems, and most reasonable companies are willing to provide a free developer account or “sandbox” environment where you can toy around with dummy data, without needing to access your users' real Acme data.
You may be provided written information about how the other application will provide data to you, but I recommend that you insist on getting a real sandbox environment that at least sends fake data in the format the third party says they use. I've been around for plenty of integrations where a third party claims they'll send XML in thus-and-such a format using REST calls, but informs you after you build your integration that they decided to write out JSON messages to a message broker (like Kafka or Amazon SNS) instead. You can just change your code quick to subscribe to an SNS feed, right? Testing with an actual system is far better than mocking out fake APIs and message streams yourself.
It sounds silly, but also make sure to log in to your own app as well to verify that you know what it means to "create a calendar event" or "update the dashboard." Verify with the people requesting the integration that you understand what actions should trigger the integration, and where the data should be displayed when it arrives at its destination.
Explore the API with an HTTP client
You're ready to start fiddling with APIs, but it's not quite time to code quite yet. Take a look at Acme's developer docs. If they have an OpenAPI/Swagger or WSDL spec that defines all for their API endpoints, you're in luck! Figure out how their auth flow works, what data their webhooks send, and what data is available for you to query. If you're really lucky, Acme will provide a client library you can use when it's time to start coding (but don't get your hopes up!).
Fire up your favorite HTTP client (I personally like Postman), and try to make some simple authenticated GET and POST requests against the Acme CRM sandbox you have access to. Verify that the third-party docs are correct, and that you can, indeed, fetch data from their appointment API endpoint. Double-check that the payload you receive has all the information you need to map data over to your app.
Once you have requests to Acme CRM working in Postman (or your favorite HTTP client), finally turn to code.
Time to build!
Now that you have a good idea of the shape of data that comes in, where additional data should be fetched from, how it should be modified and where it should wind up, it's time to start building your integration.
There's still a lot to do – you need to write your code, make it configurable enough to handle differences between customers, deploy it, monitor and support it, make the integration available within your product, etc., but asking the right questions and taking the proper steps beforehand can make all that go faster and keep rework to a minimum.
