Integrations Shouldn't Be So Hard: An Engineer's Perspective

This is Part 2 in a three-part series. Read Part 1 here. Read Part 3 here.

In our last post, we talked about why the difficulty of providing software integrations matters so much that we're creating an integration platform to make it easier. In this follow-up, I'll get into the weeds a bit on what makes building, deploying, and supporting effective integrations so incredibly difficult from the perspective of someone on an engineering team.

We'll look at some of the biggest pain points my engineering teams have experienced and that the team at Prismatic has heard from many of you:

1. Non-standard APIs, formats, and protocols
2. Unclear requirements
3. Poor documentation
4. Mixing of integration logic with main application code
5. Difficulty testing
6. Insufficient logging and alerting

But first, who's this guy?

Who am I and why should you believe that I have even the slightest idea what I'm talking about? Great questions!

I'm the CTO and a cofounder here at Prismatic. Before starting Prismatic, my cofounders and I spent the past 15 years or so building industry-leading public safety software systems. Those systems are now used in thousands of municipalities across the country to do everything from improving emergency response times (awesome!) to sending out invoices for unpaid parking tickets (some people are less excited about this). Like any legitimately large and complex system, these systems don't exist in isolation and as such we had to build hundreds of specialized integrations between our systems and the other systems that existed in our customers' IT ecosystems.

And it was really hard.

Through the years, I got to experience the difficulties of integration development in a number of different roles. In the early days I was an individual contributor, writing code to handle innumerable specialized integrations alongside my compatriots on the development team. Years later, as an engineering manager, I strove to find ways to help make the team more efficient at handling the growing number of integrations through better tooling and processes. By this time the business had really started to take off, and with it, the number and complexity of integrations we had to develop. It seemed that no matter what we did we were always strapped for capacity, with integrations taking a good chunk of it.

By the time I became an executive several years later, the business had gone through tremendous growth and I was overseeing several multi-discipline teams distributed across the nation, a couple hundred engineers in total.

Alright, so what makes integrations so hard?

Building good software is never all that easy to start with, but integrations crank the difficulty setting up a notch or two in a number of different ways. There are plenty of potential pitfalls through the entire process between selling and maintaining integrations, and certainly every team involved with them has their challenges, but for the purposes of this post we're going to focus primarily on those experienced most acutely by engineering teams.

The "Real World" is NOT composed only of pristine APIs, formats, and protocols

It's just not. It would be neat if it were, but in my experience, we've got a long way to go. For every integration we built where the other vendor had anything close to a proper REST API, for example, there were dozens more where their idea of an API was to drop a CSV on a file share, call an RPC over a raw socket, or involve something like SOAP. Formats and protocols were all over the map. Based on conversations with others who serve customers in many different software industries, our experiences are hardly unique.

While there is a plethora of wonderful technologies such as REST, Webhooks, GraphQL, etc. that aim to make building integrations a much more pleasant experience, the hard truth is that it will probably take quite a while for those things to become widely adopted in many industries. Unfortunately, your customers need their integrations now, so you can't wait until the world adopts whatever the latest trend is.

It's often unclear how an integration is actually supposed to work

It's a time-honored tradition in software development that you end up building the wrong thing because nobody really understood what was actually needed. I can imagine the first software ever built, whereupon delivery the proud creator was told to go back and do it over again as it wasn't quite what was requested. (Or it was, but that's not what the user actually meant now that they see it and…)

Integrations add their own little twist in that, unlike with most product development where your team is working with the customers to identify their requirements, you've now got a third-party vendor in the mix. Communication is always difficult, even between two parties, and adding a third party makes it exponentially more so. It's not uncommon for all three parties to leave a conversation with three different ideas of what was discussed, and for everyone to set off building their idea of what is needed only to discover much later that there was a pretty major misunderstanding.

Integration documentation tends to be lacking

Perhaps the next oldest tradition in software development is the lack of documentation. I'm sure 20 minutes after the first software was written there was another developer asking where the documentation for it could be found. Documentation is often sparse, incorrect, or flat out non-existent, so determining how another vendor's system works and how to correctly integrate with it can be tough. Add in the fact that, at least in some situations, the vendor is a legitimate competitor and thus can be, ahem, "less than helpful" at times, and now you've got a really big challenge.

Just like unclear requirements, lack of documentation is a communication problem that leaves you wading through multiple iterations with slow feedback loops instead of achieving "first time right" very often.

Third-party vendor implementation details leak into your application codebase

It always starts innocently enough. You're working on some new integration and the other vendor's API doesn't quite line up with yours. Maybe they have a different data model, maybe they're requiring a particular header be passed along in every request, maybe something else entirely. You think to yourself "If I just add another parameter to this function, add another attribute to this REST resource, then it will work." But we all know how this goes. This same scene is played out time and again, month after month. All of a sudden you look up one day and your API is an incoherent mess and your application has other people's design choices baked into it all over the place. How did this happen?

Not keeping a diligent eye on enforcing reasonable separation of concerns is a recipe for a real maintenance nightmare down the road.

Properly testing integrations is difficult

You can get a long way, of course, by following industry best practices with respect to unit testing, end-to-end testing, etc. But it's tough to feel too confident about any integration being deployed to production if it hasn't been tested against a vendor's test endpoint. Unfortunately, all too often vendors don't have test endpoints. So you are stuck with either writing your own mock endpoint, which can often take a non-trivial amount of time and effort, or trying to convince yourself, your team, your boss, and the customer that it will be fine to Test-In-Production™. (Spoiler: It often goes poorly.)

If I'm honest, I would say that I've made the choice to Test-In-Production more than once, which is how I know it's usually a bad idea. Some lessons you just have to learn the hard way, I guess.

Supporting integrations in production can be very frustrating for everyone involved

Despite our best efforts, all software eventually fails. For some reason, it often seems to happen in the middle of the night or at 4:50 P.M. on a Friday. What matters most in that moment is identifying the cause of the failure and fixing it as quickly as possible. One of the most basic and yet most important tools we use to do this is the log. Yes, the humble log often contains the information that can mean the difference between going home a hero or staying late on a Friday evening, all the while taking heat from rightfully angry customers whose integration is broken.

Checking logs is not always so straightforward, however. Oftentimes application logs are tucked away on servers where support staff, and often even developers themselves, don't have direct access to them. Getting them may require getting DevOps or Sysadmin involved. Often the end customers themselves want to know the status of their integrations, but it's not always so simple to provide them access to view just the logs that pertain to their integrations. This can be very frustrating for them, as without some way to view their own integration logs they aren't sure whether they should call Vendor A or Vendor B when there's a problem, as they have no way to know which end might be broken. So they usually end up calling both, and then the finger pointing begins. I can think of much better ways to spend a Friday evening.

What's even better than having a good way to track down a problem quickly after a customer calls is being able to track down a problem before the customer even notices. This is why it's important to have some mechanism built into every integration for alerting your staff, and perhaps select staff at your customer's IT department, for example, when the system notices a problem. Unfortunately, it's difficult to build out a sophisticated alerting system, so it often gets left in the "nice to have" category for eternity.

Wrapping Up

This certainly isn't an exhaustive list, but these are many of the kinds of problems that my teams and I experienced continuously for 15 years. Integrations were hard when we were a scrappy startup, and they were just as hard when we were the largest company in our space.

From the conversations that I and the rest of the team at Prismatic have been having with many of you who come from other B2B software industries, it's clear that we share a lot of the same stories and pains. Let's look back at that list:

1. Non-standard APIs, formats, and protocols
2. Unclear requirements
3. Poor documentation
4. Mixing of integration logic with main application code
5. Difficulty testing
6. Insufficient logging and alerting

We truly believe that there is a massive untapped potential for connecting the software that makes up our customers' technology ecosystems, but that the integrations to make it possible are often overly difficult or sometimes just plain infeasible, often due to the kinds of challenges that we discussed here. That's the motivation behind our integration platform, Prismatic. Its purpose-built integration tools are designed to specifically address each of these problems, and others like them, to make integrations easier for engineering teams.

Which of these are the biggest integration pain points for your engineering team? What else should be on this list? Got any crazy 4:50 on Friday integration stories? Send me your thoughts at feedback@prismatic.io.

This was Part 2 of a three-part series. Read Part 1 here. In Part 3, Prismatic VP of Marketing and Developer Relations Beth Harwood will address some of the integration challenges faced by other groups in B2B software companies. (Read Part 3 here.)