Integration Logs: Neglected but Essential

In the world of embedded integrations, logs are so much more than metadata. They are the means of understanding what's going on, what's going wrong, and how to get things working again.

Properly detailed logs will allow you to track all the essential stages of an integration's execution, or lack thereof. They provide real-time visibility into integration performance, support compliance audit trails, and let your team rapidly determine root causes.

Logs ensure that, having built and deployed an integration, your team doesn't waste considerable time running in circles because the integration doesn't include needed visibility.

Without logs, integration issues can be nightmares. The expectation of blind debugging sessions with your best friend, console.log(), increases stress levels when a new integration issue hits your ticketing system.

It doesn't need to be this way. Let's see what it takes to set up the logs you should have for integrations.

Integration logging done right

When you set up logging for integrations, the key is to capture enough data to be helpful, but not so much data that you get lost in a data jungle.

Key to this is understanding the data access pattern. For example, a Salesforce contact record came through the flow in the last few days, but it didn't sync correctly. You've had 5,000 executions within that timeframe, but you need to determine which one is tied to that failed sync.

To support scenarios like this, you'll want to define the core elements of the log. At a minimum, here's what you need to include:

• Timestamp
• Severity level: debug, info, warn, error
• Context: instance ID, etc.

Ensure that logs are created for the correct events: triggers, special steps, errors/exceptions, and completions. For example, if you have an integration looping over 10,000 records, you don't want to generate a log entry for every loop, but you should write one if loop 3728 of 10,000 failed.

Though logging everything can be tempting, don't do it. You'll end up with log spam, making it difficult to wade through the clutter to find what you need, not to mention the additional cost of streaming, storing, and managing all that data. In addition, overdone logging can tie up system memory, creating a cascade of failures.

Here are a few examples of what log entries should look like:

• Debug: Troubleshooting (”Fetching user data for ID:733”)
• Info: Normal tracking (”Instance ‘Salesforce Lead Sync’ started”)
• Warn: Non-critical issue (”Deprecated API endpoint used”)
• Error: Failure requiring action (”Salesforce connection timeout)

Logging in Prismatic

Logging is an essential and fully integrated part of the Prismatic platform.

Logs are retained for 14 days by default (though custom retention times are available). These logs cover integration invocations, test runs, connections, data sources, and trigger lifecycle functions (such as onInstanceDeploy).

The logs are easily accessible via the UI, with a global view of every customer and instance, enabling you to narrow things down to a single customer, instance, or connection. During development and testing, you can also work with trigger and data source logs via the Test Configuration drawer.

Logs are auto-generated for starts, ends, and steps. Additional log entries can be created by using the context.logger function in components. The UI provides icons (green for debug, gray for info, yellow for warn, and red for errors), making it easy to scan for issues. There's a search bar with filtering (severity, date ranges, flow, and type) to enable you to locate the problem area quickly.

Here’s an example of context.logger in action:

Log streaming from Prismatic

For some of our customers, the standard 14-day log data retention is sufficient. However, log streaming is necessary for many others to support business and compliance requirements.

While Prismatic has a robust logging system, we are not a purpose-built logging platform. Log streaming allows you to collect all your integration log data via a dedicated logging platform that holds your UI and app backend logs. This gives you a holistic view of everything, enabling the best data context for detailed troubleshooting.

So, which logging services does Prismatic support? Any logging system that can receive an HTTP data feed. DataDog and NewRelic are perhaps the most well-known of these systems. If you need to send logs to a system or service that doesn't support HTTP requests, you can set up a Google Cloud function to translate the Prismatic logs into something your log service can parse.

Here's an example of what Prismatic logs look like in DataDog.

Protect your integrations

While the inclusion of logging with a system is often an afterthought – something bolted on after a few failures when it becomes evident that more data is needed to troubleshoot efficiently – that is not the case with Prismatic. Given our extensive history with integrations, we know how critical good logging is and made it part of the platform from the beginning.

Building integrations without proper logging is asking for trouble. Logs prevent constant firefighting and set up your team to proactively support your integrations, handling many issues before they come to your customers' attention.

Are you ready to up-level your logging strategy? Dive into our free trial and see how Prismatic supports first-class logging for your integrations.