Post 5 of the Integration Health series
- Integrations can succeed even if execution time keeps growing. A scheduled sync may finish after a downstream process needs its data, so records arrive, but after their usefulness has passed.
- Start, end, and inclusion rules need consistent definitions. Decide whether duration includes queue time, downstream wait, retries, or replay and recovery time, and apply the definition consistently.
- Segment duration by flow, customer, trigger type, and workload. A portfolio average can hide a slowdown for one integration or customer. Compare simillar-sized workloads.
- Split first-run and retry duration reporting. Retries improve success but lengthen customer wait. Pair duration with retry rate and retry recovery rate to get the full picture.
An integration can succeed (complete successfully) while execution time keeps getting longer. Customers may not see any specific errors, but records arrive later than expected. As a result, a scheduled sync may finish after a downstream process needs its data, or a lengthening queue may turn a delay into a missed deadline.
A recorded-execution success rate tells you whether recorded executions completed. Expected execution success rate also accounts for work that never produced an execution record.
The integration execution duration answers how long an integration took to run.
If your customers are complaining about delayed data or missed deadlines, this is for you. Measuring integration execution duration is essential as soon as you have multiple customer integrations in production and need to identify performance degradation before slow executions become issues for you and your customers.
Watch the video
Watch the overview, or keep reading for the details.
What is integration execution duration?
Integration execution duration is the elapsed time from an execution's accepted trigger or start timestamp to its terminal state.
Execution duration = Terminal timestamp − Accepted-trigger or start timestamp
You'll want to clearly define start and end timestamps for the duration before you start calculating. Decide whether the duration includes queue time, downstream wait time, retries, or anything else. If replay or recovery time is relevant, you may want to include that also. Whatever you decide, apply the definitions consistently.
Why duration matters
A slow execution may finish successfully, but that doesn't mean that the output arrived soon enough to be useful. Duration sometimes reveals degradation even when the success rate stays high. Rising duration may point to larger payloads, slow dependencies, queue pressure, processing problems, retries, or other issues.
These issues need different resolutions. If API-call latency is the issue, measure it separately at the step or dependency level; whole-execution duration does not tell you anything about request latency. Set expectations for customer data freshness and workflow deadlines: a lightweight event-driven flow and a large scheduled sync need different durations – and that's when everything is working correctly.
Look beyond the average
The average lets you run comparisons but can hide a small number of very slow executions. Report the median and upper percentiles as well:
- The median, or p50, describes typical execution.
- p95 shows the duration within which 95% of executions complete.
- p99 helps expose the longest-running tail.
A healthy median paired with a rising p95 can signal that a growing subset of executions is taking longer, even if the average changes little. Use p95 or p99 to assess customer impact, and dig into the underlying execution data to determine the root cause.
How to measure execution duration
1. Define the start and end points
Choose a consistent start event (when a trigger is accepted or processing begins), and end event (successful completion or complete failure). If queue wait time matters to your customers, either include it in the definition or report a separate end-to-end duration for that value.
You must use the same definition for the entire reporting period. Document any exclusions (planned maintenance or test executions). Report excluded dependency incidents separately so customer-impacting latency is easy to see.
2. Capture execution context
For each execution, record start and end timestamps, outcome, integration and flow, customer or instance where available, trigger type, and retry or replay status. Capture the payload size or record count when workload may explain duration; use step-level logging to see where the execution spent its time.
Keep in mind that the execution count is not business-data volume: one execution may process a single record, many records, or none at all. Measure workload directly when comparing durations for integrations with substantially different payload sizes.
3. Summarize the distribution
Calculate the duration per execution, then report the average, median, p95, and p99 (when the data supports those percentiles).
Use a consistent time window and denominator. Keep individual duration data accessible since looking at the aggregates alone doesn't allow you to see the outliers.
4. Segment and alert
Break results out by flow and customer, then by trigger type, payload size, connector, and outcome. Report clean first-run and retry executions separately when retries extend the duration.
We've seen the first 10 instances of an integration all complete in a total time of less than 30 seconds only to have the 11th instance try to process a dataset that's 100 times larger and fall on its face.
A portfolio average can hide a slowdown in a single integration (or even for a single customer). Changes across many integrations may point to a shared constraint; changes that are limited to a single connector or payload range can narrow the investigation. But these are only indicators and should be fully investigated to determine causes.
Set thresholds to reflect each integration's customer experience. Alert on sustained or repeated duration breaches rather than each outlier. And, of course, a single portfolio-wide threshold will not fit integrations with widely different workloads and expectations.
Investigate the slow tail
When duration shows that work is taking longer, check step timing and dependencies to help understand why. Check for a step or API call taking too long, large payloads, repeated calls that might be batched, rate-limit responses, queue delays, or retries that are extending an otherwise successful execution.
Treat any of these as something to verify against execution data. Track dependency latency separately from total execution duration, and compare like-sized workloads before concluding that the flow itself is taking more time than it ought.
Keep retries visible
Retries can improve eventual success, but they often do so at the expense of more customer wait time. Report clean first-run duration separately from duration for executions that required retries, and pair duration with retry rate and retry recovery rate. Longer durations with more retries may suggest slower recovery from transient failures; longer durations without more retries may have a different cause.
How this fits with other health indicators
Use duration along with these other metrics:
- Expected execution success rate asks whether expected work was completed, including work with no execution record.
- Mean time to detect measures time from failure start to detection.
- Mean time to recovery measures the time from detection to restoring the defined operational state.
- Retry rate and retry recovery rate show how often retries occur and whether they succeed.
- Data volume and throughput measure how much business data is processed and at what rate.
Together, these metrics distinguish missed work, failed work, slow work, and work that completed too late to meet customer requirements.
What Prismatic provides
Prismatic execution history includes both timestamps and outcomes, and step-level timing helps identify where execution time was spent. Configured duration alerts help you know when an integration breaches a defined threshold. Execution and instance context can support investigations, though customer-level reporting depends on the availability of relevant data.
Your team must also define timing boundaries, inclusion rules, thresholds, and reporting segments. Calculate percentiles where execution data is aggregated and don't assume that execution history automatically provides p50, p95, or p99 portfolio dashboards.
Where to start
Begin with a recent rolling window and a handful of high-value integrations:
- Define start and end events, including queueing and retries as appropriate.
- Establish baselines for per-execution duration, median, p95, and p99.
- Segment by flow, customer, trigger type, workload, outcome, and other factors you care about.
- Compare clean and retry executions separately.
- Investigate tail changes with step timing and dependency data.
- Set thresholds based on integration deadlines and customer data-freshness needs and alert on sustained breaches.
A useful duration target for a given integration reflects when an integration's output is still on time for its customers. Track the distribution, keep the definition explicit, and investigate changes before slow executions turn into failures.
Series navigation
Post 4: Error composition by category
Post 5: This one.
Post 6: Coming soon
View all the posts in the Integration Health series.
