- Classify each failure as transient, permanent, or ambiguous. A transient failure (example, 503) can succeed later with the same request, and a permanent one (example, 400) needs a payload, configuration, or credential change first. An ambiguous failure, such as a payment API timing out after receiving the request, is an unknown outcome.
- Status codes are defaults that each vendor can add to. Some APIs will return a 200 with an error in the body, use a 403 to throttle, or even throw a 500 that you'll never solve with a retry. Encode each vendor's rules in a single classifier instead of scattering them throughout the flow logic.
- Attempt limits should have deadlines and stored counts. Five attempts with a 60-second timeout can waste several minutes before backoff is counted. Keep the attempt count, first-attempt time, and deadline on the job record so rescheduling doesn't reset the budget.
- Ambiguous failures should be addressed with idempotency or reconciliation, but not more retries. Use one idempotency key per logical operation, upserts on a stable external ID, or deduplication. If you don't have those protections, check the remote system first, then send the operation for review.
- Exhausted retries end in a defined state. Record the operation with context (and no secrets), park it in a dead-letter queue, and alert appropriately. Replay it only after determining whether any part of the original operation has already succeeded.
A customer integration sends an order record from your SaaS product to a third-party API and gets back a 503 error. It waits a moment, retries, and the request succeeds. The customer never notices.
But, the next time, the same integration gets a 400 error because the order contains a new field the API doesn't understand. It retries. And again. And again. Nothing changes, because nothing about the request changed. The integration is burning API quota and execution cycles on a problem no further attempts will solve.
These examples highlight the difference between a transient failure and a permanent one. A transient failure may clear up without changing the request, but a permanent one requires that something change.
Then, there’s the third kind that’s harder to understand (and work with). It results in an ambiguous failure. Here’s an example: a payment API times out after receiving a request but before responding. The payment may have gone through, and a retry could charge the customer twice.
Before each attempt, a good retry policy provides answers to the following:
- Should this be retried? (Is the failure transient, permanent, or ambiguous?)
- Is it safe to retry? (Could repeating it cause a negative side effect?)
- How long should retries continue? (What are the limits on attempts, elapsed time, etc.?)
This post looks at what goes into deciding what to retry and when to stop.
How do I tell transient failures from permanent ones?
- Transient – The same request could succeed later with no changes. Network blips, outages, and rate limits are common triggers for this type of failure.
- Permanent – The same request will keep failing until a change in the payload, configuration, credentials, permissions, or something else. Permanent doesn't mean unfixable but it does mean that continuing to retry the request is pointless until the change is made.
- Ambiguous – You don’t know for certain that the target system completed the operation. Timeouts and dropped connections after a request was sent are classic triggers for this. Only retry when repeating the operation is safe. Otherwise, check the remote state first.
The failure type doesn't always determine if the request is safe to retry. A provider-specific response may mean that a request wasn't processed. Or a timeout may leave the outcome unknown. Use idempotency or provider-specific behavior in these scenarios to decide whether a retry is the right next step.
Which HTTP errors should trigger retries?
Status codes are a decent place to start, but they don't constitute a complete policy. The provider's documented behavior (and the operation's replay safety) should determine your final decision on whether you retry.
| HTTP status | Typical class | Recommended behavior |
|---|---|---|
| 400 Bad Request | Permanent | Fix the request. Don't retry it unchanged. |
| 401 Unauthorized | Conditional | If the token may have expired, refresh it once outside the retry loop, then retry once. Otherwise treat it as permanent. |
| 403 Forbidden | Usually permanent | Check permissions and configuration. APIs may use 403 for throttling. |
| 404 Not Found | Conditional | Retry briefly only if eventual consistency or delayed creation is expected. |
| 408 Request Timeout | Transient or ambiguous | Retry only when the provider's semantics and replay safety make it safe. |
| 409 Conflict | Conditional | Inspect the conflict. It may require refetching or resolving a version mismatch. |
| 422 Unprocessable Content | Usually permanent | Correct the data. |
| 429 Too Many Requests | Usually transient | Honor Retry-After when present. Provider-specific quotas may need deferral instead. |
| 500 Internal Server Error | Often ambiguous | Retry only if safe to repeat. The response alone may not prove a write had no effect. |
| 502 Bad Gateway | Usually transient | Retry with backoff if the operation is safe to repeat. |
| 503 Service Unavailable | Usually transient | Retry with backoff. Honor Retry-After when present. |
| 504 Gateway Timeout | Transient or ambiguous | Check replay safety before retrying. |
These are only starting points, since each provider has its own implementation. Some APIs will return a 200 with an error in the body, signal throttle with a 403, or throw a 500 for problems no retry will fix. The provider's implementation always wins, so encode each vendor's rules in one classifier rather than scattering them through your flow logic. Our backoff post includes provider-specific throttling examples.
The same is also true for issues outside of HTTP. DNS hiccups, connection resets, and socket timeouts may be transient. Invalid hostnames, TLS certificate errors, and unsupported protocol settings usually require a config or environment fix before the request will be successful.
How do I build a bounded retry policy?
Once you’ve identified a failure as retryable and safe to repeat, three controls are essential: an attempt limit, an exponential backoff with jitter, and a time budget.
1. Cap the attempts and store the count
A hard ceiling for attempts is the simplest guard against infinite loops. For example, five total attempts is one initial request and four retries. Be explicit since some libraries define maxRetries as extra attempts, while others define them as total requests.
Store the count somewhere that survives a restart, such as the job record or the message's delivery count. Rescheduled work must carry its history (attempt count, first-attempt time, and deadline) because rescheduling a request is not starting a new one. Otherwise, a request retries five times, re-enqueues itself, retries five more, and the "max 5 attempts" rule isn't a rule at all.
2. Back it off with jitter
Each retry should wait longer than the last, up to a ceiling (for example, 1, 2, 4, 8, 16, then 30 seconds), with randomness added so a thousand executions that failed together don't retry together. Our backoff post covers the formulas, jitter variants, and how to read each provider's rate-limit signals.
3. Set a deadline in addition to the count
Attempt limits alone can still take quite a while to process. Five attempts with a 60-second request timeout can take several minutes waiting before backoff is even counted. Give every operation a total time budget as well:
| Setting | Example value |
|---|---|
| Maximum attempts | 5 total |
| Base backoff | 1 second |
| Maximum backoff | 30 seconds |
| Per-request timeout | 15 seconds |
| Maximum elapsed time | 120 seconds |
Stop as soon as either the number of attempts or the time budget runs out. Don't start a request when the remaining budget can't cover its timeout, and restrict in-flight requests by the overall deadline. An interactive request may need a budget of a few seconds, while a nightly batch sync can afford much longer. Regardless of how you do it, every retryable operation must have a fixed budget.
Let the provider's wait work
When a 429 or 503 carries a Retry-After header, make that the minimum delay and don’t shorten it with your own backoff. Retry-After can be either a number of seconds or an HTTP date, so parse the value carefully. And, if that wait exceeds the operation's remaining budget, don't retry early and don't keep the operation spinning. Instead, persist the work and reschedule it with a fresh budget.
How do I make retries safe when a request might have succeeded?
A retry is considered safe when running the request twice has the same intended effect as doing it once, or when the provider notes that a specific failed response means the request did not complete (and that’s the response which was received).
The best way to ensure that retries are safe is to use idempotency keys (one per logical operation), upserts against a stable external ID, and deduplication of inbound events. Our post on making webhook processing idempotent covers those in detail.
If none of these are in place, don't automatically retry after ambiguous outcomes. Instead, check the target system for the record before retrying, and if you still can't tell, send the operation to someone to manually review. Treat it as requiring reconciliation and not as a permanent failure since the operation may have succeeded. The time it takes a person to resolve one stuck order costs far less than an automated retry that creates a duplicate shipment.
What does a retry policy look like in TypeScript?
The policy should have a classifier and a loop that enforces an attempt limit, a deadline, jittered backoff, and Retry-After as a floor. The caller supplies provider-aware checks for transport errors and replay safety since a generic TypeError check is not enough to identify network failures reliably.
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283
This is example code and not a drop-in HTTP client. isTransportError must be implemented for the runtime and client in use. And you should’t classify every TypeError as a network error, because malformed URLs and invalid request options can also cause them. canRepeat should reflect endpoint-specific semantics as well as idempotency.
The example handles both standard Retry-After formats, but production code should use a strict HTTP-date parser, validate policy values, persist attempt history with the job, and add a bounded 401 token refresh (if needed). A 401 refresh is deliberately not part of this loop: the default classifier treats 401 as permanent unless your flow handles one refresh separately.
PermanentError means the request needs a data or configuration fix. ReconciliationRequiredError means the outcome should be checked before another attempt. RetryExhaustedError means the retry budget ran out on an operation that was safe to repeat. In this case, the caller should either reschedule it under a separate budget or stop.
What should happen when retries are exhausted?
It’s expected that some operations won’t succeed. However, "stop retrying" should never mean that you simply drop the data. Instead, when a failure is permanent or a budget runs out, move the operation into a defined terminal state.
- Record it with context. Capture the operation ID, customer or tenant, endpoint, error class, attempt count, timestamps, and the last response. Don't include any credentials, tokens, or sensitive payload fields.
- Park it in a dead-letter queue. That queue holds work that needs a data or config fix before it can resume. Reprocessing from it has its own limits, but it should keep the operation's identity and history so it doesn't trigger an unlimited cycle.
- Alert in proportion to the impact. One record rejected for bad data may belong in a support queue or a customer-facing validation message. A bunch of exhausted retries across hundreds of customers suggests a provider outage or an integration defect, and deserves a high-priority alert. Retries that eventually succeed are common and probably shouldn't trigger any alert.
- Tell the customer when it's their issue. Revoked access, a deleted record, or a missing required field in their system will be resolved the quickest when the customer receives a clear error message.
- Make recovery deliberate. Once the underlying issue is addressed, replay the parked work after determining whether any part of the original operation succeeded. A replay should be a controlled recovery action, not a way to loop indefinitely.
Where do circuit breakers fit?
A retry policy decides whether to try one operation again but a circuit breaker decides whether to call a failing dependency at all. Wrap the circuit breaker around the whole retry operation, so a call that exhausts its budget counts as one failure, and never retry a call that the circuit breaker has rejected. Our post on implementing circuit breakers covers breaker states, multi-tenant scoping, and more.
How do I know the policy is working?
Successful requests don’t exercise your retry logic, so you’ll need to test probable failures on purpose with mocked responses and fault injections:
| Scenario | Expected behavior |
|---|---|
| 503, then success | Retry and complete |
| Repeated 503s | Stop when the budget is exhausted |
| 400 or 422 | Fail immediately, no retry |
| 401 with an expired token | One bounded credential refresh |
| Timeout on a non-idempotent write | Reconcile without a blind retry |
| API down for hours | Stop or reschedule within the overall budget |
| Worker restarts mid-retry | Resume with the stored count and deadline |
| Retry-After is an HTTP date | Wait until the indicated time, then add jitter |
| Malformed request URL | Fail as a request/configuration error, not a transient network retry |
Then test the failure path itself. Does the request land in the dead-letter queue, does the alert fire, and can the operation be safely replayed?
Watch for retry patterns that can hide issues. A dashboard can show 99.9% success while 20% of executions required multiple retries before succeeding. To make sure nothing is missed, track the following per integration, customer, and provider:
- Retry rate and attempts per operation – How much extra traffic are retries creating?
- Recovery and exhaustion rates – Are retries fixing transient failures.
- Errors by class – For example, a spike in 422s after a deploy is probably a mapping bug that a retry will never fix.
- Dead-letter queue and age – How much failed work is waiting, and how long has it been there?
Where Prismatic fits
Prismatic gives you retries at several layers: step-level retries, flow-level automatic execution retry, and the Spectral HTTP client's retry configuration.
Here's how to apply the classification and stopping rules appropriately.
Give each failure type a single owner
Your code is rarely the only part that retries. For example, an HTTP client set to three attempts, inside a step that's set to three attempts, inside a flow retry that's set to three attempts can turn one operation into 3 × 3 × 3 = 27 requests. A webhook sender's redeliveries can multiply that again. The best approach is to let the HTTP client or step absorb transient issues, and reserve the flow-level automatic retries for longer outages in flows whose writes are safe to repeat. For configured asynchronous executions, Prismatic supports up to 10 retries after a failure, and flow-level retries start the flow again from its first step.
Classify before a retry is triggered
A step's retry handler acts on any error the step throws, and Spectral's retryAllErrors function can retry client errors too. In custom code, use a retryCondition or throw only for failures the provider-specific classifier has determined are safely retryable. For permanent failures, throw a detailed error on a step with no retry configured, or return the error as data so the flow can route it.
Retry batches, not whole syncs
Prismatic’s large data sync supports opt-in automatic retries for failed batches when retry is enabled in the flow's settings. A failed batch reruns as a whole with its original payload, up to three retries. And each retry links to the original batch in the execution view and API. This is not a per-record retry or a manual "retry batch" action from that view, so per-record idempotency still matters. Check out our large data sync docs.
Retry policy checklist
Before you ship an integration, its retry policy should have an answer for each of the following:
- Does each error pass through a classifier that labels it transient, permanent, or ambiguous, using provider-specific rules?
- Do validation, authorization, and not-found errors fail fast unless there's a documented exception?
- Are ambiguous failures retried only when the operation is safe to repeat, but reconciled when it's not?
- Is the attempt count stored where it survives restarts and rescheduling?
- Is there a budget that honors provider waits?
- Does each failure type have one retry layer that owns it?
- Do permanent and exhausted failures land in a dead-letter queue with full context?
- Can someone replay permanent and exhausted failures safely, in bulk, once the issue is fixed?
- Have you tested the failure paths, and do you track retry, recovery, and exhaustion rates separately?
Build a retry policy that knows when to stop
Retries are essential. Third-party APIs go down, networks drop connections, and rate limits are necessary to keep things sane when you are moving data between systems at scale. However, retrying every failure isn't necessary or right.
A properly-designed retry policy separates failures that retries can fix from those that need intervention. It does this by making repeated operations harmless, spacing out attempts, and enforcing limits so an issue doesn't become an endless loop. Just as importantly, it defines what happens when retries don't work.
Build the right retry policy and your integrations will retry automatically when recovery is possible, stop when it isn't, and provide your team with the info it needs to fix the rest.
Start a free trial to see how Prismatic provides a solid retry framework out of the box.

