- Classify the provider’s response before timing the retry. A status code alone misleads. A provider-specific classifier decides whether to retry, which kind of limit applies, and what scope it covers.
- Backoff without jitter creates a delayed thundering herd effect. Workers that fail together retry together on the same schedule. Full jitter picks a random delay between zero and the exponential ceiling, and it’s a strong default when reducing load is the priority.
- Provider wait hints set the minimum delay. Retry-After is either seconds or an HTTP date, and reset headers vary by provider, with X-RateLimit-Reset a Unix timestamp in one API and a duration in another.
- Workers coordinate backoff at the limit’s actual scope. Record a not_before timestamp per provider, customer, and rate-limit key, because scopes vary (app plus store for Shopify, method plus workspace for Slack). Burst, quota, concurrency, and object-contention limits each need different handling.
- Retries need a budget and idempotent writes. Cap attempts, total retry duration, and queue age, and add a circuit breaker for persistent throttling. A timed-out POST /orders is ambiguous, so use an idempotency key rather than a blind retry.
A large customer onboards, someone starts a historical backfill, and a bunch of workers begin calling the same API. Every worker gets a 429, waits 1 second, then 2, then 4, in perfect formation. The API stays unhappy, the queue grows, and the support ticket is titled "Sync stuck since Tuesday."
The problem isn't the retry, but the retry approach. You are using a generic retry strategy with providers that have their own ways of saying "not yet."
Backoff solves half the problem
Exponential backoff increases the wait between attempts:
1
That is better than a tight retry loop, but it does not desynchronize clients. If every worker fails together and uses the same deterministic formula, they retry together too:
123
Backoff without jitter gives you a thundering herd effect – with a delay. Jitter adds randomness so clients spread out. Full jitter chooses a random delay between zero and the exponential ceiling; equal jitter keeps half the delay fixed; de-correlated jitter bases the next delay on the previous one. Full jitter is a strong default when reducing load is the priority. The AWS Architecture blog compares the trade-offs.
There are two separate questions:
- Interpretation: What does this provider's response mean, and should we retry?
- Timing: If we should retry, how long should we wait, and how do we avoid synchronized retries?
Interpretation comes first. Perfect jitter with the wrong interpretation will retry an exhausted daily quota every 30 seconds until midnight.
1. Classify the response first
Don't let the HTTP status code make the entire decision. A 429 commonly means "too many requests," but some providers use 403 or even a successful HTTP response with a throttle error in the body. Conversely, many 403s and 400s are permanent failures.
Use a provider-specific classifier that returns a retry plan:
1234567891011121314
The classifier should decide:
- Whether the response is retryable at all
- Whether the limit is a burst, quota, concurrency, or object-contention problem
- What scope is affected: credential, app, tenant, store, workspace, method, endpoint, or object
- Whether the provider supplied a minimum wait
Validation, authentication, and permission errors generally should not be retried. A timeout on a write is different: you don't know whether the server processed it, so it needs an idempotency strategy rather than a blind retry.
2. Learn each provider's dialect
Parse Retry-After correctly
HTTP defines Retry-After as either seconds or an HTTP date. RFC 6585 says a 429 may include it; it is not guaranteed to be present.
123456789101112131415
The HTTP-date form depends on client/server clock sync; the seconds form doesn't. Also parse reset headers per provider. X-RateLimit-Reset may be a Unix timestamp for one API and a duration for another.
Common provider patterns
Note: These patterns were accurate when I wrote this post, but providers regularly change what they do. Verify the information before using it in production.
| Throttle behavior | Guidance | |
|---|---|---|
| GitHub | Primary limits return 403 or 429 with x-ratelimit-remaining: 0; secondary limits may do the same | Use x-ratelimit-reset for primary limits. For secondary limits, honor Retry-After; otherwise wait at least one minute. A 403 without rate-limit signals may be a permission error. |
| Salesforce | 403 with REQUEST_LIMIT_EXCEEDED for some enforced limits | This can mean a concurrent-request limit, not only daily quota exhaustion. Check Sforce-Limit-Info and the Limits resource. Daily usage is measured over the preceding 24 hours. |
| Shopify REST | 429 | Honor Retry-After and X-Shopify-Shop-Api-Call-Limit. The bucket is scoped to app plus store. |
| Shopify GraphQL | Often HTTP 200 with a THROTTLED error | Inspect the body and extensions.cost.throttleStatus; HTTP 200 does not mean the operation succeeded. |
| Stripe | 429 | Stripe-Should-Retry is a retry/no-retry signal, not a duration. Also distinguish request-rate limits from object lock contention. |
| Microsoft Graph | Generally 429 for throttling | Honor Retry-After. Some service-specific workloads may use other transient statuses, so don't classify every 503/504 as throttling. |
| Slack | 429 | Retry-After is in seconds; limits are evaluated per method and workspace. |
| Hubspot | 429 | The body identifies the policy. Distinguish the per-10-second burst limit from the account-shared daily limit, which resets at midnight in the account's time zone. |
The exact rate-limit key matters. "Credential" is sometimes a useful shorthand, but providers may scope limits to an authenticated user, installation, app/store pair, app/tenant pair, method/workspace pair, or even an individual object.
3. Use provider hints as floors, then add jitter
The delay calculation should follow this order:
- A wait-bearing provider hint sets the minimum wait:
Retry-After, a reset timestamp, or a computed refill time. - A boolean signal such as
Stripe-Should-Retrydecides whether retrying is allowed but does not set a wait. - If no wait hint exists, use your own exponential backoff with full jitter.
- Cap your own inline wait, but never clamp a provider's minimum downward.
12345678910111213141516171819202122232425262728293031
A malformed header can cause you to fall back to normal backoff. A valid but very long wait should become a deferred job, not a shorter retry. If the provider's wait exceeds the operation's deadline, defer or fail deliberately – never retry early because your local deadline is shorter.
4. Coordinate at the limit's scope
If you track backoff per request, one worker may wait while the others keep sending traffic. Record a not_before timestamp at the affected rate-limit key and have every worker check it before sending:
12
A read-compare-write sequence can race. Two workers may read the same old value and overwrite each other. That may produce one extra 429, but if tighter coordination matters, use an atomic "set if later" operation.
Different limit types need different handling:
- Burst limits: wait briefly and retry.
- Quotas: park and reschedule the work; don't hold a worker for hours.
- Concurrency limits: use a semaphore or bounded worker pool.
- Object contention: back off only the affected object.
- Cost limits: reduce query cost as well as request frequency.
The best 429 is the one you never receive. Use client-side pacing, bounded concurrency, caching, conditional requests, and provider budget signals to avoid sending traffic you already know will be rejected.
5. Give retries a budget and make them safe
Set limits on:
- Max attempts per operation
- Total retry duration
- Queue age
- A client-wide retry budget
- A circuit breaker or admission pause for persistent throttling
When the budget runs out, reschedule, fail, or alert. A retry loop that runs forever is not resilience; it is delayed failure.
Also make the operation safe to repeat. A GET is usually straightforward. A POST /orders that timed out is ambiguous because the server may have created the order before the connection failed. Use an idempotency key when the provider supports one.
A 429 is often safer to retry than a timeout because the provider rejected the request for rate limiting. But don't assume that every provider guarantees no side effects. Automatically retry a non-idempotent write only when the provider documents that the request was not processed or when your idempotency design makes repetition safe.
6. Test and log the decision
Use mocked responses and a controllable clock. Test at least:
| Scenario | Expected behavior |
|---|---|
| 429 with Retry-After: 30 | Waits at least 30 seconds, plus jitter |
| HTTP-date Retry-After | Waits until the date, with clock skew handled |
| Malformed header | Falls back to normal backoff |
| Valid long wait | Defers or fails; never retries early |
| Provider-specific reset | Interprets units correctly |
| Throttle in a 403 or 200 body | Classified as throttling, not permission or success |
| Shared quota | Workers respect one not_before value |
| Ambiguous write timeout | Uses an idempotency strategy |
| Retry budget exhausted | Retry budget exhausted |
Inject a deterministic random source in tests, or assert that delays fall within the expected range. A test that expects a random delay to be exactly 2,000 milliseconds is a test that fails on Tuesdays.
Log the provider, endpoint, rate-limit key, attempt, delay, delay source, and remaining budget. "Request failed with 429" is a useless support ticket. "Salesforce daily API quota exhausted for customer Acme; rescheduled for the next available window" closes itself. Keep credentials and sensitive payloads out of the logs.
Where Prismatic fits in
Prismatic provides several building blocks, but you decide what a provider-specific throttle means.
Step-level retries
In the low-code integration designer, a step can wait a set number of seconds, retry up to a maximum number of attempts, and optionally use exponential backoff. If the step ignores an error, its result includes an error property that you can branch on.
This is useful for transient failures, but it is not documented as an automatic Retry-After reader. For providers that specify an exact wait, read that signal in code or route the work to a rescheduling path.
The Spectral HTTP client
Spectral HTTP client documentation describes retryConfig options including retries, retryDelay, useExponentialBackoff, retryAllErrors, and the retryCondition.
For 429 handling, a custom retryCondition is usually safer than retryAllErrors, which can include 400- and 403-class errors. A classifier is still necessary for non-obvious throttle signals such as GitHub's 403 or Shopify GraphQL's 200 response.
Flow-level retry
Automatic Execution Retry supports up to 10 retry attempts, minute-level intervals, exponential backoff, a 24-hour maximum delay, and cancellation of older retries when a newer invocation has the same unique request ID.
Request retries and execution retries are different tools. A flow-level retry starts the flow again from its first step, so earlier writes must be idempotent. Use flow-level retry for minutes- or hours-long recovery, not for a per-second provider hint.
Queues and shared state
Webhook-triggered flows can run concurrently. A FIFO queue can process executions in order and reduce bursts, but it is not a substitute for a provider-aware rate limiter.
Flow State is scoped to one flow; Cross-Flow State is shared across flows within an instance. Prismatic's state documentation notes that persisted state uses last-write-wins behavior under concurrent execution and is written when execution completes successfully. A not_before timestamp can help coordinate backoff, but it is not an atomic distributed limiter. Use an external store when atomic coordination is required.
Make the right assumptions
Rate limits are how providers keep their APIs available. Many teams make the mistake of building an integration with simple throttling and thinking that'll be enough. But things are rarely that straightforward.
Instead, assume the throttle may come back as a 403 or a 200. Assume the provider may give you a header you've never seen before. Assume a backfill may exhaust a daily quota before lunch. Then classify the response, honor the provider's floor, add jitter, coordinate workers at the right scope, and reschedule work that cannot succeed soon.
The first 429 tells you to slow down. And that's essential information. From there, you should take charge of what slowing down means and how to do it most effectively, given the full provider context.
