How Do I Make Webhook Processing Idempotent

Key takeaways

Webhook providers deliver at least once by design. A soft timeout (5.1 seconds against a 5.0 limit), a lost 200 response, or two near-simultaneous retries can each deliver the same event twice. GitHub is the exception: it doesn't automatically redeliver.
Idempotency for webhooks consists of delivery deduplication and idempotent business logic. You need both. Deduplication alone fails if a process crashes halfway through an event already marked seen, and idempotent logic alone repeats expensive work on every retry.
A check-then-process deduplication has a race condition. Two copies of an event can both pass the "have I seen this?" check before either records it. Claiming the event with a unique-key insert (INSERT ... ON CONFLICT DO NOTHING) lets the datastore arbitrate.
Marking an event as processed before you should ignores failed events. Track processing, completed, and failed as separate states, and reclaim stuck processing events with a lease or fencing strategy so a slow worker and a recovery worker don't both act.
Absolute state changes are much better for retry safety. SET status = 'paid' is idempotent and balance += amount is not. Keep the dedup window longer than the provider's retry window, because a 1-hour TTL against a 72-hour retry period protects only the first hour.

Many integrations eventually handle the same webhook twice. Not because you did anything wrong, and not because the provider is broken. It's how the system works, and ignoring that reality is how you end up debugging a support ticket titled "customer charged twice" at 4:45 PM on a Friday.

Let's talk about why this happens, and how to stop it from happening to you.

Why webhook retries can deliver the same event twice

By design, many providers retry failed webhook deliveries, but retry counts, time windows, and ordering guarantees vary. The sender doesn't know whether your database saved the record. It generally treats a timely success response as delivery success, but it cannot know whether your downstream database or API committed the operation. If it didn't get that confirmation, it assumes the worst and tries again. This is correct behavior for them, because the alternative (dropping an event because your server choked) is worse.

GitHub is an exception here. It does not automatically re-deliver failed webhook deliveries. You can do so manually, if needed.

This "at least once" functionality tends to show up in a few flavors:

The soft timeout – Your endpoint receives the payload and starts doing work – writing to a database or calling an API. It takes 5.1 seconds. The sender's timeout is 5.0. They drop the connection, mark it a failure, and queue a retry. Your code finishes cleanly a second later, completely unaware anyone gave up on it.
The "success" that isn’t heard – You process the event, write the record, and return a 200. The response gets lost on the way back through a network hiccup or something. As far as the sender knows, you never answered. Retry incoming.
The near-simultaneous retry – A retry lands while the original request is still processing, or multiple delivery attempts reach different workers close enough together that both begin processing before either records the event as complete.

123
[Sender] POST /webhook --> [Your API]  (processing... 5.1s)
[Sender] <-- timeout (5.0s) -- [Your API]  (saves to DB)
[Sender] -- retry POST --> [Your API]  (saves to DB again)

If your handler treats every incoming POST as a brand-new, never-before-seen event, that’s common. As a result:

A payment capture may be submitted twice.
An order may be created twice in an ERP.
A notification may go out to a customer team a second time.

None of that happens because retries are bad. It happens because "process the event" wasn't safe to run more than once.

The answer is to make your handler not care how many times it is called with the same event.

Two problems are wrapped together here

"Idempotency" is one word, but it's two separate problems:

Delivery deduplication – Have I seen this event?
Business logic idempotency – If I run this operation again, does anything change?

You need both. Deduplication alone doesn't save you if your process crashes halfway through an event you've already marked "seen." And idempotent business logic alone doesn't save you from doing a bunch of redundant, possibly expensive work every time a retry rolls in. Take them one at a time.

The definition of idempotency

Regardless of how you pronounce this term (there are several common ways), idempotency means that performing the same operation multiple times has the same effect as performing it once. For webhooks and integrations, this matters because deliveries are rarely guaranteed to happen only once. An idempotent integration recognizes duplicates, typically by tracking an idempotency key, and ensures that a repeated "payment succeeded" or "order created" event doesn't charge a customer twice, create duplicate records, or trigger the same action again. The result is a system that can safely retry failed requests without fear of unintended side effects.

Step 1 – Pick an idempotency key

You need something stable that identifies this specific event, not this specific HTTP request. Use the provider’s documented stable event identity, and namespace it by provider, tenant, subscription, and event type where necessary.

Only hash the payload as a last resort, when there's no ID to key off of. Hashing has issues. If the provider stuffs a timestamp or retry counter into the body, every retry hashes differently, and your deduplication function does nothing. A payload hash only detects byte-identical duplicates; it can miss logically identical events whose timestamps, retry counters, or serialization differ. The underlying question here isn't, "Do these two payloads look identical?" it's, "Does the provider consider these the same event?" Those may not result in the same answer.

Step 2 – Claim the event atomically, before you act on it

The simple version looks like: check if I've seen this ID, and if not, process it.

1234
if (!(await alreadyProcessed(idempotencyKey))) {
  await processEvent(event);
  await markProcessed(idempotencyKey);
}

This has a race condition sitting right in the middle of it. Two copies of the same webhook can arrive close enough together that both pass the "have I seen this?" check before either one finishes recording it.

Congratulations, your idempotency system has faithfully recorded that you created two orders.

The fix is to make "check" and "claim" a single atomic operation and let the datastore arbitrate the collision, instead of doing it yourself with two round trips:

12345678910
CREATE TABLE webhook_events (
  idempotency_key TEXT PRIMARY KEY,
  status TEXT NOT NULL,
  received_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

INSERT INTO webhook_events (idempotency_key, status)
VALUES ($1, 'processing')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING idempotency_key;

Only the request that inserts a row gets to continue. Everyone else knows another request already claimed the event and can stop competing for it. What happens to an event that gets claimed but never finishes is a separate problem, which we'll get to next.

Step 3 – Don't mark it complete before it's complete

If you mark the event processed before doing the work:

1
receive webhook → mark processed → create destination record

...and the destination call fails, you've now recorded something that didn't happen. The retry shows up, sees idempotencyKey already marked processed, and proceeds to ignore an event that never finished.

You'll also need a recovery policy for events left in processing when a worker dies, such as reclaiming them after a timeout. Reclaiming a stuck processing record requires a lease or fencing strategy; otherwise, a slow original worker and a recovery worker could both act.

Now "I've seen this event" and "I've successfully finished this event" are two different facts, which they need to be, because you don't control the network, the provider, or the API. If you POST /orders and the connection dies before the response comes back, you can't know whether the order got created. That's not a you-problem to solve with more logging – it's the shape of a distributed system, and the answer is to design for it instead of attempting to eliminate it.

Step 4 – Make the business logic itself idempotent

This step sometimes gets overlooked because step 2 feels like it solved the problem. However, that step solves "Don't process the same event twice," not "What if the operation itself isn't safe to repeat?"

Prefer absolute state changes over deltas. SET status = 'paid' is idempotent. balance += amount is not. Run it twice, and you've invented money, which is kinda fun until finance calls you.
Upsert instead of insert when the downstream system has the right uniqueness constraint and update semantics.
Pass idempotency keys to the APIs you call. If the destination API supports its own Idempotency-Key header, use a key derived from the event identity and the specific downstream operation, subject to that API’s idempotency semantics. This covers a failure mode your own dedup can't: the outbound request succeeds on the far end, but you never get the response back (so from your side it looks like it failed). Without a downstream key, a retry can create a second record even though your delivery-level dedup worked as designed.

Step 5 – Acknowledge fast, and remember for long enough

Return a 2xx once the event has been durably claimed and queued for processing. Use a transactional outbox or atomic queue operation. If that isn’t possible, add recovery that detects claimed-but-not-enqueued events. Don't wait until all downstream work finishes. For providers that use 2xx acknowledgments, returning one usually stops automatic retries – but retry semantics are provider-specific.

Don't guess at how long to remember an event. Check the provider's retry window. If the provider retries for 72 hours, a 1-hour TTL on your dedup store protects you for exactly the first hour and then stops protecting you. Also consider whether events can be manually replayed well outside the normal retry window (many providers let customers do this from a dashboard), and weigh retention against the cost of a duplicate: an extra marketing email is annoying, a second invoice is more likely to create an unhappy customer.

This pattern is well-worn, and if you've built more than one webhook for an integration, you've probably reinvented some version of it already. The hard part is implementing and maintaining it correctly across every provider, especially once "every provider" also means "every customer," each with its own definition for "unique" and "retry."

Where Prismatic fits in

A lot of the above functionality gets easier, but none of it goes away entirely with Prismatic. It probably shouldn’t, because what counts as safe to repeat depends on the API you're integrating with, and no platform is going to consistently guess that for you.

Prismatic's webhook-triggered flows run asynchronously and can be processed concurrently by default, which means an integration built without any of the above still has the same race conditions it would anywhere else. If ordering matters, or you want to use Prismatic's built-in short-window deduplication, you can put the flow behind a FIFO queue and set a Deduplication ID in the trigger's Flow Control config. Point it at the stable deduplication key you derive from the provider’s documented event semantics, and Prismatic will ignore duplicates of that ID within a 10-minute window without you having to stand up your own store.

But that's not a full replacement for longer or more deliberate deduplication. For that, Prismatic gives you Flow State and Cross-Flow State to persist processed IDs, cursors, or checkpoints between executions: Flow State is scoped to a single flow and Cross-Flow State is shared across flows for the same customer instance. Persisted state on its own doesn't give you the atomic claim from Step 2. Since Prismatic explicitly supports concurrent execution, a simple "read the ID, then write the ID" against that state has the same race we walked through above. If you need a longer window and parallel processing, back it with a store that gives you the atomic guarantee you need, rather than assuming a key-value check solved it.

Prismatic can automatically retry failed asynchronous executions when retry is configured, and supports manually replaying an execution with its original webhook payload – which is another argument for Step 4.

Please remember that retries and replays are only helpful tools if what they're re-running is safe to re-run. Prismatic can run and retry executions, persist configured state, and provide execution logs so you can see what happened to a given delivery. Deciding what the idempotency key is, and what "successfully processed" means for the API – that part's still yours. And it should be, since it requires a proper engineering answer, which varies greatly from API to API.

Make the second delivery smooth

Retries might seem to be the issue here, but they aren't. Without them, a network blip can turn into lost data. It’s good business to have providers (and your own infrastructure) retrying after transient failures. The mistake is building an integration that only expects the first attempt to show up.

Assume that the event arrives twice. Assume that two copies can arrive at the same instant. Assume that the API call can succeed without you ever finding out. Assume that your process can die at the single most inconvenient moment possible. Then build the function so another attempt gets you back to the right answer.

The first delivery does the work. The second one? It should be absolutely boring.

How Do I Make Webhook Processing Idempotent So Retries Don't Create Duplicates?