Idempotency Keys Passed Code Review in 9 of 12 Payment Systems. All Nine Still Duplicated Writes.

An idempotency key is not the whole system

Every payment API tutorial says the same thing: generate a unique key per logical operation, send it in a header, check for it before processing, done. It reads like a five-minute addition. That is exactly the problem. Teams treat an idempotency key as a checkbox, when what they actually need is a small state machine with its own crash-recovery story.

An audit of twelve production payment platforms, published on DZone in 2026, found that idempotency logic which had passed code review and survived unit testing still let duplicate writes through in nine of the twelve systems. None of the nine were missing an idempotency key. All nine were missing at least one of the states that key needs to track.

The six failure modes below are not hypothetical. They come from that DZone audit and a separate catalogue of recurring failures published on Java Code Geeks, and they keep showing up under different names at different companies because they are properties of distributed systems, not bugs in any one codebase.

Failure mode 1: the check-then-write race

The most common implementation checks for an existing key, and if none is found, proceeds to process the request before writing the key back. Two requests carrying the same key, arriving milliseconds apart, both run that check before either has written anything. Both find nothing. Both proceed. In a payment system, that is two charges for one purchase.

This race is easy to reproduce with retries: a client-side timeout that fires a second request while the first is still in flight, a load balancer that retries on a slow response, or a mobile app that resends a tap the user already made. None of these require unusual load. They require two requests close enough in time that a read-then-write sequence has room to interleave.

-- Wrong: read, then write. Not atomic.
SELECT * FROM idempotency_keys WHERE key = $1;
-- if not found here, proceed to charge, then:
INSERT INTO idempotency_keys (key, status) VALUES ($1, 'completed');

-- Right: attempt the write first, let the constraint decide.
INSERT INTO idempotency_keys (key, status, created_at)
VALUES ($1, 'in_progress', now())
ON CONFLICT (key) DO NOTHING
RETURNING key;
-- zero rows returned means another request already owns this key

The fix is to make the write itself the source of truth: attempt an INSERT protected by a UNIQUE constraint on the key column, and treat a conflict as the duplicate signal rather than an error to recover from. The database is doing the concurrency control that application code cannot do reliably on its own.

Failure mode 2: the stuck "in progress" record

A request writes its key as in_progress, begins processing, then the process dies before it finishes, most often during a rolling deployment when a pod is evicted mid-request. The key now exists forever in a state that is neither missing nor complete, and every retry against it has no correct action: reprocess and risk duplicating whatever side effects already ran, or wait and risk the operation never finishing.

Two states, in_progress and completed, cannot represent this correctly. The fix is a third state: in_progress with a short TTL (30 seconds is a reasonable starting point for most HTTP operations), completed, and failed, backed by a separate scheduled job that expires stale in_progress records so they can be safely reclaimed and retried. This state needs to live in a durable store, not only a cache, since losing it on an eviction recreates the exact bug it exists to prevent.

Failure mode 3: TTL and clock skew

A TTL on an idempotency key is a bet on how long a duplicate might realistically arrive after the original request. When that assumption is wrong, the key expires before the duplicate shows up, and the system reprocesses it as new. In one documented case, a 24-hour TTL was sized around normal delivery times, and a Kafka consumer fell 26 hours behind during a partition rebalance. Every message replayed after that point found an expired key and was processed a second time.

Clock skew compounds this in a quieter way. NTP drift between application servers and a store like Redis is normally tens to hundreds of milliseconds, and can reach seconds during VM migration. That means the application’s view of "still valid" and the store’s view of "expired" can disagree right at the boundary, which is precisely when a retry is most likely to land, since retries are usually triggered by a timeout close to that same boundary.

The fix has two parts: build in an explicit skew buffer instead of trusting timestamps compared across two clocks, and delegate expiry to the store’s own atomic operations (an atomic SET with an expiry flag, for instance) rather than checking a stored timestamp in application code. Size the TTL to the longest plausible delivery delay in your own system, not a default copied from someone else’s blog post.

Failure mode 4: the region that does not know yet

Active-active deployments replicate the idempotency key store across regions asynchronously, which introduces a lag of roughly 10 to 200 milliseconds under normal conditions, and considerably more during failover or network congestion. A retry that lands on the other region right after the original request can find no key at all, because replication has not caught up, and reprocess a request the first region already completed.

Two fixes address this at different layers. Sticky, consistent-hash routing at the load balancer keeps retries for the same key on the same region, which sidesteps the replication lag entirely. Alternatively, moving specifically the idempotency key store to synchronous replication, even when the rest of the stack stays eventually consistent, closes the gap directly. Stripe’s own public documentation describes exactly this trade-off: synchronous replication for the key store itself, paired with a 24-hour retention window for client convenience.

Failure mode 5: the key was not as unique as it looked

The last failure mode is upstream of all the others: the key itself. Sequential or millisecond-timestamp identifiers generated independently on multiple application instances can collide under load. Hashing the request payload to derive a key leaks whatever is in that payload, breaks the moment a legitimate retry carries a slightly different body, and introduces its own collision risk. And UUIDs are not automatically safe either: cloned VM images with poorly seeded random number generators have produced two instances generating identical sequences of "random" identifiers.

UUID v7 avoids most of this. It combines a millisecond-precision timestamp with enough random bits to stay collision-resistant across instances, and the time-ordering improves index locality as a side benefit. Generate the key client-side or at the edge of your system, and never derive it from the payload.

Failure mode 6: idempotency that stops at the front door

The first five failure modes assume the operation you are protecting is a single write. In practice, a request that gets past your idempotency check often triggers work further downstream: a webhook fired to a customer’s system, an event published to a queue for other services to consume, a notification email sent through a third-party API. Each of those downstream steps can have its own retry logic, and none of them automatically inherit the dedup guarantee the front-door key provided.

This is the layer mismatch documented in the DZone audit: API-level idempotency correctly prevents a second charge, but the event the first request published to process that charge asynchronously gets redelivered by the queue’s own retry policy, and a downstream consumer with no deduplication of its own processes it twice. The front door was never broken. The hallway behind it was.

The fix is to stop treating idempotency as a single gate and start treating it as a property that has to hold at every hop capable of an independent retry: the API request, the event publish, and each consumer that reads that event. A consumer-side dedup table keyed on the event ID, checked with the same atomic-insert pattern as the API layer, closes the gap that the front-door key was never positioned to cover.

A checklist, not a feature flag

None of these six failure modes are exotic. They are the specific, recurring ways a well-intentioned idempotency key implementation still lets duplicates through, and each one has a narrow, testable fix.

What to test before you trust it

Unit tests do not catch most of this, because these are timing and infrastructure failures, not logic errors. A single test run on a single machine cannot reproduce a check-then-write race, a mid-request crash, or replication lag between two regions.

• Fire concurrent, identical requests in a load test and assert exactly one side effect occurred, not just that both requests returned success.
• Kill the process deliberately mid-request in a chaos test, then confirm the key resolves correctly on the next retry instead of hanging.
• Simulate clock skew directly between the application and the key store rather than assuming NTP keeps them in sync.
• Test a retry that is deliberately routed to a different region than the original request, and confirm it does not reprocess.
• Force a queue to redeliver an already-processed event and confirm the downstream consumer, not just the original API, rejects the duplicate.

None of these six fixes require a new architecture. They require treating the idempotency key as what it actually is: a small state machine with its own crash-recovery story, applied at every hop that can retry independently, and tested the same way the rest of the system is tested for failure, not just for correctness.