Idempotency Keys for Agent Tool Calls Need a Third State: "We Don't Know"
A reserve, execute, record ledger is the easy half. The crash between execute and record is where agents double-send.
An agent that books a courier, sends an invoice or issues a refund will eventually run a tool twice. The usual advice is sound as far as it goes: derive an idempotency key, store it in a ledger, and replay the stored result on retry.
Several practitioner write-ups cover the basic ledger well, for example this DEV Community post on idempotency keys for agent tool execution, and a separate analysis of retries in nondeterministic LLM pipelines. This piece covers what they mostly skip: the window between the side effect happening and your database finding out.
Where the key comes from decides everything
If the key is born inside the retry loop, every retry gets a new one and you have no idempotency at all. If it is derived from the model's raw output, a re-sampled response with slightly different wording produces a different key for what the business considers the same action.
The key should come from things that survive a crash and a resume: the workflow run ID, the step number, the tool name, and a canonical form of the arguments (sorted keys, normalised whitespace, amounts as integers in minor units). Hash those together. The model proposes the call; the orchestrator decides whether this call has already been made.
This also separates two kinds of retry that need opposite handling. A transport retry (the network dropped, the worker restarted) should return the same result for the same key. A sampling retry (the model produced a bad plan and you ask again) is a different call and gets a different step number. Mixing them up is how an agent ends up refunding the same order twice with two slightly different reasons.
The ledger most people draw
The standard design has two moves around the side effect: reserve the key before calling the tool, record the result after. In Postgres the reserve is an insert that does nothing on conflict, and you check whether a row came back.
CREATE TABLE tool_ledger (
idem_key text PRIMARY KEY, -- hash(workflow_id, step_no, tool, canonical_args)
tool text NOT NULL,
state text NOT NULL CHECK (state IN ('reserved','done','failed')),
result jsonb,
attempts int NOT NULL DEFAULT 1,
reserved_at timestamptz NOT NULL DEFAULT now(),
finished_at timestamptz
);
-- Step 1: try to claim the call. Zero rows back means someone else already did.
INSERT INTO tool_ledger (idem_key, tool, state)
VALUES ($1, $2, 'reserved')
ON CONFLICT (idem_key) DO NOTHING
RETURNING idem_key;
-- Step 3: record the outcome after the side effect returns.
UPDATE tool_ledger
SET state = 'done', result = $2, finished_at = now()
WHERE idem_key = $1 AND state = 'reserved';If the insert returns nothing, someone has already claimed this call. If the row says done, return the stored result. This handles concurrent workers and clean restarts, and for tools where the downstream API accepts your key it is close to enough.
The crash between execute and record
Now kill the worker after the courier API returns 200 and before the UPDATE runs. The ledger row says reserved. The side effect has happened. Nothing in the database can tell you that.
A two-state ledger forces a bad choice on the next worker that finds a reserved row. Retry, and you may book two couriers. Refuse, and you strand a workflow whose action did in fact complete. Both are wrong some of the time, and which one is worse depends on the tool.
“A reserved row older than its lease does not mean "failed". It means "I don't know", and the code should say so.”
The same state appears on a timeout. If the HTTP client gives up after 30 seconds, the request may still be processing on the other side. The only outcome you can classify with confidence is a failure that happened before any bytes were sent, such as a DNS error or a refused connection. Everything after transmission is unknown until proven otherwise.
Make the unknown state explicit in code
The wrapper below keeps the ledger honest. It marks a call failed only when the error proves nothing was sent, and otherwise leaves the row reserved so the lease timer, not optimism, decides what happens next.
async function runTool(key, tool, args, handler) {
const claimed = await ledger.reserve(key, tool); // INSERT ... ON CONFLICT DO NOTHING
if (!claimed) {
const row = await ledger.get(key);
if (row.state === 'done') return row.result; // replay, no side effect
if (row.state === 'failed') throw new ToolFailed(row);
// state is 'reserved': previous attempt died or is still running
if (Date.now() - row.reservedAt < tool.leaseMs) throw new InFlight(key);
return reconcile(key, tool, args, handler); // outcome unknown
}
try {
const result = await handler(args, { idempotencyKey: key });
await ledger.finish(key, 'done', result);
return result;
} catch (err) {
if (err.definitelyNotSent) await ledger.finish(key, 'failed', null);
throw err; // otherwise leave it 'reserved'
}
}Three ways to resolve an unknown outcome
Reconciliation is a per-tool policy, and you should write it down next to the tool definition rather than discover it during an incident. In practice there are three options, and a tool usually supports only one or two.
| Tool type | Resolution | Example | Risk if wrong |
|---|---|---|---|
| Provider honours your key | Retry with the same key; provider returns the original result | Payment intent creation with an Idempotency-Key header | Key pruned by provider before the retry |
| Provider lets you look up by your own reference | Read back, then decide to replay or execute | Courier booking tagged with the run ID | Eventual consistency hides a booking that exists |
| Neither | Mark at-most-once; escalate to a person | Plain SMTP send, some webhooks | Missed action instead of a duplicate one |
The third row is the uncomfortable one. For an irreversible, non-queryable action, a duplicate and a miss are both failures, and the agent has no business choosing between them. Park the run, show a human the ledger row and the last request log line, and let them decide. That costs minutes. A silent double-send of a customer email costs trust.
Provider keys are not a free pass
Passing your key downstream helps only if the provider stores it long enough. Stripe documents that keys may be removed from the system after at least 24 hours, so an agent run that pauses overnight for approval and then resumes can replay a key the provider no longer remembers. Your own ledger must outlive the longest pause your workflows allow, and the retry path should check your ledger first and the provider second.
Also check what the provider does when the same key arrives with different arguments. Some reject it, which is useful: it catches the case where your canonicalisation drifted between deploys. Others silently return the first result, which hides the drift. Test both behaviours before you rely on either.
What to measure
- Count of ledger rows in reserved state older than the lease. This is the unknown-outcome rate, and it should be near zero.
- Replay rate: calls served from a done row. A sudden rise means something upstream is retrying more than it used to.
- Time to resolution for unknowns, split by policy. If the human-review queue grows, a tool needs a provider key or a lookup endpoint.
- Canonicalisation mismatches: same workflow and step, different argument hash. This points to a nondeterminism bug before it becomes a duplicate.
Agents will keep gaining access to tools with real consequences, and the ledger is where that access gets audited. The next useful addition is probably not a smarter model but a ledger that can say, per call, exactly what it knows and what it is guessing.
Frequently asked questions
Related reading
Zero Data Retention Covers a Request Path, Not a Vendor. The Fireworks Breach Reached the Paths Outside It.
The Fireworks AI breach exposed Notion AI transcripts despite a zero data retention commitment. The promise applied to inference requests. The transcripts sat in a feature that stores by design.
Developer Trust in AI Is Conditional: Only 6.6% Trust It With Important Decisions
Stack Overflow's 2026 survey: 80% use AI for an hour a day, 48% trust it only when they can verify it, 6.6% trust it with important decisions. Verification cost sets the ceiling.
Postgres 14 Reaches End of Life on 12 November. On RDS, the Date That Bills You Is 1 March 2027.
The Postgres community stops patching version 14 on 12 November 2026. Amazon RDS keeps it in standard support until 28 February 2027, then charges. Here is how to read the gap.