The Fix That Charged Twice
A payment timeout can occur after the remote service has committed the operation. Retrying without preserving the identity of the logical purchase can therefore create a second effect.
Watch the demonstration
The experiment is also written out in full below.
The experiment
The claim under test
A plain retry after a lost payment response charges the order twice; a stable operation key, claimed in the same transaction as the charge, keeps it to one charge without blocking new purchases.
The question
If a payment’s confirmation is lost after the money has moved, does the retry charge the customer again?
Scope
A purpose-built simulation: a demo store called Field Supply, a network hop with a fault injector, a payment service and a PostgreSQL ledger, connected by real network calls on one machine. The same code runs as an unsafe build and a corrected build with identical retry rules. The result covers this checkout workflow and the scenarios below. It does not cover a real payment provider, real money, other retry policies, or exactly-once delivery across a distributed system.
How it was run
Both builds use the same retry rules: the store’s payment client waits 1 second for an answer, then retries, up to 3 attempts with a backoff starting at 250 ms. The unsafe build retries with no operation ID. The corrected build saves a stable operation ID with the order before the first attempt and sends it on every attempt; the payment service claims that ID, writes the charge and stores its answer in one database transaction, and the ledger also allows only one charge per order. A scripted harness ran ten scenario runs: the unsafe baseline, five scenarios on the corrected build, and four of those again on the unsafe build for comparison. The filmed run is the same lost-response case, started by hand from the store page with the fault injector armed, once on each build. Every charge count comes from the ledger rows for the order, never from what the store was told, and each ledger row is chained to the one before it by hash, so a row cannot be quietly edited or removed.
The challenge introduced
A fault injector on the network hop lets the first payment attempt reach the payment service and commit its charge, then throws away the 201 response. The store waits out its 1 second timeout with the money already taken and no way to tell whether it was.
Pass and fail criteria
Fixed before the run, so the experiment could not be rewritten once the results arrived.
Unsafe baseline: the charge commits, its response is lost and the store retries, leaving two charges for one order. If that does not happen, the run has not reproduced the failure and cannot test the fix.
Corrected retry: under the same fault, the retry carries the same operation ID, receives the original result, and the order has one charge.
Concurrent retries: five simultaneous requests under one operation ID produce one charge, and the other four receive the original result.
Changed amount: a retry under the same operation ID that asks for $99 instead of $49 is refused with 422, the order is flagged for review, and the ledger keeps one $49 charge.
Separate purchase: two genuine orders produce two charges, so the fix does not block real purchases.
Key lifetime: a redelivery one second inside 24 hours is answered from the stored result without a new charge; one second past 24 hours the operation ID has expired and the ledger’s one charge per order rule refuses it with 409.
Unsafe comparisons: without the fix, the simultaneous requests, the changed amount and the redeliveries each add charges, while two genuine orders are still two charges.
What the run showed
In the filmed run on 9 October, the unsafe build charged order ord_af95de1f411755f9 twice: ledger rows #496 and #497, $49.00 each, written 1.2 seconds apart, while the store received a single success response. Under the same lost response, the corrected build wrote one charge, row #225 for order ord_c6d81c6f581d1d74, and answered the retry with the original result. In the scripted run that followed, all ten scenario runs matched their expectations and all 40 checks passed. The corrected build held each order to one charge: after the lost response, across five simultaneous requests (four answered with the replayed original), when the retry changed the amount to $99 (refused with 422), and across redeliveries one second either side of 24 hours (replayed, then refused with 409). Two genuine orders were each charged once. Given the same cases, the unsafe build charged $98.00 for the baseline, five times ($245.00) for the simultaneous requests, $49.00 plus $99.00 for the changed amount, and three times ($147.00) for the redeliveries. A repeat run five minutes later and an earlier run on 8 October produced the same counts, every ledger hash chain verified, and all three packaged runs are listed in the evidence below.
Verdict
Supported for this workflow: the lost response produced a duplicate charge only in the unsafe build, and the corrected build held every tested retry to one charge while separate purchases still went through.
The experiment behaved as the claim predicted, for the systems, versions and conditions stated in the scope. It is not a general guarantee.
Authority result: Not applicable. Recorded separately from the verdict, because permission and correctness are different properties.
What was running
- Tested on
- 2026-10-09
- Application revision
- prove-it-charged-twice 1.0.0 (unsafe and corrected builds)
- Environment
- a local lab on one macOS machine
- Run identifiers
- exp_20261009T042341Z_f9bb, exp_20261009T042900Z_4af9
- Evidence reviewed by
- Kevin Kissi
Figures
Illustration, not measured data
Captured from the run
Captured from the run
Evidence
Every artifact below was reviewed and cleared for release. Material that was examined but cannot be published is listed without a download.
Evidence package from the scripted run after the film (opens in a new tab)
Dataset from run exp_20261009T042341Z_f9bb
All ten scenario runs: each check’s expected and observed values, every request and response, each order’s four-lane timeline and ledger rows, structured logs, and a SHA-256 manifest of every file. Its ledger exports cover every order in the lab database at the time, including the filmed rows #496, #497 and #225, with both hash chains verified.
Download dataset135 KBCleared by Kevin Kissi on 2026-10-13.
Evidence package from the repeat run (opens in a new tab)
Dataset from run exp_20261009T042900Z_4af9
The same ten scenario runs five minutes later, with the same counts in every scenario and the same package layout.
Download dataset137 KBCleared by Kevin Kissi on 2026-10-13.
Evidence package from the earliest run, 8 October
Dataset from run exp_20261008T223217Z_1701
The earliest packaged run, against a fresh database, with the same 40 of 40 result. Reviewed and held back, because its test report names the machine the lab ran on.
Reviewed, not published.
Cleared by Kevin Kissi on 2026-10-13.
Limitations and reproduction
What this result does not establish
- It is a simulation. No payment provider and no real money were involved; a charge is a row in the lab’s own PostgreSQL ledger.
- The timings come from one machine and are not performance claims.
- The 24 hour scenario shifts the lab’s clock per request instead of waiting a day. The shift is recorded in every trace and exists only in the lab.
- One retry policy was tested: a 1 second timeout and up to 3 attempts. Other timeouts and backoff schedules, and a client that creates a new key for every attempt, were not tested.
- The 24 hour key lifetime is the lab’s own choice. The result does not show that any payment provider’s idempotency contract behaves the same way, and it does not establish exactly-once delivery across a distributed system.
- The film shows Zof’s Idempotency Testing Agent and verification engine checking the corrected build in the Zof console. Their records are not part of the published evidence, so this verdict rests on the lab’s own requests, traces and ledgers.
Reproducing this
The lab runs as one script that creates a fresh database, starts both builds, runs the automated tests and every scenario, then writes a new evidence package and re-hashes it. Identifiers, hashes and millisecond timings change from run to run; charge counts and amounts do not, because those are what the checks assert. A published package can be checked without the lab:
- Download an evidence package below and unzip it.
- Recompute the SHA-256 fingerprint of every file and compare it with manifest.json. A changed byte shows up as a mismatch.
- Read results.json for each scenario’s expected and observed values, and ledger/unsafe.json and ledger/corrected.json for the hash-chained ledger rows.
The lab’s source code is not published, so the run cannot be repeated from public materials. The evidence packages can still be checked on their own.
The companion article
Idempotency under ambiguous payment outcomes
A payment timeout can occur after the remote service has committed the operation. Retrying without preserving the identity of the logical purchase can therefore create a second effect.
The experiment uses a synthetic order and a controllable payment service. A fault injector discards the response only after the service records acceptance. Capture that injection point explicitly. A timeout before acceptance tests a different condition and cannot establish the ambiguous-outcome failure shown here.
Count committed payment effects by order identity, not HTTP attempts. Preserve the request sequence, operation key, payment records, and reconciliation query. The healthy expectation is one logical charge for one authorized purchase, while legitimate separate purchases remain possible.
The corrected fixture assigns a stable key to the logical operation and coordinates duplicate handling at the commit boundary. A client-side key alone is insufficient if the receiving service ignores it. Test overlapping retries, changed parameters under the same key, a different purchase with a different key, and behavior after the documented retention window. Provider contracts differ; the lab must state its own semantics rather than assuming indefinite deduplication.
Run the unsafe and corrected variants against the same fault schedule. Add a normal purchase control to reveal fixes that merely reject everything. If reconciliation is asynchronous, define when the test may conclude and what pending means before execution.
The result supports a bounded claim about duplicate effects in this workflow. It does not prove exactly-once delivery across a distributed system. The durable business invariant is the important distinction: repeated transport attempts should not create an additional unauthorized charge. Publish the ledger evidence alongside the film so the viewer can inspect that claim directly.
Transcript
What was said on camera, edited for readability. The argument is unchanged. This is not the companion article.
Read the transcript
One order, two charges
One order, one click, two charges. While testing a checkout platform before production, we uncovered a flaw that could charge a customer twice for the same purchase.
A synthetic $49 purchase
This is a synthetic $49 purchase. The payment service accepts it. Then our fault injector drops the response before the store receives confirmation. From the store’s perspective, the request timed out. From the payment service’s perspective, the money already moved.
The request timeline
On the Zof AI console, we follow the failure through the request timeline. Request sent, payment committed, response lost. The application retries because it wants to finish the job. But this implementation treats the retry as a new payment. The dangerous moment is the gap between what happened and what the caller knows happened.
The durable record decides
Now we inspect the payment ledger using the order identifier. We do not infer the number of charges from the number of requests. Two requests can correctly produce one charge. One success message can conceal two. The durable record decides.
The candidate correction
Our candidate correction gives the logical purchase a stable operation key and enforces duplicate handling where the payment is committed. Reusing a key with different payment details must not silently reuse the wrong result. Concurrent retries also need a defined outcome.
We rerun the lost response, then send overlapping requests, then try a different legitimate purchase. That last case checks that our fix has not stopped all payments.
Resolve the uncertainty first
We bring the request trace, payment records, and limits of the key’s lifetime together for review. A timeout is uncertainty, not evidence that nothing happened. Reliable recovery starts by resolving that uncertainty before repeating an irreversible action. That is the difference between finishing the purchase and charging for it again.
Technical sources
Context for the mechanism under test. These describe intended behaviour; the artifacts above are what establish this episode's result.
Public validation domain catalog
Idempotent requests (opens in a new tab)
Stripe. Example provider-specific retry contract; not a universal retention rule
Request this experiment against your workflow
We will run the same controlled demonstration on a system you choose, and publish the evidence to you rather than to the internet.
