Do not take our word for it.
Real reliability questions. Controlled experiments. Evidence you can inspect.
Every claim in this series has a run behind it.
Each episode takes one thing people say about software reliability, builds a fixture that can prove it wrong, and publishes what the run produced. The scope is stated, the limitations are stated, and an experiment that fails is published as a failure.
How an experiment works
Every episode follows the same four steps, in the same order, whatever the result turns out to be. The criteria are fixed before the run, so the experiment cannot be quietly rewritten once the numbers arrive.
- 01
The claim
One proposition about reliability, stated so it can be wrong.
- 02
The experiment
A controlled fixture, with the pass and fail criteria fixed before the run.
- 03
The evidence
The artifacts the run produced, published so you can check them.
- 04
The verdict
Supported, contradicted, or inconclusive, inside a stated scope.
The three results an experiment can have
- Supported within the tested scope
- The experiment behaved as the claim predicted, for the systems, versions and conditions stated in the scope. It is not a general guarantee.
- Contradicted by the experiment
- The run produced the outcome the claim said it would not. The claim does not hold under these conditions.
- Inconclusive
- The run did not establish the conditions the claim needs, so it neither supports nor contradicts it. The experiment is reported anyway.
An experiment that has not run yet has no result at all, so it is not listed here as one. Nothing on this site describes a planned experiment as a completed demonstration.
The experiments
Showing all 2 experiments.
- 01Transaction Integrity
The Fix That Charged Twice
A payment timeout can occur after the remote service has committed the operation. Retrying without preserving the identity of the logical purchase can therefore create a second effect.
Supported within the tested scope2 min
- 02Test Quality
The AI Wrote 100 Tests. Did They Catch Anything?
Generated test volume is an output measure. Defect detection is an outcome measure. A useful evaluation needs known faults, healthy controls, and a clear boundary between generation and access to the answer.
Contradicted by the experiment2 min
The argument behind the experiments
The Reliability Thesis sets out why verification, authority and evidence have to change as software becomes autonomous. PROVE IT takes one claim from that argument at a time and tries to break it in a lab. If the two ever disagree, the experiment wins.
Read The Reliability ThesisRequest this experiment against your workflow
We will run the same controlled demonstration on a system you choose, and publish the evidence to you rather than to the internet.
