The AI Wrote 100 Tests. Did They Catch Anything?
Generated test volume is an output measure. Defect detection is an outcome measure. A useful evaluation needs known faults, healthy controls, and a clear boundary between generation and access to the answer.
Watch the demonstration
The experiment is also written out in full below.
The experiment
The claim under test
Tests Zof generates from a written spec, run blind, catch at least 7 of 10 planted faults and raise at most 1 false alarm on 10 harmless changes.
The question
If an AI writes the tests for an application, do those tests notice when the application is wrong?
Scope
One application: Northside Studio, a standalone booking app with synthetic customers and bookings, described by Booking spec v3 (10 sections, 20 rules). One generated suite, 20 hidden versions of the app (10 faults and 10 harmless changes, with 2 more faults held back for a later round) and one automated check. The result covers this suite against this app and this fault set. It is not an accuracy score for test generation in general, and it does not measure a suite after human review.
How it was run
Zof's test generation service, its scenario and test case agents, wrote cases from Booking spec v3 using one scenario per spec section: 117 cases from 10 scenarios. The lab translated them into executable checks; 55 were runnable against the app and the other 62 were excluded before the run. The lab then planted the faults and sealed the manifest, so the suite was written before the faults existed and frozen before they were revealed. Every runnable test ran on the healthy app and on all 20 hidden versions. A fault counts as caught only when a test that passes on the healthy app fails on that version and the failure relates to the planted change. Several failing tests on one fault count as one detection.
The challenge introduced
Ten planted faults in hidden versions of the app, from a refund window one day short to a confirmation email that drops the time zone, mixed with ten harmless changes such as a renamed helper or an upgraded date library. All twenty were sealed in a manifest before the run.
Pass and fail criteria
Fixed before the run, so the experiment could not be rewritten once the results arrived.
Pass: at least 7 of the 10 planted faults caught, each fault counted once however many tests fail on it.
Pass: at most 1 false alarm across the 10 harmless changes.
A test that already fails on the healthy app counts toward neither number, because it cannot tell a fault from a working app.
The suite is generated before the faults exist and frozen before they are revealed. Nothing is regenerated against the revealed manifest.
What the run showed
Check chk_e403cf97be52c80e ran on 15 October. On the healthy app, 24 of the 55 runnable tests already failed (10 failed and 14 stopped with errors), which left 31 that could tell a fault from a working app. Those caught 3 of the 10 faults: admins can't see cancelled bookings, a reschedule keeps the old slot reserved, and search ignores the end date, each through a failing test that checks that behaviour. The other 7 faults changed no test result at all: the short refund window, double booking, a reused booking reference, bookable past slots, the doubled cancellation fee, price rounding and the confirmation email's time zone. No harmless change produced a false alarm. Nobody reviewed the suite in this run, so its reviewed lane is identical to the original. Against a bar of 7 caught and at most 1 false alarm, the suite caught 3.
Verdict
Contradicted on this app: the generated suite caught 3 of 10 planted faults against a bar of 7, although it raised no false alarm on the harmless changes.
The run produced the outcome the claim said it would not. The claim does not hold under these conditions.
Authority result: Not applicable. Recorded separately from the verdict, because permission and correctness are different properties.
What was running
- Tested on
- 2026-10-15
- Application revision
- Northside Studio booking lab, Booking spec v3
- Agent or model version
- Zof TestScenarioAgent and TestCaseAgent at platform commit 42c6f4e7 on a gpt-6-astra deployment (low reasoning effort); checks translated by gemini-2.0-flash at temperature 0
- Environment
- the Zof console v1 pilot, against a simulated booking app
- Run identifiers
- chk_e403cf97be52c80e, gen_20261015T123257Z
- Evidence reviewed by
- Kevin Kissi
Figures
Illustration, not measured data
Evidence
Every artifact below was reviewed and cleared for release. Material that was examined but cannot be published is listed without a download.
Evidence package from the check (opens in a new tab)
Dataset from run chk_e403cf97be52c80e
The sealed fault manifest, the frozen original and reviewed inventories of all 55 runnable tests, and every test's result on the healthy app and on each of the 20 versions, with a SHA-256 list of every file. Two strings naming internal infrastructure were redacted; REDACTIONS.md lists them.
Download dataset928 KBCleared by Kevin Kissi on 2026-10-17.
Limitations and reproduction
What this result does not establish
- One app, one specification and one fault set. Ten faults are an exploratory sample, not an accuracy score for test generation in general.
- This run had no human review: no test was edited, rejected or added, and review time was zero. The film walks through a separate session with a reviewed suite of 100 tests; that session's records are not part of the published evidence, so its scorecard is not used here.
- Only one generated scenario per spec section was used, out of 10 to 24 the generator wrote for each, and 62 of the 117 cases could not be run against the app.
- The generation calls ran on a gpt-6-astra deployment rather than the model the agents are configured for, with the prompts unchanged, and a second model translated the cases into checks. Another model or translation could score differently.
- The faults were planted automatically by the lab and sealed before the run, not by a separate engineer as the film describes.
- The console stopped before a person confirmed each call. The calls above were checked against the run files: every catch is a test that passes on the healthy app failing on the faulty version, and every miss leaves all 55 results unchanged.
- The booking app is a standalone simulation with synthetic customers and bookings.
Reproducing this
The check is defined by its exported files: the sealed manifest of the 20 versions plus the 2 held back, the frozen original and reviewed inventories, and the result of every test on the healthy app and on each version. The score can be recounted from those files alone:
- Download the evidence package below and unzip it.
- Read manifest.sealed.json to see which slots are faults and which are harmless changes.
- Compare each runs/slotNN.original.json with runs/healthy.original.json. A fault is caught when a test that passed on the healthy app fails on that version.
- Check the files against the SHA-256 list in EVIDENCE.json, and read REDACTIONS.md for the two strings removed before publication.
The booking lab and the generation service are not published, so the run cannot be repeated from public materials. The package lets anyone recompute the score.
The companion article
Measuring generated test effectiveness
Generated test volume is an output measure. Defect detection is an outcome measure. A useful evaluation needs known faults, healthy controls, and a clear boundary between generation and access to the answer.
Create a reviewed specification and a stable healthy application. A separate author prepares fault variants and seals the manifest before the testing operator begins. Keep equivalent behavior changes and genuinely faulty changes distinct. Some code mutations do not alter observable behavior and should not be silently scored as missed defects.
Freeze the generated inventory before opening the manifest. Retain both raw and human-reviewed cases. Record edits, discarded cases, setup effort, and the assertions responsible for detections. This allows readers to assess the combined workflow without attributing all of its value to generation alone.
Score at the defect level. Ten assertions failing on one bug are not ten defects found. Confirm a detection by relating a failed assertion to the intended injected fault, ideally with a healthy control and a minimal reproduction. Report unrelated failures separately. False alarms on healthy variants create investigation work and belong beside successful detections.
Publish the complete matrix: variant, expected behavior, executed cases, observed result, attribution, and disposition. Include missed faults even when they appear obvious after the reveal. Do not regenerate against the revealed manifest and present the improvement as a blind result; label it a subsequent informed round.
The experiment measures performance on its particular specification and fault distribution. It cannot establish enterprise-wide accuracy from a handful of variants. Its practical value is diagnostic: it shows which classes of assertion need better requirements, stronger review, or additional validation. A smaller inventory that detects meaningful failures may be more valuable than a large collection of redundant happy-path checks.
Transcript
What was said on camera, edited for readability. The argument is unchanged. This is not the companion article.
Read the transcript
An AI wrote a hundred tests
An AI just wrote a hundred tests. That sounds productive. Here’s the question I actually care about: if the software were wrong, would those tests notice?
Keeping the answer sealed
Today, we’re giving Zof a specification and a working application. A separate engineer prepares faulty versions without telling the operator where the faults are. The test inventory is frozen before we reveal them. That separation matters. Otherwise, we could quietly teach the test to find the answer.
Three numbers
We measure three things. Which faults are detected? Which escape? And how often does healthy behavior get reported as broken? Generating more cases only helps if those cases add useful distinctions.
The first execution
Watch the first execution. We preserve the original generated cases, then record every human correction in a separate reviewed version. If a person supplies the assertion that catches the bug, that contribution belongs in the result. We’re evaluating the workflow, not pretending the person disappeared.
The fault manifest
Now, the fault manifest is opened. Each row is one independently defined defect. Several failing cases can point to the same defect. That still counts as one detection. A green case on a faulty version may mean the case never exercised that behavior. We inspect the evidence before calling it a miss or a catch.
What still got through
The article includes the clean controls, missed faults, rejected tests, and review time. This is a small benchmark on a declared application, not a universal accuracy score. But it answers a much better question than how many tests appeared. Show me what changed our ability to find a defect. Then, show me what still got through.
Technical sources
Context for the mechanism under test. These describe intended behaviour; the artifacts above are what establish this episode's result.
Human review and requirement-based generation workflow
Requirement to case and execution relationships
Request this experiment against your workflow
We will run the same controlled demonstration on a system you choose, and publish the evidence to you rather than to the internet.
