AIによるテスト生成だけでは不十分

Large language models made it trivial to draft test cases from a Jira ticket, an OpenAPI spec, or a UI screenshot. Authoring was the visible bottleneck, so a tool that produced a hundred plausible cases in seconds felt like the answer. Teams celebrated faster first drafts and broader initial coverage, and they were right to.

The popularity is rational. But authoring was never the entire enterprise problem. A test you generated on Monday is a liability by Friday if nothing keeps it aligned with the system it was written against.

Generation earns its place at the start of a workflow, when a human still owns the judgment and a maintained system owns what happens next. Used that way, it removes real toil.

Generated tests drift the moment the system changes. They cannot prioritize without a System Graph to tell them what changed and what depends on it. They do not choose safe environments, respect data policy, or produce audit-grade evidence by default. They flag failures; they do not reproduce, remediate, or verify them.

Without governance and maintenance around it, generation becomes another source of CI noise, a suite that turns red for reasons no one trusts, until the team learns to ignore it. This is the same pattern that made static scripts a liability, explored in Testing Fleets, not test scripts.

AI-generated code now accounts for roughly 41% of codebases by Zof's research, and the volume of change has outrun the suites meant to validate it. The cost center moved from "writing tests" to "keeping validation accurate under continuous change." Generation addresses the first; it is silent on the second.

Reliability is operated, not authored once. That means deciding what to validate for a given change, executing inside human-defined boundaries, interpreting evidence in context, and closing the loop when something breaks. We make the full version of this argument in The AI code testing imperative.

Generation answers "can we write a test for this?" The enterprise question is "is the right thing still being validated after the system changed 400 times?"

Consider a payments service. On day zero, a generation tool drafts 220 tests from the OpenAPI spec and acceptance criteria. They pass, coverage looks healthy, and the suite is merged.

Over the next thirty days the team ships 400 changes: a renamed field, a new idempotency requirement, a refactored retry path, a third-party webhook migration. The generated suite has no map of any of it. Some tests now assert against fields that no longer exist and fail loudly. Others still pass while silently validating dead code paths. Nobody can say which 40 of the 220 actually matter for the change that shipped this morning.

Day 0    220 generated tests, all green
            │
Day 30   400 changes shipped
            │
  Generation only ──► drift: false reds + silent false greens
            │
  System Graph + Fleets ──► run the 40 that matter, retire the noise,
                            attach evidence to the change that ran them

The columns are not competitors. Generation becomes one input on the left that a control plane consumes on the right. The point is that authoring is a single step in a loop, not the loop itself.

The common objection is that modern generators self-heal: they re-run, detect a broken selector or assertion, and rewrite it. This helps with surface-level brittleness. It does not solve the harder problem, because self-healing optimizes for keeping a test green, not for keeping it correct.

A test that rewrites its own assertion to match changed behavior can paper over a regression instead of catching it. Maintenance has to be anchored to a model of what the system is supposed to do and what changed, which is what the System Graph provides. Without that anchor, "self-healing" can quietly erode the coverage it claims to preserve.

A control plane coordinates agents, policies, evidence, and integrations across the whole reliability loop. Generation becomes one capability inside it, not the product. Testing Fleets maintain validation as the system changes, Remediation Fleets propose fixes that humans authorize, and the governance layer keeps every action policy-bound and auditable.

Procurement should score vendors on operated reliability outcomes, not on lines of generated code. The decision is closer to a build-versus-buy question for reliability infrastructure than a tooling purchase, which we cover in Build versus buy test automation.

If a tool leads with generation, the right questions are about everything that happens after the draft. Use this checklist in a proof of concept rather than a slide review.

Autonomous reliability infrastructure connects generation, where it is useful, to maintained fleets, graph context, telemetry, and optional governed remediation. Tests become assets in an operated system, not disposable drafts. The loop runs Understand, Test, Reproduce, Remediate, Verify, with humans setting boundaries at every gate.

The result is validation that stays proportional to risk as the system moves. For a deeper treatment of the operating model, see the autonomous reliability infrastructure guide.

AI test generation is a feature. Enterprise reliability is a platform. The hard part was never writing the first version of a test; it is keeping the right things validated, safely and auditably, while the system changes underneath you.

Evaluate tools on closed-loop outcomes operated over time, escaped defects, reproduction time, maintenance load, and evidence quality, not on demo-day velocity. If a vendor cannot answer the day-30 question, generation is all you are buying.