Design a verification API that cannot confuse incomplete with passed

The contract below is a vendor-neutral proposal, not a shipped API specification.

A verification service receives a request for two checks. One passes; the other never starts. The response contains zero failures. Should a deployment controller proceed?

If the controller reads failure_count == 0, it might. The mistake is in the contract: the service has described the results it possesses without accounting for the work it owes. An empty result list is also a list with no failures.

Design the response around a stronger invariant: permission requires explicit passing evidence for every required check, bound to the requested subject and policy, from a completed execution. Anything less withholds permission. That does not mean every interruption is a defect. It means missing knowledge cannot authorize an action.

The useful deliverable is a protocol both producers and consumers can implement, with adversarial examples that expose unsafe interpretations before integration.

Transport success, execution progress, and verification outcomes answer different questions. HTTP 202 Accepted means a request was accepted for processing, not that processing completed. RFC 9110 calls that response intentionally noncommittal and recommends describing current status and pointing to a status monitor.[1] A successful response from that monitor can still describe unsuccessful verification.

Existing APIs illustrate the distinction. GitHub's Checks API has a run status and a separate conclusion; a completed run can conclude with failure, cancellation, or timeout.[2] That is precedent for separating concepts, not a specification to copy unchanged.

In this proposal, execution has these states:

Each reported check separately carries pass, fail, or unknown. An unknown outcome requires a reason: unsupported, timeout, cancelled, error, or, in a nonterminal snapshot, pending.

A completed run can contain a failed check. It can also contain an unsupported check: the coordinator finished accounting for the work, but could not establish the required property. Conversely, a timed-out run can retain useful passing results. Execution and findings remain independently visible; the permission rule combines them conservatively.

The consumer must know which checks were required before it interprets the answer. Otherwise, a producer can accidentally make its own work look complete by omitting an unavailable check from both inventory and results.

Persist the accepted request separately. Bind it to a run ID, an immutable subject identifier, a policy revision, and a nonempty inventory of versioned check IDs. Each ID denotes one execution obligation; parameterized cases need distinct IDs. The result echoes those bindings, but the consumer compares them with its own trusted request.

For the small contract here, every inventory entry is required. Optional diagnostics belong in a separate run. That is restrictive, but avoids silently deciding that an unsupported required check has become advisory. A larger design can support optional checks if requirement levels are fixed in the accepted request and policy.

Coverage needs more than a percentage. Derive these counts from validated results:

Two reported records out of two expected checks gives complete accounting, even if one result is unknown. It does not give complete passing evidence. Publish both distinctions. Never trust a producer-supplied coverage: 100 as authorization.

This is the proposed v1.0 schema shape. It is neither a formal JSON Schema document nor a vendor's shipped contract. The accompanying Python validator supplies the executable constraints for this example.

{
  "contract_version": "1.0",
  "run_id": "run-17",
  "subject": "sha256:example-artifact-17",
  "policy": "release-policy@3",
  "inventory": ["api-contract@1", "authorization@2"],
  "execution": {"status": "completed"},
  "results": [
    {"check_id": "api-contract@1", "outcome": "pass", "reason": null},
    {"check_id": "authorization@2", "outcome": "unknown", "reason": "unsupported"}
  ]
}

The subject string is a visibly synthetic placeholder, not a computed digest. A production request must identify the actual artifact and relevant environment or configuration; a mutable branch name is insufficient. A result also needs authenticated provenance before an action controller can trust it. Echoing identifiers does not authenticate anything.

Every field shown is required. Unknown fields, versions, enum values, and unexpected check IDs are rejected in this deliberately strict example. Pass and fail require a null reason; unknown requires a recognized reason. A completed run must account for every inventory member and cannot contain a pending check.

Results remain an array until uniqueness is validated. Converting them immediately into a dictionary keyed by check ID can overwrite an earlier failure with a later pass. Reject duplicate IDs, even when their outcomes agree: duplicates signal an ambiguous snapshot or retry protocol.

Also reject duplicate JSON object members during parsing. RFC 8259 says object names should be unique and explains that behavior with duplicates is unpredictable across implementations.[3] Checking result IDs after a parser has already discarded duplicate outcome members is too late.

An unsupported check means the configured verifier cannot evaluate that obligation. A timeout means it did not establish an outcome within a budget. Cancellation means execution was stopped through the cancellation protocol. None says the system under examination violated its requirement.

The distinction determines recovery. Unsupported work needs another capability or a deliberate policy change. A timeout needs investigation of budget, dependencies, and execution state. Cancellation needs a decision about whether the work is still wanted. Reporting all three as fail inflates defect counts; reporting any as pass manufactures evidence.

A caller-side deadline is especially important. gRPC documents that DEADLINE_EXCEEDED can be returned even when a state-changing operation completed successfully, because its response arrived too late.[4] Therefore, a client timeout should withhold permission and reconcile the known run through its status endpoint. It must not fabricate a terminal server result or blindly submit potentially side-effecting work again.

Similarly, requesting cancellation is not proof that workers stopped. Keep the run nonterminal until cancellation is acknowledged according to the service's execution model. Define whether late evidence is retained, and never let a late pass silently rewrite a cancelled attempt into an authorized one.

Here, “deny” means a valid response cannot satisfy this gate. “Reject” means the consumer cannot accept the response as a valid contract instance. Both prevent action, but they route operational work differently.

Do not give downstream clients a producer-defined success flag that competes with the detailed evidence. Calculate permission after structural validation, binding checks, inventory reconciliation, and duplicate rejection.

The fixture's final predicate is deliberately small:

allow = (
    state == "completed"
    and counts["passed"] == counts["expected"]
)

That expression is safe only because earlier validation established a nonempty trusted inventory, one result per ID, recognized outcomes, and complete accounting for a completed run. Without those preconditions, all([]) or two duplicate passes can produce a misleading success.

Fail closed at the next boundary too. A deployment controller should proceed only when the evaluator returns exactly allow. Parsing errors, evaluator exceptions, lost responses, and unknown decisions must stop the protected action, rather than invoke a permissive fallback. Log enough context to distinguish a policy denial from a malformed producer response.

For retrying services, define one authoritative attempt snapshot. Do not combine a passing check from yesterday's attempt with another check from today's attempt merely because their names match. This fixture binds a single run and rejects duplicates; durable attempt reconciliation belongs in the service around it.

The companion uses only Python's standard library. From its directory, run:

python3 -B fixture.py

It generates synthetic response messages, feeds their JSON text to the consumer, and compares decisions and selected coverage counts against explicit expectations. It does not execute an application, contact a service, or demonstrate a product's verification capability.

The recorded run matched 32 of 32 expectations. The happy path allowed the action. A completed response missing a required check was rejected, as were both matching and conflicting duplicate results. A partial response with one passing check was denied with one missing check. A timed-out run with no results was denied with zero failures and two missing checks.

Other cases cover unsupported checks, per-check timeout, cancellation, runner error, running/pending snapshots, inventory shrinkage, stale subject and policy bindings, unknown versions, duplicate JSON members, and empty inventories. A timed-out run containing two passing results still denies permission: this proposal requires normal completion, even when surviving evidence looks favorable.

The run receipt records the interpreter, timestamp, expected and actual decisions, and SHA-256 hashes of the executable fixture and generated inputs. These assertions demonstrate the consumer's behavior for those cases. They do not prove the design correct for every possible message or establish authenticity of incoming evidence.

Treat contract version, policy revision, and check revision as separate dimensions. Changing a wire field is different from changing which checks are required or what a particular check evaluates. All three can change the interpretation of a pass.

This teaching consumer accepts exactly v1.0 and rejects unknown fields. That simplifies review but makes additive evolution disruptive. A production design may allow a namespaced extension object while keeping authorization-relevant fields closed. Negotiate versions explicitly; never translate an unfamiliar outcome into pass for compatibility.

The fixture limits responses to 64 KiB and inventories to 100 checks. A real service also needs bounded reads before buffering, authenticated transport, evidence retention, durable state transitions, deadlines, and resource isolation. Its parser and validator cannot stop a trusted-looking producer from lying about work it never performed.

Finally, required-inventory coverage is not coverage of all possible defects. A complete pass establishes only the obligations selected by that policy, on the identified subject, under the represented conditions. Choosing sufficient checks remains an engineering responsibility.

The contract's job is narrower and valuable: preserve the difference between “evaluated and passed,” “evaluated and failed,” and “not established,” so a consuming system cannot accidentally authorize work by counting only the failures it happened to receive.

Download the complete Python example. Extract the archive and follow its README. The examples are synthetic demonstrations, not production benchmarks.