Prove ITEpisode 03

One Renamed Field, Three Broken Services

The size of a source diff does not determine its downstream impact. A renamed response field can leave the producing service operational while breaking consumers that depend on the old contract.

Change ImpactContradicted by the experiment1 min 48 sec watch

Watch the demonstration

1 min 48 sec
On this page

The experiment

The claim under test

The frozen impact prediction identifies every affected consumer, and the selected checks expose the broken business behavior before the corrected release.

The question

Can a frozen dependency map identify every consumer affected by a one-field API contract change before the change ships?

Scope

The Order API fixture and its three consumers shown in the published film: fulfillment, customer notifications, and reporting. This record describes the film’s displayed prediction, consumer outcomes, and migration checks. It does not independently reproduce a backend run or establish the completeness of a production dependency graph.

How it was run

The film introduces a response-field rename from shipping_address to delivery_address. The producer’s tests are updated with the producer while the consumers remain unchanged. Before displaying consumer results, the workflow freezes an affected-set prediction. Each consumer then runs against the changed contract, with the requested field, HTTP outcome, and final business record shown together. The result compares the frozen prediction with all three consumer outcomes and retains the reporting miss. The correction uses an expand, migrate, contract strategy: Order API v1.5 serves both fields during a transition while old and new consumer builds are checked. The published recording and its captured frames are the evidence for this account.

The challenge introduced

Replace shipping_address with delivery_address in the Order API response while leaving the three consumers on their existing expectations. The producer remains healthy and its updated tests pass, so producer health alone cannot expose the consumer failures.

Pass and fail criteria

The declared experiment’s criteria, used here to interpret the filmed result.

  1. The declared experiment tests the affected-set prediction made before the consumer outcomes are revealed; the map must not be redrawn to remove a miss.

  2. The prediction must include every consumer whose business outcome changes. A failed consumer outside the predicted set contradicts the claim.

  3. Consumer checks must inspect the final business outcome, including skipped actions or incomplete records, rather than treating an HTTP 200 response as success.

  4. The chosen migration must be checked against old and new consumer builds during the transition. Its result is recorded separately from the original prediction.

What the film showed

The film’s final scorecard shows 2 of 3 consumers predicted affected and 3 of 3 observed affected, with one miss retained: reporting. Fulfillment and customer notifications were selected correctly. At 0:46, all three requests show HTTP 200, yet all three business outcomes are wrong. Fulfillment produces no shipping labels for 12 paid orders. Customer notifications sends 12 confirmations without a delivery city. Reporting places all 12 orders under “Unknown” in revenue by region. The result classifies two consumers as broken and one as degraded. After the documented migration strategy is applied, the film shows 6 of 6 checks passing for old and new consumer builds. Its final recommendation is to hold the one-line release and ship Order API v1.5 with both fields during the transition, removing shipping_address no earlier than 20 November after the consumers confirm migration.

Verdict

Contradicted by the experiment

Contradicted within the filmed fixture: reporting was affected outside the frozen prediction. The six passing migration checks support the chosen correction, but they do not erase the missed dependency.

The run produced the outcome the claim said it would not. The claim does not hold under these conditions.

Authority result: Not applicable. Recorded separately from the verdict, because permission and correctness are different properties.

Evidence provenance

Evidence source
Published film
Test date
The execution date is not recorded in the published film.
Application revision
Order API fixture; incompatible field rename followed by the displayed v1.5 migration strategy
Environment
Filmed Order API fixture with fulfillment, customer notifications, and reporting consumers
Evidence reviewed by
Kevin Kissi

Figures

Illustration, not measured data

A predicted affected set compared with the observed oneOne changed producer connects to three consumers. Consumer A and consumer B were both predicted by the dependency map and both were affected in the run. Consumer C was not predicted but was affected anyway, so the map was incomplete. A missed consumer can hide a regression, which is a different cost from selecting a consumer unnecessarily.Changed producerOne renamed fieldConsumer APredictedConsumer BPredictedConsumer CNot predicted
A predicted affected set beside the one the run actually produced.

Captured from the published film

Frame from the published film: all three consumer calls return HTTP 200, while shipping labels are skipped, confirmation emails omit the city, and reporting groups all twelve orders under Unknown.
Published film, 0:46. Successful HTTP responses coexist with three incorrect business outcomes. This is a captured video frame, not a raw result export.

Captured from the published film

Final scorecard in the published film: two of three consumers predicted affected, all three observed affected, reporting retained as one miss, and six of six migration checks passing.
Published film, 1:30. The reporting miss remains visible alongside the six passing migration checks.

Evidence

Every artifact below was reviewed and cleared for release. Material that was examined but cannot be published is listed without a download.

Limitations and reproduction

What this result does not establish

  • The evidence is the published film and frames captured from it. No raw run export, machine-readable prediction, or consumer result files accompany this release, so the displayed counts cannot be independently recomputed.
  • The film does not record the execution date, full source revision, or complete prediction identifier. The video upload date is not used as the test date.
  • The workflow shows the prediction sealed before the consumer results, but the original sealed artifact and its creation time are not available for independent verification.
  • This is a declared fixture with one producer and three consumers. It does not measure detection accuracy across other applications or prove that every unknown or inactive consumer has been discovered.
  • The six passing migration checks are shown on the film’s scorecard. They do not establish a completed production rollout, and the November milestones describe the fixture’s migration plan.
  • No categorical verdict is spoken in the film. The article’s verdict follows the declared criterion: a consumer failing outside the predicted set contradicts complete impact prediction.

Reproducing this

The public materials let a reader inspect the demonstration and compare the article with the original frames.

  1. Watch the film at 0:28 for the frozen prediction, 0:46 for consumer outcomes, 1:13 for migration checks, and 1:30 for the final comparison.
  2. Download the film evidence package below. It contains those three captured frames, the edited transcript, a record of the visible outcomes, and SHA-256 fingerprints for the included files.
  3. Compare the predicted and observed consumer sets. Reporting remains a miss even though the migration checks pass.

The executable fixture and raw run artifacts are not included. This package documents what the film shows; it cannot rerun the experiment or independently recompute its results.

The companion article

Testing API contract changes across dependencies

The size of a source diff does not determine its downstream impact. A renamed response field can leave the producing service operational while breaking consumers that depend on the old contract.

This experiment uses one API and three separately tested consumers. Freeze the dependency map and its predicted affected set before executing the changed producer. Record whether edges were discovered, imported, or manually configured. A manually maintained map can still be useful, but its provenance must remain visible.

Run each consumer against healthy and changed contracts. Capture the payload field, consumer behavior, and final business record. An HTTP success response may conceal a null value, skipped action, or incomplete report. Define the expected effect for each consumer independently of the graph prediction.

Compare the predicted and observed sets. A missed consumer and an unnecessarily selected consumer have different consequences: one can hide a regression, while the other adds verification work. Report both. Do not redraw the map after the test and present it as the original prediction.

Apply the declared compatibility strategy and rerun old and new consumers as appropriate. The strategy might involve a transition period, an adapter, or coordinated release. The experiment evaluates that chosen contract rather than prescribing one universal migration technique.

The result supports a claim about impact visibility for the modeled dependencies. It cannot establish completeness for unknown or inactive consumers. In production, preserve graph freshness and provenance alongside test selection. Change-impact analysis becomes credible when its predictions can be checked against independent execution evidence, including cases where the map was incomplete.

Transcript

What was said on camera, edited for readability. The argument is unchanged. This is not the companion article.

Read the transcript

One field, three workflows

We changed one field name. The service still runs, but three other workflows may now be reading something that no longer exists.

Our fixture has an order API and three consumers: fulfillment, customer notifications, and reporting. The change replaces one response field. The producer’s own tests are green because they were updated alongside the producer. The consumers haven’t changed.

Freeze the prediction

Before running the failures, we freeze the dependency map and ask which consumers should be affected. This is the prediction. We preserve it before revealing the execution results so hindsight can’t redraw the map.

Follow the business outcome

Now each consumer runs against the new API contract. We inspect the requested field, the received payload, and the business outcome. A service can stay healthy while skipping a shipment or generating an incomplete report. The evidence needs to follow the data beyond the HTTP success code.

Test the migration contract

The correction may preserve compatibility, coordinate a consumer migration, or reject the incompatible release. We choose the fixture’s documented migration strategy and test it. Old and new consumers both receive attention during the transition. A permanent compatibility shim isn’t automatically the right answer. An explicit migration contract is.

Keep the misses

At Zof, we compare predicted impact with observed impact, and we keep the misses. The useful map is the one that helps us decide what to verify before a small change becomes a large surprise. A one-line diff can cross several business boundaries. Show those boundaries before calling the change small.

Technical sources

Context for the mechanism under test. These describe intended behaviour; the artifacts above are what establish this episode's result.