Agents Propose, Humans Authorize: The Principle Behind Governed Autonomy
Why \"agents propose, humans authorize\" is the founding design rule that separates a credible reliability control layer from reckless autonomous fixing.
The asymmetry that breaks naive autonomy
Reliability work is not one task. It is a loop: understand the system, test it, reproduce the failure, remediate, then verify the fix held. Four of those five steps are safe to automate aggressively. Mapping a dependency graph, running validation, reproducing a defect in a sandbox, confirming a fix against evidence, none of these mutate production. Get them wrong and you waste compute, not customers.
Remediation is the exception, and the asymmetry is total. A proposed fix that is wrong does not merely fail to help. It actively changes a running system in a direction nobody verified. In a fintech stack, "a direction nobody verified" includes the authorization path, the ledger, the reconciliation job, the thing that decides whether a transaction settles. The cost of a bad autonomous read is zero. The cost of a bad autonomous write is unbounded.
This is why "fully autonomous remediation" is not a feature to aspire to. It is a category error. It collapses the safest and the most dangerous steps in the loop into one undifferentiated capability and ships them under the same trust assumption. A serious enterprise does not want more AI doing more things unsupervised. It wants control over the one step where being wrong is catastrophic. The discipline is knowing exactly where to draw that line and refusing to move it for convenience.
Why "trust the model" was never a control
The instinct of vendors selling unsupervised fixing is to argue the models are good enough now. The data does not support resting your control posture on model quality. Industry research puts roughly 41% of codebases as AI-generated, and around 45% of AI coding tasks introduce a critical flaw or security issue. The cost of poor software quality sits near $2.41 trillion. A better model shaves the defect rate; it does not zero it. And you cannot audit a probability distribution.
That last sentence is the whole argument for a security lead. Your job is not to maximize the expected value of a change pipeline. It is to be able to stand in front of a regulator, an auditor, or your own board and demonstrate that every change to a sensitive system was authorized by someone accountable, on evidence that still exists. "The model was usually right" is not a control. It is a confession.
There is a behavioral failure underneath the technical one. Roughly 80% of developers bypass policy or guardrails when those guardrails add friction. This is rarely malice, it is the rational response to governance that lives in a wiki, a checklist, or a Friday change-advisory meeting. Any authority model that depends on humans remembering to follow a document will be skipped at the exact moment it matters most. The implication runs both directions: governance that humans route around is theater, and so is autonomy that no human can see or stop. The fix for both is the same, encode the authority into the system where it cannot be skipped, and make the human decision a fast, evidence-rich gate rather than a meeting.
What "propose" has to mean
The principle only holds if "propose" is rich. A proposal that arrives as a raw diff with no context forces the human reviewer back into the role the autonomy was supposed to remove: guessing at blast radius from a code change. That is how you get rubber-stamping, which is the bypass failure wearing a tie.
A credible proposal carries its own evidence. It should state, before any human looks at it:
- What it touches and what depends on that. Computed from a live System Graph that maps services, dependencies, and CI/CD into one change-aware model, so the blast radius is derived, not estimated from lines changed.
- What was validated and how. Coordinated Testing Fleets that plan and execute validation aware of the change and its dependents, not a static suite that ignores the graph. The reviewer reads which paths were exercised and what regressed.
- What the real exposure is. Reachability-based prioritization, asking whether a flaw sits on a path actually reachable in your deployed system, can mean 70 to 90% less exploitable exposure to triage. A reachable vulnerability routes to a strict human gate; an unreachable one does not have to block a release.
- What the fix proposes to do, reversibly. The remediation is staged, not applied. Remediation Fleets do the hard work of constructing the fix; the authority to merge it stays with a named human.
When the proposal is this complete, the human decision compresses from "investigate this change" to "authorize this verdict." That is the difference between a gate that scales and a queue that drowns.
Governance is the engineering, not the paperwork
The reflex in many orgs is to treat policy, approval, and audit as compliance overhead bolted on after the engineering is done. Invert that. In a control layer, governance is the engineering, it is the part that is genuinely hard, and the part that decides whether autonomy is an asset or a liability.
Concretely, that means three properties have to be first-class, not configurable afterthoughts:
- Authority is derived, not declared. The system, not the change author, decides whether a change needs one approver, two, or a change-control step, based on what it touches in the graph and what policy says, so no one can self-assign a lower tier to skip review.
- The audit trail is mandatory and tamper-evident. Every action, proposed or authorized, leaves a record. The absence of a human in a low-risk auto-merge raises the bar on the evidence trail; it does not lower it.
- Evidence survives the boundary. For changes executing inside a regulated environment, Edge Runners run as signed capsules inside the customer enclave and emit audit-ready evidence from inside that boundary, so the record holds up in a compliance review instead of living in a CI log someone can edit.
This is what separates governed autonomy from reckless autonomy in one sentence: in a governed system, the agent's authority to change production is something you configured, can inspect, and can prove. In a reckless one, it is something you hoped.
What to do Monday morning
You do not need to re-architect anything to apply the principle. Start with three moves a security lead can drive directly.
- Write the no-autonomy list. Name the surfaces where a human always authorizes: authentication, authorization, payments, the ledger, irreversible operations, regulated data. Be conservative here and only here. Everything not on it is a candidate for governed automation.
- Audit one existing autonomous tool for provability. Pick a bot already acting in your pipeline and ask: can it show you what it changed, on what evidence, and who authorized it? If the answer is no, you have ungoverned autonomy in production today.
- Require evidence with every proposal. Mandate that any proposed change, human or agent, arrive with blast radius, validation results, and reachability before it reaches a gate. Make the gate read evidence, not vibes.
For the broader picture of where this fits, the build-vs-buy tradeoff for a governed control layer is worth its own look, as is the financial-services view of these controls under regulatory load.
The bottom line
Verwandte Leitfäden
Verwandtes Produkt
Lesen Sie weiter
From Microsoft Scale to a New Category: How TAS23 Became Zof
The founder arc behind Zof: running engineering at Microsoft scale, a 2023 conference talk, and the reframe from QA tooling to governed reliability infrastructure.
The Closed Loop: Why Reliability Is Five Steps, Not One Tool
A founder's case for why reliability is an operating loop, not a tool: Understand, Test, Reproduce, Remediate, Verify, built for SREs drowning in AI-speed change.
The Silent Enemy: A First-Principles Look at the Cost of Rework
Rework, not slow developers, is what kills engineering momentum. A first-principles look at why it scales with AI-generated code and how to attack it at the source.
