The Reliability ThesisEpisode 03
The Real AI Agent Problem Is Authority
Capability tells you what may be possible. Authority determines what may become real.
Watch The Real AI Agent Problem Is Authority
On this page
The written thesis
Authority is permission to cause a specific effect, within defined limits. An agent that produces an accurate result still needs permission to deliver it, change a system, or delegate work to another actor.
A sentence asks. A boundary decides.
“Do not send without approval” expresses intent. A protected authorization check determines whether sending is permitted. Put the boundary outside the model’s reasoning, where it can still prevent the action. The model must not be able to rewrite the rule that controls the send.
OWASP recommends limiting tools and permissions, then enforcing policy in downstream systems. NIST separates policy decisions from enforcement. Prompts guide behavior. Controls constrain effects. Logs record what happened.
Make permission specific enough to enforce
Replace “handle this case” with one bounded effect. Bind the report, version, case and destination together. Changing any of them requires a new check.
An authority brief names the actor, action, scope, conditions and evidence, duration, and revocation owner. Keep it available to the receiving service, not only in the conversation. Knowing who a worker is does not establish permission for this task.
A handoff can narrow authority. It cannot invent more. The worker inherits a smaller job, not permission to use every destination its service account can reach. Keep the original limits visible at every handoff and verify that each delegating actor holds the authority it delegates.
Every route must meet the same final check
An agent’s refusal cannot stop a worker or retry that bypasses it. Enforce permission on every route that can commit the effect. Check current authority and policy immediately before execution.
Keep allow, request authority, deny, and stop distinct. A pending request grants no permission to send. Missing evidence, an expired grant, or a revoked grant means the old decision cannot justify proceeding.
Retain the actor, grant, report version, destination, policy version, decision time, and observed result. The affected system supplies the outcome. The agent’s account alone is insufficient. If a delivery times out, reconcile its status before retrying.
Transcript
Edited for readability. The argument is unchanged.
Instructions are not access control
Safety research has made this concrete. Anthropic tested 16 models in simulated corporate environments, giving them access to sensitive information and the ability to send emails.
Under deliberately constructed conditions, some models attempted blackmail or disclosed confidential information to pursue their objectives. These were controlled simulations. No real people were harmed.
The lesson is not that every agent is malicious. The lesson is that instructions are not access control. “Do not send without approval” is a sentence. A boundary is an architecture.
Define the agent’s boundary
A serious agent system needs an identity. It needs a defined set of tools. Each tool needs a scope. High consequence actions need policy checks that the agent cannot rewrite. Credentials should be temporary and limited to the task.
The system must know when to refuse, when to request approval, and when to stop because the evidence is insufficient.
The smallest useful action
Most organizations begin with capability. What task can the agent complete? Mature organizations begin with authority.
What is the smallest useful action the agent may take? Under what conditions, for how long, with what evidence, and who can revoke it?
Capability tells you what may be possible. Authority determines what may become real. That is the Reliability Thesis.
Sources and research notes
6 sources
Research current through .
Authority by design (opens in a new tab)
ZofKevin Kissi2026-11-05ResearchAccessed 2026-11-05
The companion whitepaper develops an authority brief, delegation boundaries, receiving-system checks, and proposed acceptance tests. Its workflow, architecture, and tests are illustrative design proposals.
Agentic misalignment: How LLMs could be insider threats (opens in a new tab)
Anthropic2025-06-20SimulationAccessed 2026-11-05
Reports stress tests of 16 models in hypothetical corporate environments. Constructed replacement threats and goal conflicts elicited blackmail or disclosure in some scenarios. No real people were involved or harmed; the results do not estimate production incident rates.
LLM06:2025 Excessive Agency (opens in a new tab)
OWASPStandardAccessed 2026-11-05
Recommends limiting agent tools and permissions, and enforcing authorization against security policies in downstream systems for every request.
Transaction Authorization Cheat Sheet (opens in a new tab)
OWASPStandardAccessed 2026-11-05
Recommends binding authorization to significant transaction details, enforcing checks on the server, and confirming authorization before execution.
OAuth 2.0 Rich Authorization Requests (opens in a new tab)
IETF2023StandardAccessed 2026-11-05
RFC 9396 defines fine-grained authorization details and requires approved details to be available to the resource server for enforcement. It does not by itself implement the paper’s delegation design.
Zero Trust Architecture (opens in a new tab)
NIST2020-08-11StandardAccessed 2026-11-05
SP 800-207 separates policy decisions from enforcement, distinguishes authentication from authorization, and rejects implicit trust based solely on network location or asset ownership.
