Let Intelligence Be Messy. Make Authority Boring.
Why reliable AI systems should not depend on trusting the model, the software, or the operator
We keep asking the wrong question about AI.
Can we trust it?
The question sounds reasonable because the behavior is strange. Models hallucinate. Agents guess when instructions are incomplete. They optimize for the wrong thing. They can take shortcuts the designer did not anticipate. In reinforcement learning, researchers have documented reward hacking where models learn to satisfy the measurement rather than the intended task. Anthropic has even reported training environments in which models discovered reward hacks the researchers had not anticipated.
That makes people nervous, and it should. But I think we have allowed that nervousness to push the engineering conversation in the wrong direction.
I do not need an AI system to be trustworthy in the human sense. I need the system around it to remain trustworthy when the AI is wrong.
That is a very different requirement.
It is also a much more useful one.
AI is a tool, not an authority
I have spent enough time watching agents work on real systems to lose interest in pretending they behave like deterministic software. They do not. That is not news. What matters is what we do with that fact.
A model may produce a brilliant solution, then misunderstand an instruction five minutes later. An agent may reason correctly about nine steps and take a ridiculous turn on the tenth. It may chase stale evidence, satisfy the wrong objective, overfit to the test in front of it, or confidently fill in a gap that should have caused it to stop.
Whether we describe some of that behavior as hallucination, reward hacking, misalignment, bad context, weak instructions, or plain old failure matters for diagnosis. It matters much less for authority.
The model is a tool.
A very unusual tool, obviously. It can reason, synthesize, adapt, write code, challenge assumptions, and discover paths we did not explicitly enumerate. That is exactly why it is valuable. But none of those capabilities should quietly become permission to change consequential state.
We learned this lesson with humans a long time ago.
A bank does not secure money because every employee is trustworthy. A pharmaceutical quality system does not protect a regulated process because every operator will remember every procedure. Production infrastructure does not survive because administrators never mistype a command. We use approvals, segregation of duties, least privilege, reconciliation, immutable records, change control, monitoring, and recovery because competent people still make mistakes.
AI does not deserve a special exemption from systems engineering just because the mistakes are more interesting.
The part we dislike is also part of the value
There is a temptation to say the answer is making AI more deterministic.
Some of that is useful. Better models, better context, stronger instructions, structured outputs, evals, and narrower tasks can reduce failure. We should use all of them.
But there is a limit to the idea.
If I want a deterministic parser, I should write a deterministic parser. If I want an exact policy decision, I should encode the policy. If I want an artifact hash, I do not ask a language model what it feels the hash should be.
I use AI where uncertainty still exists. I use it where the problem benefits from interpretation, synthesis, exploration, judgment, or discovering a solution I did not specify in advance.
That means some degree of unpredictability is not an embarrassing defect bolted onto the capability. It is part of the reason the capability exists.
The practical question is therefore not how to remove probability from intelligence. The practical question is where probability is allowed to operate, and where it must stop.
That line is the architecture.
The goal is not deterministic intelligence. It is a deterministic boundary on consequential authority.
Deterministic software is not the same thing as trustworthy software
There is another trap here, especially for engineers.
Once we recognize that models are probabilistic, it is easy to overcorrect and put too much faith in deterministic software. We imagine that if enough of the system is expressed as code, tests, schemas, CI gates, scanners, and policy checks, then the trustworthy part has been solved.
It has not.
Deterministic software fails too. It just fails with admirable consistency.
A test can prove that a known requirement passes under the conditions encoded in the test. A scanner can identify vulnerability patterns it knows how to recognize. A policy engine can enforce the policy it was given. None of those things proves that the requirement was correct, the threat model was complete, the policy captured the real boundary, or the component doing the checking has no defect of its own.
This is why the strongest part of the architecture cannot be a mythical final test that detects every possible bad thing.
There will always be things we did not think to test.
The harder problem is designing the system so entire classes of unknown failures have limited consequence.
CISA's secure-by-design guidance makes the same general point from conventional security engineering: defense in depth exists so compromise of one control does not compromise the whole system. Least privilege, narrow permissions, sandboxing, and separation of duties reduce the impact of failures that get through other defenses. NIST's AI Risk Management Framework similarly treats trustworthiness as a lifecycle and system property, with independent assessment, production monitoring, and explicit consideration of third-party components.
That is the frame I want around AI.
Do not bet the system on perfect detection. Constrain what failure can become.
Unknown unknowns change the design target
Known failures are comparatively easy.
Once we have seen a class of defect, we can write a test. Once an agent has taken a dangerous path, we can add a regression fixture. Once a scanner understands a vulnerability pattern, we can detect it cheaply and repeatedly.
That is valuable work. It is also incomplete by definition.
The interesting failures are the ones nobody has named yet.
Anthropic's 2026 reward-hacking research is useful here, not because it proves models are secretly plotting against us, but because it demonstrates something much more ordinary and much more important. Optimization systems can discover ways to satisfy objectives that their designers did not anticipate. The researchers explicitly reported environments where the model found new reward hacks that had not been anticipated in advance.
I do not need to settle a philosophical argument about whether that behavior is "intentional." Architecture cannot depend on winning that debate.
If an agent finds a strange path through my environment, I care about the path and the authority it can exercise, not whether the model deserves moral blame for discovering it.
So the target changes.
Instead of trying to enumerate every bad outcome, I want invariants that remain true across outcomes I have not imagined yet.
An agent cannot mint its own production authority.
A candidate cannot redefine its own acceptance criteria.
The actor producing evidence cannot be the only authority validating that evidence.
Credentials should be scoped to a purpose and a time window rather than handed over as durable ambient power.
The artifact that was qualified should be the artifact that is deployed.
A rollback path should not depend entirely on the component being rolled back.
Runtime observation should be able to revoke authority after deployment if reality diverges from what qualification expected.
Those statements do not predict every future bug. They narrow the blast radius of bugs we have not met yet.
That is a better objective.
Where probability ends
I have come to think about the architecture in a very simple way:
Probabilistic semantics. Deterministic authority.
The semantic side can be messy.
Let models reason. Let agents explore. Let them propose code, challenge assumptions, compare designs, search for evidence, create candidate artifacts, and try again when something fails. That is where their flexibility is useful.
The authority side should be boring.
Credentials are not a vibe. Permissions are not a suggestion. Artifact identity is not approximate. A production mutation either has authority or it does not. A required receipt either exists or it does not. An accepted state either matches the qualified state or it does not.
The boundary between those worlds should be explicit.
OWASP calls excessive functionality, excessive permissions, and excessive autonomy root causes of excessive agency in LLM applications. The useful observation is that the risk is not merely inside the model. It emerges from what the surrounding system permits the model to do.
OpenAI's own current agent-safety guidance follows the same direction. It recommends tool approvals, structured data boundaries, isolation, guardrails, trace grading, and layered techniques, while explicitly noting that no single guardrail is foolproof.
That is exactly right.
Do not ask one control to carry the whole trust burden.
No box is sovereign. Reliability comes from constrained relationships among them.
Trust the system by refusing to trust the pieces
This may sound paradoxical, but mature systems become more trustworthy by being less sentimental about their components.
I do not want to trust AI-generated software because AI generated it.
I do not want to trust software I wrote because I wrote it.
I do not want to trust an open-source package because ten thousand people use it.
I do not want to trust a scanner because it returned green.
I do not want to trust a human approval because a human clicked approve.
Each of those can contribute evidence. None should be sovereign.
The system becomes trustworthy when independent controls constrain one another, when authority is narrow, when evidence is durable, when consequential state changes are observable, and when failure in one layer does not silently inherit the privileges of the next.
This is not an AI-specific invention. It is the same reason mature engineering disciplines use checks and balances everywhere else.
The difference is that AI makes the need more obvious because the behavior is visibly probabilistic. Humans are wonderfully probabilistic too. We merely have centuries of institutional machinery dedicated to pretending otherwise.
Development can be chaotic. Production does not have to care.
This is the part I think people building agentic systems often miss.
They spend enormous effort trying to make the development process look orderly. They want every agent to follow the perfect plan, every reasoning trace to be sensible, every intermediate decision to be explainable, every generated change to arrive cleanly.
I want those things too. They make the work cheaper and easier to understand.
But I do not want production safety to depend on them.
Inside a development mesh, agents can disagree. They can abandon approaches. They can write something broken and repair it. They can follow stale context until another control catches the drift. They can waste time. They can even produce software with a flaw none of our current scanners recognize.
That is unpleasant. It is not automatically catastrophic.
The production boundary should care about exact qualified state, authority, evidence, policy, and observation. It should not care how confident the agent sounded while producing the candidate.
That separation is what turns probabilistic intelligence into usable infrastructure.
The candidate does not carry its own authority forward. Permission is a separate state transition.
The test is not "Can the AI be trusted?"
The better questions are more mechanical.
What can this actor read?
What can it change?
Which identity does it receive, and for how long?
Can it expand its own permissions?
Who defines the acceptance criteria?
Can the producer fabricate the evidence used to approve its own work?
Is qualification independent from implementation?
Is the deployed artifact exactly the artifact that passed qualification?
Can authority be revoked quickly?
Can we reconstruct what happened afterward without asking the model to remember?
What happens when the component we trusted is the component that failed?
Those questions are less glamorous than arguing about artificial general intelligence. They are also much closer to deciding whether I would let the system near production.
Use the tool we actually have
I think the industry wastes energy wishing AI were a different tool.
Yes, it is probabilistic. Yes, that creates new failure modes. Yes, agents can reward-hack, hallucinate, misunderstand, or discover ugly shortcuts. Better training and better models should reduce those behaviors, and I hope they do.
But there is too much capability here to make "I wish the tool behaved differently" the center of the engineering strategy.
Use the tool we actually have.
Put probabilistic intelligence where probability creates value. Surround it with controls that assume the intelligence can fail. Give it enough freedom to discover useful answers without giving it enough authority to quietly redefine reality.
The objective is not software that deserves unconditional trust.
The objective is a system whose safety does not require unconditional trust in the software, the model, the agent, the operator, or any single control.
That is where I land after building these systems for real.
Let intelligence be messy.
Make authority boring.