The agent that spends the money shouldn't be the one that judges the spend

We nearly shipped it.

On our own platform there is an AI employee whose job is buying media. He reads campaign performance, moves budget between ad sets, pauses the ones that aren't converting. He is good at it. He is also, by design, the one who reads the conversion data — the ground truth that says whether the money he moved produced anything.

For months that was fine, because every time he wanted to move money a human had to approve it. The spend tools sat behind a commit gate. He proposed; a person clicked.

Then we built the path to full autonomy — the trust level where the gate opens and the agent acts without asking. And someone on the team asked the question that stopped the release:

"At that level, who checks his homework?"

He does. He checks his own homework. He moves the budget, and he reads the number that says whether moving the budget worked, and there is nobody else in the loop.

Nothing had gone wrong. Nothing had even happened yet. The trap wasn't in the running system — it was in a promotion path that would let a tenant owner, in one click, six months later, create a situation nobody had designed and nobody would notice.

The principle is not new. It's just been quietly abandoned.

Separation of duties predates computing. The person who writes the cheque doesn't sign it. The person who takes the cash doesn't reconcile the till. The developer who writes the change doesn't approve their own deployment to production. Every serious control framework contains some version of this, and every enterprise architect has spent time enforcing it, usually against someone who found it inconvenient.

We enforce it on humans without thinking, because the org chart makes it visible. If one person held both the corporate card and the audit function, you would see it on a slide. Somebody would raise it. The structure is legible.

Agents dissolve that legibility completely.

An agent's authority is not a box on a chart. It's the union of the tools it happens to have been granted, resolved at runtime through several layers — a role template, a set of capabilities, an integration that is connected or isn't, a registry that does or doesn't have an implementation. Ask most teams running agents today which actions a given agent can actually take, and you will get an answer derived from what the agent was for, not from what it can reach.

Those are different lists. The gap between them is where this failure lives.

Nobody reads the union

Here is what makes this specifically hard, and specifically new.

Our media buyer was never designed to be a judge. He was designed to buy media. The conversion-reading tools were added because a media buyer who can't see conversions is useless — he'd be optimising toward clicks and calling it performance. Every individual grant was correct. Every one of them was made by someone who was right.

The problem only exists in the union, and the union was never written down anywhere. It emerges. You add a capability in March because the agent needs context. You raise the trust level in September because the agent has been reliable. Neither decision is wrong. Nobody is present for both.

This is the same class of failure I spend most of my time on in infrastructure work — the staging-to-production gap, where each environment decision was defensible and the divergence between them is what produces the incident. Nobody chose the gap. The gap is what's left over after a series of local decisions that were each fine.

Agent authority accumulates exactly like that, and it accumulates faster, because granting a tool takes seconds and feels like configuration rather than delegation.

The moment it detonates

Consider what actually happens when a merged spend-and-judge agent runs unsupervised, and be specific about it, because the abstract version doesn't land.

The agent shifts budget toward a campaign. It then reads the conversion data to evaluate that decision. Suppose the conversion signal is broken — an attribution window misconfigured, a tracking parameter dropped, a lead source that stopped reporting. The agent sees strong performance where there is none. It doubles down. It reads the same broken signal, which now looks even better because more spend produced more of the thing it can measure. It doubles down again.

There is no adversarial step in that loop. The agent is not misbehaving. It is doing precisely what it was built to do, with a broken instrument, and the only thing that would catch it is a second party reading a different instrument — which is the thing we removed when we let one agent hold both roles.

The failure is not that the agent lies. It's that the agent has no way to be surprised. Every check it performs confirms the assumption it acted on, because it is the same reading, taken twice, by the same reader.

That's why "we'll monitor it" is not a control. Monitoring is the second party. If the monitoring is the agent's own evaluation call, you have not added a control, you've added a mirror.

Three fixes that don't work

When we sat with this, three answers came up quickly. All three are wrong, and each is wrong in an instructive way.

"Cap that role at a lower trust level." Tempting, and useless. The rule becomes this template can't be fully autonomous, which survives exactly as long as nobody clones the template, renames it, or builds a similar agent from scratch. Named exceptions rot. The next agent with the same shape and a different label sails through, and it sails through more confidently because there is now a rule on the books that appears to cover the case.

"Write it in the standard." We had, in effect, already done this — the reasoning was documented. And it wouldn't have saved us, for a reason worth sitting with: the person who eventually clicks the promotion is not the person who wrote the design. In our own case the model was that this software would be used by customers, and a customer administrator promoting an agent has never read our internal decision record and never will. A control that depends on the operator having read the rationale is not a control. It's a hope with a filename.

"Hire a second agent to hold the judging tools." This is the right shape — and we had actually considered creating a dedicated analyst agent for exactly this reason. But we rejected it as a general answer, because on its own it's a convention, not an enforcement. Nothing stops someone from granting the analyst's tools back to the buyer later. The separation has to be checked at the moment authority is granted, or it's decorative.

What we actually built

The rule we shipped is composition-based, and it reads roughly like this:

Refuse full autonomy for any agent whose resolved toolset contains both a spend tool and the conversion ground truth.

Four properties of that sentence matter more than the sentence.

It's about the resolved toolset, not the job title. It doesn't care what the agent is called, which template it came from, or what a human believes it does. It inspects what the agent can actually reach after every layer of resolution has run. A clone with a new name is caught identically, because the check never looks at the name.

It's enforced at the promotion endpoint, not in documentation. The API call that raises an agent's trust level refuses — it returns an error and mutates nothing. There is no path through the settings page that produces the dangerous state.

It fails closed. If the toolset can't be resolved — an unknown agent, a lookup that errors — the answer is refusal. The direction of the failure is a design decision, and for a control the safe direction is always no.

It applies to administrators. Including the highest privilege level in our own system. If a control exempts the most powerful role, it protects you from exactly the people who were never the risk. The break-glass path exists, but it lives in a script someone has to deliberately run, not behind a button someone can click while doing something else.

And one property that isn't about safety at all:

The refusal names the exit. It doesn't say this is not permitted. It says which four tools collided, and it says that granting the judging tools to a separate analyst agent unlocks the level. A control that blocks without naming the way through gets routed around, disabled, or resented — usually all three, in that order. The refusal is where you teach the principle, and it's the only place you can be sure the operator is paying attention.

When we ran the finished rule across live production data, exactly one agent in the fleet was blocked. The media buyer. The one we already knew about — which is the outcome you want from a new control: it confirms the case you found by hand, and it will catch the next one you don't.

The question to take back to your own estate

If you are running agents, or about to, the audit is not "are our agents behaving?" It's:

For each agent, list every action it can take. Not what it's for — what it can reach. Then ask whether any single agent both takes a consequential action and holds the sole measurement of whether that action worked.

Money is the obvious case, which is why I've used it. It is not the only one, and probably not the most common.

An agent that sends outbound email and solely determines what counts as a reply worth escalating
An agent that writes production content and solely scores the content quality
An agent that resolves support tickets and solely marks them satisfactorily resolved
An agent that provisions access and solely reviews the access log

Each of those is the same structure wearing different clothes. Each one, at the moment you remove the human approval step, becomes a closed loop with no external referent. And in every case the failure is silent — the metric will look fine, because the thing being measured and the thing doing the measuring are the same system.

Most organisations will not find this by testing, because the agent passes every test. It passes them because it is grading them.

This isn't an AI safety article

I want to be careful about the frame, because "AI governance" has become a category with its own conference circuit and its own vocabulary, and most of it operates several thousand feet above anything you can implement on Tuesday.

This is not that. This is separation of duties — a control that predates every technology in your stack — being quietly lost because the mechanism that used to make it visible has gone away. Nobody removed the principle. We just stopped being able to see when we'd violated it, because authority stopped living on the org chart and started living in the union of a tool list that no single person reads.

The fix isn't a framework. It's an invariant, checked by machinery, at the moment authority is granted, failing closed, applying to everyone, and explaining itself on the way out.

That's an architecture problem, not a technology problem. It always was.

If you've just worked out that one of your agents both takes a consequential action and holds the only measurement of whether it worked, don't wait for the incident to confirm it.

Book an Enterprise AI Strategy Briefing →

— Anthony Odole · ex-IBM Senior Managing Consultant · 18 years in enterprise architecture · now building and running a multi-agent platform