All postsStrategy
The CIO's Test for AI Governance
An AI policy says what agents should do. Their accounts and tools decide what they can do. A three-question test for every policy line shows a CIO which parts of AI governance hold, and which exist only on paper.
A customer service agent drafts replies and, when a customer has a case, issues a credit note. The company's AI policy says agents change no financial record without approval. At a risk review, someone asks a plain question: who approved the credit notes the agent issued last week?
The answer takes an afternoon to find. The agent runs on an integration account. That account can post credit notes, because the person who set it up copied the permissions of a customer service lead, and the lead can post them. Nothing in the system asked for an approval. The policy was right. Nothing in the systems enforced it.
It is one of the most common gaps we find, and it is not tied to one industry. Nor is it carelessness. The policy was written for people, who read it. Agents do not.
Three questions for every policy line
Take each line of the policy and ask:
- Which system enforces it? Not which team or which document. Which system says no.
- What happens when it is broken? The action is refused, a named person gets an alert, or nothing happens.
- What evidence shows it held? A record you can produce before the meeting ends.
Then mark the line: enforced, partly enforced, or written only. In our experience, many lines that touch agents come back partly enforced the first time. Often the control exists for people and was never extended to the accounts agents use.
Five lines, tested
| Policy line | Enforced by | Evidence |
|---|---|---|
| Every agent has a named business owner | The identity system issues each agent its own account, and only when the agent register names an owner. When the owner leaves and nobody takes over, the account is suspended. | Register entry linked to the account |
| Agents change no financial record without approval | The agent's account can create a draft but cannot post it. Posting needs an approver's own login, tied to that exact draft. | Approval log: who approved which change, and when |
| Customer data stays in approved regions | Agents reach AI models only through endpoints in those regions, including any fallback model. Other endpoints are blocked at the network. | Endpoint configuration and the log of blocked calls |
| We can stop any agent quickly | One switch per agent revokes its credentials and stops its queued actions from running. It is tested every quarter. | Date and result of the last test |
| A model change is a system change | Prompts, model versions and thresholds go through change control and are tested on last month's real cases before release. | Change record with test results |
The controls themselves are old: identity management, segregation of duties, network rules, change control. What is new is the actor. An agent is neither a person nor a classic service account, because it chooses its next step. So the controls must sit where the agent acts, in its account and its tools, not in its instructions.
Two details decide whether a line really holds. The first is the fallback: a data rule that covers the main model but not the one the agent switches to when the main one is busy is only partly enforced. The second is the stop switch: revoking credentials is not enough if actions already in the queue still run. Test both on purpose.
One owner per line, end to end
The test usually exposes a second problem. Legal or risk writes the line. IT builds the control. Internal audit checks the evidence. Each team does its part well, and nobody owns whether the line holds.
So give each line one owner across all three answers, usually a senior person in IT or risk. Once a quarter, that owner answers one question: does this line still hold, and how do we know? When a model changes or a new agent arrives, the owner decides whether the control still covers it.
Set the strength by the worst action
Full controls on every agent turn governance into a queue, and teams find ways around queues. So set the strength by what an agent is able to do.
- Reads and summarises. An owner and read access limited to the data in scope.
- Drafts work a person sends. Add a record of what it drafted and what the person changed.
- Changes records in a system of record. Add approval or review of its changes, and a tested stop switch.
- Moves money or commits the company to a customer, supplier or regulator. Approval before each action, and a second approver above an agreed amount.
Place each agent by the worst action it can take, not the action it usually takes. The credit note agent above belonged in the last group, though it spent most of its day drafting. Often the better fix is to remove the permission rather than add the control.
This also lets a CIO say yes faster. A summarising agent gets an answer in days, and careful review is kept for the few agents that can do real damage.
Wherever the agent came from
Agents reach an enterprise three ways: IT builds them, business teams build them on low-code platforms, and vendors add them to products the company already runs. The table stays the same. Only the source of the evidence changes. For your own agents, your teams produce it. For a vendor's agent, look to the product's settings and logs. Where they cannot yet, limit what that agent may change and raise the gap at the next product review.
What we turn down, and what we would change
We decline to build a governance dashboard before the controls exist. A dashboard that reports policy status nothing enforces gives a board false comfort, which is worse than no dashboard.
What we would change: we used to start the test with the agents IT had built, because they were documented. Now we start with the agents switched on inside products the company already runs. They are the ones least likely to be on anyone's list.
Where to start
Put the policy on one page. For each line, write the three answers and the status. Pick one line that is written only and covers the agents with the most power. Enforce it this quarter. Next quarter, run the test again and take the page to the audit committee, not the policy.
A policy says what the company intends. Accounts and tools decide what an agent does. Governance is the work of closing that gap, one line at a time.