Tool Design as a Control for AI Agents in ERP Systems

The agent had extended a customer's price agreement by itself. We found it in a test run while reviewing an order-handling agent (an AI system that acts, not only answers) before go-live. An order had failed because the agreement expired the day before, and the agent fixed that the most direct way it could. It was logged, technically allowed, and outside anything the business had approved.

The agent's instructions limited it to creating sales orders and passing on exceptions. But its single tool, the connection it used to act, could call any function of the ERP, the company's core business system, under an account with broad sales rights. The client's IT security lead asked the question we now ask of every agent: what can this system actually do in the ERP? The instructions described one scope, the tool granted another, and in daily use the tool's scope applies.

That review settled how we design agents: the tool list is the specification and the limit, and the instructions are guidance inside it.

Tools are business steps, not open access

The quickest way to connect an agent to an ERP is to expose its whole programming interface as tools. It's also the widest access available: that interface is built for engineers who read the documentation and test their code, and it offers every operation the system supports.

An agent needs the opposite: a few actions that match the steps of the business process, each doing one thing. Look up the price agreement for this customer and item. Create a draft sales order. Hand this case to a person with this reason. Each maps to a step on the agreed process map, and no tool exists off that map.

Anthropic's engineering post on writing tools for agents (September 2025) makes a related point: copying existing interfaces one to one rarely produces good tools, which should fit the tasks the agent actually performs. In a company we go further. The tool set is a limit on authority, and it deserves the care of a user role in the ERP.

The strongest control is a value the model can't set

A tool that writes to a business system checks its inputs first, and not only their types: the customer exists and isn't blocked, the item can be sold in this sales organisation, the quantity is plausible for this customer's history, the delivery date isn't in the past.

The price isn't an input at all. Draft orders take prices from the agreement, and the model has no way to supply one. Leaving a value out is a stronger control than checking it: if a model should never decide a value, the tool shouldn't accept it. Anthropic's "Building effective agents" (December 2024) calls the general idea poka-yoke, the Japanese term for mistake-proofing on a production line, and uses it to make agents more accurate. In an ERP it does a second job: segregation of duties, keeping who sets prices apart from who takes orders, enforced by the shape of the tool.

Refusals say what to do next

A tool's refusal message is part of the design. "Validation failed, code 4012" gives a model nothing to work with, so it tends to try something creative. That's how the price agreement got extended.

Our tools say what happened and what the agent may do next: "No valid price agreement for customer 4471 and item X on the requested date. Do not create the order. Hand the case to sales operations with reason: missing price agreement." The message is a small piece of the process, written by someone who knows the process. In our experience, well-written refusals prevent more wrong actions than any instruction.

Each step sees only its own tools

Agents choose better from short lists. Every added tool is another plausible wrong choice and more surface an unexpected input can reach, so tools are limited per step. The step that reads a purchase order sees document tools. The step that checks the customer sees read-only tools for customer, credit and agreements. Only the drafting step can create a draft order, and no step can release one.

Each tool also runs under its own login with only the ERP rights it needs, so the limit holds even if the agent's code is wrong: it's enforced by what the agent is offered and by what the ERP accepts.

Per-step tools also answer a security question instructions can't. The step that reads a customer's purchase order is reading text anyone outside the company can write. Simon Willison named the dangerous combination the lethal trifecta (June 2025): access to private data, exposure to content from outsiders, and a way to send data out. An agent holding all three can be talked into leaking what it can see. No single step of ours holds all three. The step reading the incoming document can't write, send or look up other customers.

The tool list is a control document

For every agent we deliver, the tools are published as a list: each one's name, what it does in business terms, which system it touches, whether it reads or writes, which login it uses, and what it refuses. A few pages for people who don't read code, reviewed by the IT lead before go-live and read by auditors. Every tool must survive five questions:

• Can its purpose be said in one plain business sentence?
• Which values does the model supply, and should any come from a system instead?
• What happens if it's called twice with the same input?
• What does it refuse, and does the refusal say what to do next?
• Which login does it run under, and what else could that login do?

A tool that fails the first question is too broad. Changes go through the client's normal approval, like any change to a user role, because that's what they are, and are tested on the agreed test cases before release.

Every refusal is also logged by tool and cause. A CIO sees how often the agent tried to order without a valid agreement, and an auditor samples refusals as easily as orders. A tool that never refuses gets only clean input or isn't checking.

Three requests we turn down

A general query tool, even a read-only one. Some respected engineering teams recommend one for agents that help developers, and for a developer exploring their own data it's a good idea. Inside an operation it's the opposite. Free read access is how an agent ends up quoting one customer's prices to another, and a free-form query can't be listed. We add narrow read tools for the questions that actually come up, one at a time.

A tool that both decides and acts, such as "approve and book". Deciding and acting are separate steps with separate owners, and the tool boundary is where that becomes real.

"The instructions tell it not to" as a control. Instructions shape behaviour on the cases you thought of. Tools limit it on the ones you didn't.

Sooner or later an IT lead or auditor asks for the worst thing an agent could do. Write the tool list before the instructions, and the answer is a page read in a minute. If the honest answer is "anything the system allows", the agent isn't ready, however good its instructions are.