Regulated AI Is an Operating Problem

The servers would have no route to the internet, ever. The head of security at a national digitalisation agency said so in the first architecture meeting, and it decided the rest of the project. The agency wanted AI to sort the tickets coming into its service desk: 100% on its own premises, cut off from outside networks, with no dependency on a model vendor.

We built it that way, and it is in daily use. The lesson goes well beyond offline systems. In a regulated operation, choosing and running the model is the easy part. The hard part is what decides whether it may run at all: where data goes, what each step sees, how changes are approved, what evidence exists afterwards.

Where it runs is a design input

Teams often design around a vendor's model and ask about hosting at the end. In a regulated operation, "where may this data be processed" removes options before design starts. It decides whether a vendor's model is acceptable, in which region, under which contract, or whether models must run on the client's own hardware.

For financial firms in the EU, the Digital Operational Resilience Act (DORA) has applied since January 2025. It treats reliance on outside technology providers as a risk to be managed, documented and exited if needed, and a vendor's model is that kind of dependency. So hosting, the approved models and providers, and the way out are agreed in writing before design and go into the contract. The architecture follows.

One consequence: build where you will run. Build on a vendor's model, deploy on a local one, and they behave differently in ways tests miss. The agency's system was built in the same kind of environment as the live one.

Each step sees only the data it needs

Data minimisation is usually a policy. We make it a property of each step. A step that decides which team should handle a ticket needs the ticket text and the list of teams. It does not need the requester's identity number, their history or the attachments. Plain code assembles a view with only those fields before the model sees anything.

That reduces what could leak and puts the data flow in one table: step, data in, data out, where it runs. A data protection officer usually asks for it first, and producing it from the design is far easier than rebuilding it from logs.

We do not rely on scrubbing personal data out of free text before sending it elsewhere. Reliably removing names, addresses and ID numbers from free text is harder than it looks, and a filter that works on most tickets has still sent some out. Free text is processed where the data is allowed to be.

Instructions, thresholds and models are software

Live systems change through change control: request, test, approval, record. The model's written instructions (prompts), its thresholds and its version are part of the live system, but look like settings anyone with access can edit in a minute.

In a regulated environment they are treated like software. A change to the instructions is tested against a fixed set of real cases with known right answers and approved like any release. The model version is fixed and changes only through a release. At the agency the environment enforced this on its own: code and models entered only through the approved transfer process, so nothing could change silently. Elsewhere we make the settings part of the release.

Treating every change as a release has its own failure: if a one-word fix needs the same committee as a new model, the team stops fixing small things and the system drifts from the work. So before go-live we agree classes of change with the client's risk team, written down so nobody argues on the day a fix is needed.

One item is easy to miss: the test set itself. Every change is approved because it passes the set, so whoever can edit the set can quietly lower the bar. Cases are added through the same change process, and removing one needs the process owner's sign-off and a written reason.

Every release carries a file of evidence

When a regulator, auditor or risk committee asks how the system works, the answer should already exist. Each release carries a file of evidence: the data flow table, the models and versions in use and where they run, the test results against the previous release, the approval record and the list of changes. The release process produces it; nobody writes it for the audit. It also makes change safe for the client's own team after handover, because it records what working looked like on release day.

Record-keeping and data minimisation pull in opposite directions

Auditability pushes towards logging everything. Data protection pushes towards keeping as little personal data as possible, as briefly as possible. A system that logs full ticket text forever is very auditable and very hard to defend.

We separate what a decision needs to be explained from what it needs to be repeated. The case record keeps the decision, pointers to the evidence, the model and rule versions, and a fingerprint of the input that proves what it was without storing it. The full input stays in the source system under its existing retention rules. When that record is deleted on schedule, the decision can still be explained, though no longer re-run on the original text. The client agrees that trade-off per workflow, in the design.

There is a second reason to promise explanation rather than repetition. Even with the same model, instructions and settings, a language model does not always give the same answer twice. Horace He and colleagues at Thinking Machines Lab traced the main cause to server load, which changes how requests are grouped and with it the arithmetic behind each answer ("Defeating Nondeterminism in LLM Inference", September 2025). One changed word can change a classification. Promising a regulator to reproduce any decision on demand promises what the technology cannot reliably do. We record model version, settings, input fingerprint and output, and promise that any decision can be explained from that. That promise we can keep.

The fallback is a control

The question that ends most regulated design reviews: what if you have to switch this off today? A regulator asks after an incident elsewhere, a provider withdraws a model, or a new rule starts. DORA asks the same of every technology service behind a critical function, as an exit strategy.

So every model step has a route without the model. In ticket sorting, a ticket the model cannot classify, or every ticket while the model is off, goes to a general queue that people work as before, with whatever fields simple rules can fill already filled. Each release measures that fallback too, so the model's margin over it is on record. That answers what risk committees ask sooner or later: why use a model here at all? Switching the model off is a tested routine with a named owner, not an emergency.

The controls outlast the model

Being offline made the agency's system easier to own: fixed versions, no outside calls, no subscription to reprice, nothing leaving the building. The model inside could be replaced tomorrow. The hosting decision, the data flow, the change control and the evidence would stay, because in a regulated operation those are the system.