How Good Systems Earn Their Users' Trust

Three weeks after a goods-receipt system went live in a medical manufacturer's warehouse, its numbers looked healthy and the team's workload had not dropped. We spent a morning with the team to find out why. The most senior warehouse lead had a printed list on the desk beside the system. Every receipt it confirmed, she checked again by hand against the delivery: lot numbers, expiry dates, quantities.

She was not resisting change. She had done this work for years, she was accountable for what went into stock, and nothing in the system told her when it could be trusted. So she trusted it never, and checked everything. The system was working. In practice, it was also bypassed.

We now treat this as a question of calibrating trust, because trust can be measured and designed for. It is a controls problem too. The control written in the process documentation was not the control that operated: the auditor tests the system's confirmations, and the check that counted happened on a printed list nobody files.

Calibrated trust is the target

The aim is for people to trust the system as much as it deserves, case by case. Too much trust leads to rubber-stamping, which is a problem of the approval screen. Too little leads to the printed list and a team doing the work twice, which is a problem of the organisation around the system, and the subject here. Training rarely fixes it. People calibrate trust from evidence on their own cases, not from explanations of how the system works.

Two numbers show where trust is missing

We track two numbers from the first day of go-live, per segment of work and per team, never per person. In that engagement we had watched only the system's own numbers, which is why it took three weeks and a morning on the floor to find the printed list. Either of these would have shown it in the first few days.

Check rate. How often people open the delivery note, the ERP record or a list of their own before accepting a proposal they could have accepted directly. Most of it comes from data the system already has: evidence panels opened, exports taken, the time between a proposal appearing and being accepted, and edits made in the ERP directly instead of through the workflow. A high check rate on a segment where the system is reliably right means trust is lagging the evidence. That is information, not a behaviour to stamp out.

Correction latency. The time between a person correcting the system and the system changing its behaviour for that kind of case. Of the two, it is the one we watch most closely. If an expert corrects the same mistake three times and nothing changes, she concludes, correctly, that her corrections do not count, and she checks everything. If the rule changes within days and she watches the mistake disappear, trust grows faster than a demonstration could make it.

The research points the same way. Dietvorst, Simmons and Massey found in 2015 that people lose confidence in an algorithm faster than in a person after seeing both make the same mistake, and avoid it even when they have seen it outperform a person. Their follow-up in Management Science (2018) found that people were considerably more likely to use an imperfect algorithm when they could adjust its output, even slightly. In a lab, that is a slider. In an operation, it is a correction that sticks.

The correction log records both dates, and any correction open for more than a week goes to the weekly review with the name of whoever owns the fix.

Shadow mode ends on criteria the team agreed

Before a system acts on anything, it runs in shadow: it processes the same cases as the team, its proposals are recorded, and the team works as before. Each day the proposals are compared with what people actually did.

Shadow mode is common. What makes it work is how it ends. The exit criteria are agreed with the team and the process owner before it starts: how many cases each segment must see, which disagreements rule a segment out, and a review of every disagreement. Segments leave shadow one at a time as each meets the criteria, and the old way of working is retired the same way. The first segments to go live are the ones where the system is strongest, and the hard ones follow once the team has watched it be right for a while. The team watches evidence build against a standard it helped set, instead of hearing a go-live date announced.

Every disagreement ends in one of four outcomes

Once a week in the early months, the team and an engineer go through every case where the system and a person disagreed, and close each with one recorded outcome.

• The system was wrong. A rule or mapping changes, with a target of days, not weeks.
• The person was wrong. Usually a habit that has drifted from policy. The process owner decides whether the habit or the policy changes, and tells the team.
• Both were defensible. The process owner decides, and the decision becomes a written rule. These cases are the unwritten policies that lived in people's heads, and they are often the most useful to capture.
• The input was the problem. A supplier document or master data record caused it, and it goes upstream to be fixed at the source.

The balance between the four shows how the system is maturing. Early on, most disagreements tend to be system errors, and later most are policy questions. When disagreements are rare and mostly about policy, the system has earned the team's trust in that segment.

The experts become rule owners

A deeper cause of bypassing is often a change of role nobody named. Before the system, the senior warehouse lead was the person who knew how to book a difficult delivery. After it, she was a user of a tool doing her old job, with a vague duty to check it. That feels like a demotion, whatever the project plan calls it.

So roles change explicitly. The most experienced people become owners of the rules and exception causes in their area. They approve tolerance changes, decide how new supplier label formats are handled, and their names are on the rules. The system's accuracy becomes their responsibility and their achievement, and the knowledge that lived in one head becomes rules the whole team can read.

The sponsor states the purpose in writing

Most operations teams wonder whether a new system is there to replace them. If nobody answers, they answer it themselves. Before shadow mode, we ask the sponsor to write down three things and share them with the team: what the team will do more of, what it will do less of, and what the business intends to do with the capacity. In the work we take on, that purpose is capacity for growth.

We also ask for throughput targets to be suspended during the transition. People measured on receipts per hour have good reason to avoid a system that slows them down while they learn it.

Trust follows correction speed

In the end the printed list was one of the most useful things in that warehouse. It marked the cases the system had not yet earned, and it showed that her corrections were taking too long to count. Once a correction changed the system within days, she stopped printing it. Measure correction latency from the first week of go-live, and keep it under a week.