How Good Systems Earn Their Users' Trust

Three weeks after a system for checking incoming deliveries went live in a medical manufacturer's warehouse, its numbers looked healthy and the team's workload had not dropped. We spent a morning with the team to find out why. The most senior warehouse lead had a printed list on the desk beside the system. Every delivery it confirmed, she checked again by hand: batch numbers, expiry dates, quantities.

She was not resisting change. She had done this work for years, she was accountable for what went into stock, and nothing in the system told her when it could be trusted. So she trusted it never, and checked everything. The system was working. In practice, it was also bypassed.

We now treat this as a question of matching trust to evidence, because trust can be measured and designed for. It is a control problem too: the auditor tests the system's confirmations, and the check that counted happened on a printed list nobody files.

The target is trust that matches the evidence

The aim is for people to trust the system as much as it deserves, case by case. Too much trust leads to approving without looking. Too little leads to the printed list and a team doing the work twice, the subject here. Training rarely fixes it. People build trust from evidence on their own cases, not from explanations of how the system works.

Two numbers show where trust is missing

We track two numbers from the first day of go-live, per type of work and per team, never per person. In that warehouse we had watched only the system's own accuracy, which is why it took three weeks and a morning on the floor to find the list. Either number would have shown it in the first few days.

How often people re-check. How often someone opens the delivery note, the record in the core business system or a list of their own before accepting a proposal they could have accepted directly. Most of this comes from data the system already has: evidence opened, exports taken, time before accepting, edits made around the workflow. A high rate where the system is reliably right means trust is lagging the evidence. That is information, not behaviour to stamp out.

How fast a correction counts. The time between a person correcting the system and the system changing its behaviour for that kind of case. We watch this one most closely. If an expert corrects the same mistake three times and nothing changes, she concludes, correctly, that her corrections do not count, and she checks everything. If the rule changes within days and she watches the mistake disappear, trust grows faster than any demonstration could make it.

The research points the same way. Dietvorst, Simmons and Massey found in 2015 that people lose confidence in an algorithm faster than in a person after seeing both make the same mistake, and avoid it even when they have seen it outperform a person. Their follow-up in Management Science (2018) found people were considerably more likely to use an imperfect algorithm when they could adjust its output, even slightly. In a lab, that is a slider. In an operation, it is a correction that sticks.

The correction log records both dates, and any correction open for more than a week goes to the weekly review with the name of whoever owns the fix.

The trial run ends on criteria the team agreed

Before a system acts on anything, it runs alongside the team in a trial, often called shadow mode: it handles the same cases, its proposals are recorded, and the team works as before. Each day the proposals are compared with what people actually did.

What makes a trial run work is how it ends. The criteria for ending it are agreed with the team and the process owner before it starts: how many cases each type of work must see, which disagreements rule a type out, and a review of every disagreement. Types of work leave the trial one at a time as each meets the criteria, starting where the system is strongest; the hard ones follow once the team has watched it be right for a while. The team sees evidence build against a standard it helped set, instead of hearing a go-live date announced.

Every disagreement ends in one of four outcomes

Once a week in the early months, the team and an engineer go through every case where the system and a person disagreed, and close each with one recorded outcome.

• The system was wrong. A rule changes, with a target of days, not weeks.
• The person was wrong. Usually a habit that has drifted from policy. The process owner decides whether the habit or the policy changes, and tells the team.
• Both were defensible. The process owner decides, and the decision becomes a written rule. These are the unwritten policies that lived in people's heads, often the most useful thing to capture.
• The input was the problem. A supplier document or a wrong record in the company's reference data caused it, and it is fixed at the source.

Early on, most disagreements tend to be system errors; later, most are policy questions. When disagreements are rare and mostly about policy, the system has earned the team's trust for that type of work.

The experts become rule owners

A deeper cause of bypassing is often a change of role nobody named. Before the system, the senior warehouse lead was the person who knew how to book in a difficult delivery. After it, she was a user of a tool doing her old job, with a vague duty to check it. That feels like a demotion, whatever the project plan calls it.

So roles change explicitly. The most experienced people become owners of the rules and the common causes of exceptions in their area. They approve changes to what counts as close enough, decide how new supplier label formats are handled, and their names are on the rules. The system's accuracy becomes their achievement, and knowledge that lived in one head becomes rules the whole team can read.

The sponsor states the purpose in writing

Most teams wonder whether a new system is there to replace them, and if nobody answers, they answer it themselves. Before the trial, we ask the sponsor to write down three things and share them with the team: what the team will do more of, what it will do less of, and what the business intends to do with the capacity. In the work we take on, that purpose is capacity for growth.

We also ask for output targets to be suspended during the transition. People measured on deliveries checked per hour have good reason to avoid a system that slows them down while they learn it.

Trust follows correction speed

In the end the printed list was one of the most useful things in that warehouse. It marked the cases the system had not yet earned, and it showed that her corrections were taking too long to count. Once a correction changed the system within days, she stopped printing it. Measure how fast corrections count from the first week of go-live, and keep it under a week.