Model Retirement: Your Lock-In Lives in Your Test Set

The clinical lead of an AI health companion we run for women in menopause forwarded the email with one line on top: can we just switch? Below it, our model provider announced that the model version the companion depended on would be retired on a fixed date a few months away. The recommended replacement was newer, faster and better on every published benchmark.

We ran the replacement against the companion's acceptance set that afternoon. Overall accuracy went up, which was the easy part to read. It also routed some messages to the clinician portal less often, shortened the summaries clinicians read so that details they relied on dropped out, and was more willing to fill a missing detail in a user's message with a plausible guess. In a health product, those particular cases are where the risk sits.

Model lock-in is set less by the provider's contract or API than by whether you own a set of real, labelled cases that can qualify a replacement in days. Without one, a workflow tends to stay on its current model until the retirement date forces a rushed switch.

Every hosted model has a retirement date, announced or not. Anthropic's "Commitments on model deprecation and preservation" (November 2025) states that retiring past models is, for now, necessary to make new ones available, and commits to preserving the weights of every model it has publicly released. Preserved weights are not continued access. A workflow built on a model is built on a component with a known lifespan, and the architecture has to plan for replacement.

Pin the whole bundle, never an alias

Most providers offer aliases that point at the latest version of a model family. They are convenient in development and dangerous in production, because the model behind the alias can change with no change on your side. The workflow that passed its acceptance run last month may be running on a different model today.

A production workflow references an exact, dated model version. It is pinned together with everything that was tuned against it: the prompt, the output schema, the decoding settings and every threshold that routes cases. That bundle is one release, recorded in the release evidence, and changed only by a release. A rollback restores the bundle, not only the model name. Restoring an old model under a new prompt is a third configuration nobody has tested.

Pinning has a price. A pinned model receives none of the provider's later fixes, so the workflow trades silent drift for known staleness. For anything that reaches a clinician or a user, that trade is worth making, and the pin is reviewed whenever the provider publishes a fix that touches the task. For an internal draft that a person always rewrites, we would be less strict.

If a provider does not offer pinned versions with notice before retirement, a consequential workflow does not run on that provider. Yan, Bischof, Frye, Husain, Liu and Shankar gave the same advice, pin model versions in production, in "What We Learned from a Year of Building with LLMs" (O'Reilly, May 2024), citing a team whose intent classification dropped about 10% when it moved between two versions of the same model. The rule is not new. The harder part is treating the retirement that follows as a planned event with a date and a budget.

A thin interface makes the model replaceable

The workflow never talks to a model directly. It talks to a small interface per task: summarise this message for the clinician portal, classify this message into one of these routes. Each interface has a fixed input and output schema, and every response is validated against it. Behind the interface sit the model, its prompt and its settings.

The rest of the system does not know which model is in use. Swapping one is a change behind one interface, tested against that interface's cases, and two models can run side by side on the same inputs, which is how a replacement is qualified before it carries real work.

Qualification compares error profiles and churn

A candidate replacement runs on the full acceptance set, then on a recent sample of production cases with known outcomes. The headline accuracy is the least interesting output. Two numbers matter more.

The first is the error profile: which cases the old model got right and the new one gets wrong, and which fields and segments they fall in. A replacement that is better overall but newly wrong on messages in one language, or on messages that should reach a clinician, does not go live until those cases are handled, by a prompt change, a validation rule or a narrower scope.

The second is churn: the share of cases where the new model gives a different answer from the old one, whether or not either is right. Accuracy can rise while churn is high, because the new model fixes some old errors and makes new ones elsewhere. Churn is what the people in the workflow feel. A clinician who has learned which kinds of message the system summarises badly finds that the list has quietly changed, and their experience is suddenly worth less. High churn means a longer shadow run and a briefing for the reviewers, even when the accuracy line looks like good news.

The disagreement set also makes review affordable. Only cases where the two models differ need a person, so the clinical lead reviews the differences, grouped by kind, rather than the whole set, and signs off the switch as they signed off the original go-live.

Thresholds do not transfer

Every threshold in the workflow was set against the old model's behaviour: the score below which a message is routed to a clinician, the similarity at which two messages are treated as the same question. A new model distributes those scores differently. A threshold carried across unchanged is a new, untested rule wearing an old number. So every threshold is recalibrated on the qualification cases, and the recalibration is part of the change the clinical lead signs.

Every extracted value points to its source

The most dangerous difference that afternoon was not the routing or the shorter summaries. The acceptance set catches those, because they show up as differences against known answers. It was the guessed detail. A plausible made-up value passes format checks by design.

So every value a model extracts must point to where it appears in the source message: the exact passage of text. The validation layer confirms the value is actually there. A value with no location is treated as missing, not found. This check does not care which model is in use, which is exactly why it matters at a model change. It turns a new model's creative gap-filling into an ordinary, visible exception.

Replacements run in shadow first

After qualification, the candidate runs in shadow on live traffic for long enough to see the usual range of messages. It receives the same inputs as the production model, its outputs are compared field by field, and nothing it produces reaches a user or the clinician portal. Shadow running also measures cost per completed case at real volume, which a new model can move either way: a different price per token, a different token count for the same prompt, a different share of cases needing a second pass.

Work backwards from the retirement date

The retirement date is not the deadline. The deadline is the latest date the qualification can start and still finish safely. Count backwards: the retirement date, minus the sign-off, minus a shadow run long enough to see the usual range of cases, minus the qualification and prompt work. For a product like the companion, that often puts the real start date only weeks after the provider's email arrives. The handover records that calculation, so the client's team does not discover it with a month to go.

A retirement is also the cheapest moment to ask whether the task still needs the model it had. The workflow has to be qualified anyway. Sometimes a smaller model passes the same cases at a lower cost per case. Sometimes a step the model handled can now be done by deterministic code, because the information that used to arrive as free text now arrives in a structured form. The replacement is rarely only a like-for-like swap.

A better model is a riskier migration

The replacement that looks best is the one to watch most closely. When a new model is clearly stronger, reviewers relax, spot checks thin out, and the sponsor wants the switch done quickly, all at the moment the errors have moved to places nobody has learned to look. A weak replacement fails qualification and gets fixed. A strong one passes, and its few new errors land in segments where the old model was reliable and review had long since stopped. So the first weeks after a switch get more review, not less, aimed at exactly the segments where churn was highest.

The acceptance set ages faster than the model

An acceptance set built at go-live describes the people, documents and questions of that year. Two years later, part of the business it describes has gone and new parts have arrived. A replacement that passes an old set has been qualified for a company that no longer exists in that form. So the set is refreshed continuously from production: every case a person corrected becomes a candidate, and every quarter the owners add recent cases and retire ones that no longer occur. A set that has not changed for a year gives false comfort at exactly the moment it is needed.

Switching cost is set by the acceptance set

For a model, the contract and the API are the cheap part of lock-in, and with the interface in place the swap is a small code change. The expensive part is the qualification run, the review of differences, the recalibration, the prompt changes and the sign-off, and all of it leans on a set of real, labelled cases the client trusts. With a well-kept set, a switch is days of work. Without one it becomes a small project that keeps getting postponed, which is what lock-in looks like day to day. A multi-vendor abstraction layer does less for a company's independence than that set, kept current, with named owners.

The exit is part of the handover

Re-qualification is a line in the running cost of the workflow, expected at least once a year, and the client's own team runs it without calling us. The handover names the models the workflow has been qualified on, not only the one in use. Where hosting rules allow, one of them is an open-weight model the client can run on its own infrastructure, so there is always a route that does not depend on a single provider's roadmap. The handover also records the provider's notice period and the backwards calculation above. And the qualification is rehearsed once during handover, on a real candidate, while the builders are still there.

Two things we decline: upgrading because public benchmarks improved, since they contain none of the client's own cases, languages or layouts; and fine-tuning a hosted model unless the client accepts that the fine-tune retires with its base model.

A scheduled piece of maintenance

The model a workflow starts on is rarely the one it ends on, and the acceptance set is what carries across. When the next retirement email arrives, count back to the latest safe start and put that date in the calendar.