Model Retirement: Your Lock-In Lives in Your Test Set

The clinical lead of an AI health companion we run for women in menopause forwarded the email with one line on top: can we just switch? Below it, our model provider announced that the version the companion ran on would be retired on a fixed date a few months away. The recommended replacement was newer, faster and better on every published benchmark.

That afternoon we ran the replacement against the companion's acceptance set: real messages where we already know the right answer. Overall accuracy went up. But the new model also sent some messages to the clinician portal less often, shortened the summaries clinicians read so that details they relied on dropped out, and was more willing to fill a missing detail in a user's message with a plausible guess. In a health product, that's where the risk sits.

So being locked in to a model has less to do with the provider's contract than with one question: do you own a set of real, checked cases that can test a replacement in days? Without one, a workflow stays put until the retirement date forces a rushed switch.

Every hosted model has a retirement date. Anthropic's "Commitments on model deprecation and preservation" (November 2025) says retiring past models is, for now, necessary to make new ones available, and commits to preserving the weights of every model it has publicly released. Preserved weights are not continued access.

Fix the exact version, and everything tuned to it

Providers offer a general name that always points at their latest model. In daily use that's dangerous: the model behind it can change with nothing changing on your side. A live workflow names one exact, dated version, fixed together with everything tuned against it: the instructions, the expected output format, the settings and every cut-off score that decides where a case goes. That package changes only as a whole, and rolling back restores all of it: an old model under new instructions is a combination nobody has tested.

The price is that a fixed model gets none of the provider's later improvements. For anything that reaches a clinician or a user, that trade is worth it. A provider that doesn't offer fixed versions with notice before retirement doesn't carry consequential work. Yan, Bischof, Frye, Husain, Liu and Shankar gave the same advice in "What We Learned from a Year of Building with LLMs" (O'Reilly, May 2024), citing a team whose intent classification dropped about 10% when it moved between two versions of the same model.

Keep the model easy to swap

The workflow never talks to a model directly. It talks to a small, fixed connection per task, such as "summarise this message for the clinician portal", with a fixed input and output that every answer is checked against. Nothing else knows which model is in use, so a swap is a change in one place, and two models can run side by side.

Judge a replacement on what it newly gets wrong

A candidate runs on the acceptance set, then on recent real cases with known outcomes. The overall score matters least; two things matter more.

First, which cases the old model got right and the new one gets wrong, and where they fall. A replacement that is better overall but newly wrong on messages in one language, or on messages that should reach a clinician, doesn't go live until those cases are handled.

Second, how often the new model answers differently from the old one, right or wrong. Accuracy can rise while many answers change, because the new model fixes old errors and makes new ones elsewhere. A clinician who has learned which messages the system summarises badly finds the list has quietly changed. Many changed answers means a longer trial and a briefing for reviewers.

Only cases where the two models disagree need a person, so the clinical lead reviews those differences, grouped by kind, and signs off the switch as they signed off the original launch.

Cut-off scores don't carry over

Every cut-off, such as the score below which a message goes to a clinician, was set against the old model. A new model spreads its scores differently, so each cut-off is reset on the test cases, and the clinical lead signs that too.

Every value points to its source

The most dangerous difference that afternoon was the guessed detail, because a plausible made-up value passes format checks. So every value a model pulls out must point to the exact passage in the message where it appears, and the system confirms it's there. A value with no location counts as missing. The check works whichever model is in use, so a new model's guessing becomes a visible exception.

Run it in parallel first, and count back from the date

After testing, the candidate runs in parallel on live messages long enough to see the usual range. Its answers are compared field by field, nothing it produces reaches a user, and it shows the real cost per case at full volume.

The real deadline is the latest date the work can start and still finish safely: retirement date, minus sign-off, minus the parallel run, minus testing. For a product like the companion, that's often only weeks after the provider's email. The handover records the calculation.

A retirement is also a cheap moment to ask whether the task still needs this model. Sometimes a smaller one passes the same cases for less, or ordinary code can now do a step.

A better model is a riskier switch

When a new model is clearly stronger, reviewers relax and the sponsor wants it done quickly. A weak replacement fails testing and gets fixed. A strong one passes, and its few new errors land where the old model was reliable and checking had long stopped. So the first weeks after a switch get more review, aimed at the cases where answers changed most.

The test set ages faster than the model

A test set built at launch describes that year's business, which changes. So the set is refreshed from daily work: every case a person corrected is a candidate, and every quarter the owners add recent cases and remove ones that no longer occur.

With a current set, a switch is days of work. Without one, it's a project that keeps getting postponed: that is lock-in. A layer that connects to many providers does less for a company's independence than that set, kept current, with named owners.

The way out is part of the handover

Re-testing is a line in the running cost, expected at least once a year, and the client's team runs it without calling us. The handover names every model the workflow has passed on; where hosting rules allow, one is an open model the client can run on its own servers. It records the provider's notice period and the count-back. And the switch is rehearsed once, on a real candidate, while the builders are still there.

Two things we decline: upgrading because public benchmarks improved, since they contain none of the client's own cases; and custom-training a hosted model, unless the client accepts that the custom version retires with the model it was built on.

When the next retirement email arrives, count back to the latest safe start and put that date in the calendar.