silent upgrades · what drifts · the canary file · what to write down
Your instructions did not change and the output did. Nobody told you, nothing broke, and the difference is small enough that you will notice it in about three weeks — usually from somebody else, about a piece that already went out.
Not everything, and not randomly. The parts that drift are the parts you never specified, which is why the fix is upstream of the drift rather than downstream of it.
Drift is not a failure of the tool and not a reason to distrust it; it is what happens to anything you left unspecified, and it was going to happen the first time somebody else did the work too.
People look for: AI model drift, changed AI output, model update testing, prompt stability, and version records.
The left column is what people worry about. The right column is what actually happens. The difference between the two rows is whether you ever wrote the thing down.
| The part of your output | Does it drift? | Why |
|---|---|---|
| How long the answer runs | Yes | You never gave a length, so it uses its own default and the default moved |
| How much it hedges | Yes, and most visibly | Caution is tuned deliberately between versions. Nothing in your ask fixed it |
| How it opens | Yes | An unspecified opening is a house style, and the house changed |
| Whether it follows a rule with a test in it | Rarely | A rule that can be marked right or wrong leaves nowhere to drift to |
| Whether it keeps your named things | No, if you said so | Keep every proper noun, number and date is checkable, so it holds |
| Whether the facts are right | Not in the way you fear | It was never a reliable source of facts. That did not change, and it is not drift |
One file, five items, run on the first of the month. Each item is a small ask whose right answer you already know: a passage that must keep three named things, a rule that must be reported and not fixed, a question your standard says to refuse. Run it, read it, and either nothing moved or you have found out in a minute rather than in a client’s email.
Drift appears most clearly in requirements that were never made testable; controlled examples separate meaningful change from ordinary variation.
Everything you did not decide was decided for you, and defaults are not promises. When the default moves you experience it as the tool changing, and what actually happened is that a gap in your instructions became visible.
Keep it concise drifts because nothing in it can be marked. No paragraph over sixty words cannot drift, because there is a right answer and the model either meets it or does not. Every rule you tighten is a piece of your work taken out of the weather.
Which model produced the work, and on what date. One line in the record from day 20, costing nothing. Six months later, when somebody asks why two pieces from the same quarter read differently, that line is the whole answer.
A monthly report began opening with longer summaries after an upgrade. Its required fields still held; only the unspecified introduction moved, so the team added one measurable opening rule.
Two reports used the same brief but different model versions. The record explained the change, and a small style canary showed which uncontrolled defaults had moved.
A previously accepted workflow began refusing a harmless transformation. A narrow test confirmed the boundary change, allowing the team to revise that step without rebuilding everything else.
Model drift is a noticeable change in output after the provider replaces or retunes the model while your instructions and source material stay the same. It often appears in unspecified qualities such as length, tone, structure, hedging, or refusal behaviour rather than in every part of the work.
A prompt fixes only what it states and demonstrates. Everything left open is handled by the model's current defaults, and those defaults can change between versions. Sampling also creates ordinary variation, so compare several controlled outputs before deciding that an update caused the difference.
No, but it can reduce the part that matters. Write important requirements as testable rules, keep a small set of known-input examples, record the model and date used for consequential work, and review the outputs that carry financial, legal, safety, or reputational risk.
Use a handful of short tasks whose acceptable behaviour is already known: preserve named facts, follow a measurable style rule, report rather than repair a defect, and refuse only where the standard requires it. The file detects meaningful movement quickly; it is not a general model benchmark.
Rerun work only when future outputs depend on the changed behaviour or when the original result must remain reproducible. Published prose usually does not need regeneration. Templates, automations, regulated material, high-volume workflows, and safety-sensitive decisions deserve targeted retesting.
Anything you did not specify is on loan, and the lender can call it back.