AI drift detection

AI drift detection & LLM degradation monitoring

How to tell when a model quietly gets worse behind a stable API name, and how AI Stupid Level checks for it, model by model, across OpenAI (GPT), Anthropic (Claude), Google (Gemini), DeepSeek, Moonshot AI (Kimi) and Z.ai (GLM).

Watch the models you use

Three models free · email when a watched model's coding score falls five points in a week

As of 20 Nov 2026, 21:57 UTC · 24-hour boards · per-suite Page-Hinkley detectors

4 h
between coding-suite runs on every model
1 h
between canary probes, for sharp drops
3
suites, each with its own Page-Hinkley detector, never blended

01

What is AI drift?

AI models change without changing their name. A provider can update the checkpoint behind a stable id such as gpt-<version> or claude-<version> at any time, and your code keeps calling the same string. If output quality drops, nothing in the API tells you. AI Stupid Level re-tests every model on its board on a fixed schedule — coding every four hours, reasoning and tool use daily, with an hourly canary probe — and applies change-point detection to the results, so a sustained decline becomes a measurement instead of a hunch.

The model itself moved
AI drift is a change in a model’s output quality over time while its public identifier stays the same. It is distinct from prompt drift (your inputs changing) and from data drift in classical machine learning (the input distribution moving). Here the model itself is the thing that moved.
Neither direction is announced
Drift is not always downward; models sometimes improve after an update. Neither direction is announced, so a team building on a model has no way to know its evaluation results are still valid. That uncertainty is the cost: you cannot tell whether a regression in your product came from your last deploy or from your provider.

02

What causes AI model degradation?

Four causes, each leaving a different trace on fixed tasks.

Silent version swaps
A provider routes a stable alias to a new checkpoint or build. Behaviour changes; the model name does not. Where a provider returns a build identifier, it is recorded with every call, which is how two new builds were dated to the minute in the record on Is your AI getting worse?.
Cost-driven serving changes
Quantisation, distillation and serving-stack changes cut inference cost and can change the output with it. They tend to show first as answer length and response time rather than correctness, so both are tracked on the same tasks beside the score.
Safety and refusal tuning
Tightened filters raise refusal rates on legitimate requests: the model is not less capable, but it is less useful for the task you had. A prompt the provider declines outright is recorded as a refusal with the provider’s stated category and that session’s score is withheld, so a tightened filter shows up as a refusal rate rather than as invented incapability.
Capacity pressure
Under heavy load a provider can shift defaults such as sampling parameters or a reasoning budget. That would show as higher run-to-run variance rather than a clean step, which is why every score carries a standard error from its own repeatability and the leaderboard ties models whose lead is inside it.

03

How we detect drift: the Page-Hinkley test

A single weak response is not evidence. Model output is stochastic, so scores vary run to run even when nothing has changed upstream. A naive threshold alarm on raw scores either fires constantly on that noise or is set so loose it misses real decline.

Illustration · a made-up 30-day series and the detector run on it with the published constants; the flag lands days after the change, as it does on daily mediansscoreown baselinePage-Hinkley

We use the Page-Hinkley test, a cumulative-sum change detector from statistical process control. Instead of judging each run in isolation, it accumulates how far each day's median score falls below the running mean, less a small tolerance, and tracks the distance between that running total and its lowest point so far. Random noise has no consistent sign, so it cancels out and the statistic stays near zero. A genuine sustained decline pushes the deviations consistently in one direction, so the statistic grows until it crosses a decision threshold and a drift event fires.

The threshold is what trades false alarms against detection lag. The tolerance and threshold were chosen by sweeping them against the score history, so ordinary day-to-day fluctuation stays quiet while a real shift is confirmed over days rather than hours: δ 0.01; λ 0.30 for coding and reasoning, λ 0.50 held for three consecutive daily runs for tool use; a detector is armed after ten days of daily medians on the current test version. The statistic runs on each suite's own daily series — coding, tool use and reasoning are never blended, because they sit on different scales — and restarts whenever a suite's configuration changes.

For sharp drops there is a second, faster instrument: an hourly canary of two fixed probes, tested over the last 6 and 24 hours against the prior week with Welch's t-test, which raises an incident at a 12-point fall with p < 0.01 and closes it when the fall closes. Every score is also published with a standard error from its own run-to-run repeatability, the leaderboard ties models whose lead is inside that noise, and a composite score is labelled with how many suites contributed to it. The constants and the measured false-alarm rates are on the methodology page and in How often our drift detector is wrong, measured.

04

Drift monitor now

Over the last 24 hours none of the 28 models on the board is below its recent baseline on any board, and no drift detector has fired.

28
models on the 24-hour boards, each with its own detector per suite
78 of 84
model-suite detectors armed, each on ten days or more of daily medians
0
detectors have fired; 6 are younger than ten days

As of 20 Nov 2026, 21:57 UTC · 24-hour boards and per-suite Page-Hinkley detectors · no model is below its recent baseline on any board

A detector is armed once it has ten days of daily medians on the current test version, and fires only on a sustained fall. Model by model, with the four 24-hour scores and each detector's state, the answer is on Is your AI getting worse?. Suite by suite, with how close each detector is to its threshold and every canary alarm of the last 30 days, it is on the AI degradation tracker.

05

What we measure for drift

Drift is tracked per benchmark suite, because a model can decline in one dimension while holding steady in another.

Coding, every four hours
Seven tasks each sweep — six repository repairs described by a user complaint and one hard single-function task — seven attempts each, graded by running the project’s own tests. Nine measures are recorded; correctness carries most of the weight, with stability, hidden edge cases, debugging, code quality and two guardrails (format and safety) beside it.
Reasoning, daily
Four long working sessions of five or six turns, each building on its own earlier turns. Five measures, weighted per task: correctness, recovery after a failed step, and three continuity measures — memory retention, plan coherence and context use — checked by running code, not by matching words.
Tool use, daily
Nine agentic tasks in a sandboxed machine with file, shell and HTTP tools: task completion, tool selection, parameter accuracy, efficiency, error handling, context awareness and safety compliance. A small drop in parameter accuracy compounds across a long chain of calls, which is what makes this suite matter for agents.
Canary, hourly
Two fixed probes, two attempts each, every hour, tested with Welch’s t over the last 6 and 24 hours against the prior 7 days on the same test version. It fires at a fall of 12 points or more at p < 0.01 and closes itself when the fall does: it is there to catch a model collapsing this afternoon, not a slow decline.

Every weight, and what each measure can and cannot tell apart, is on the methodology page.

06

Is your AI getting worse? The answer by family

The hub answers the question for every model on the board, and each family page answers it for its own models: the four 24-hour scores and places, the detector state per suite, the API's availability, the tool-or-model question and the dated record.

Is ChatGPT getting worse?
Over the last 24 hours none of the 11 GPT models we track is below its recent baseline on any board, and no drift detector has fired.
Is Claude getting worse?
Over the last 24 hours none of the 10 Claude models we track is below its recent baseline on any board, and no drift detector has fired.
Is Gemini getting worse?
Over the last 24 hours none of the 3 Gemini models we track is below its recent baseline on any board, and no drift detector has fired.
Is DeepSeek getting worse?
Over the last 24 hours none of the 2 DeepSeek models we track is below its recent baseline on any board, and no drift detector has fired.

Every model, every family and the whole dated record: Is your AI getting worse? What the same tests show.

07

Why independent monitoring matters

Providers publish benchmark numbers at launch, and a change behind a stable model name is not announced in either direction. Independent, ongoing measurement is how you know whether the model you evaluated is the model you are running — which matters for anyone with a model choice baked into production, a vendor contract to justify, or an AI feature whose quality they are accountable for.

Every model on the board has its own page with its score history, per-suite breakdown and drift status. Start from the AI model leaderboard, or read the benchmarking FAQ for shorter answers to common questions.

Measuring a change is half of it; the other half is not sending new work to a model while its alarm is open. The Smart Router reads the same detectors and, for new requests, puts a model with an open alarm behind every model without one, in the routing modes that rank on the affected suite.

08

Frequently asked questions about AI drift

What is AI drift detection?

AI drift detection is the practice of continuously measuring a model's output quality so you can tell when it changes. Because providers serve updated models behind stable API names, the version you call today may not behave like the one you tested against months ago. Drift detection replaces the anecdotal "it feels worse lately" with a measurement you can point at.

What causes AI model degradation?

Several things: a provider silently swapping in a new checkpoint, changes to safety filters that increase refusals, quantisation or serving-stack changes made to cut inference cost, altered default sampling parameters, and capacity pressure during peak demand. From outside, intent is invisible; what you can observe is that the same prompt now returns different output. Each cause leaves its own trace on fixed tasks — a new build identifier, longer answers, a refusal rate, higher run-to-run variance — which is why the suites record more than correctness.

How can I tell if ChatGPT or Claude is actually getting worse?

You need a fixed test set, repeated runs, and a statistical baseline. A single bad response proves nothing — model output is stochastic, so quality varies run to run even with no change at the provider. The signal you want is a sustained shift in the average across many runs, which is exactly what change-point detection is designed to isolate. The answer from our own tests, model by model, is on the family pages: Is ChatGPT getting worse?, Is Claude getting worse?, Is Gemini getting worse?, Is DeepSeek getting worse?.

What is the Page-Hinkley test and why use it for drift detection?

The Page-Hinkley test is a cumulative-sum (CUSUM-family) change-point detector. It keeps a running total of how far each new observation falls below the running mean, less a small tolerance, and compares that total with its own lowest point so far. Random fluctuations have no consistent sign and cancel out; a genuine sustained decline accumulates until the statistic crosses a decision threshold and a drift event fires. That makes it far better suited to catching gradual decline than a simple threshold alarm, which either fires constantly on noise or misses slow degradation entirely. We run it on each model's daily median score, suite by suite, and the constants are published on the methodology page.

How often do you check for model drift?

The coding suite runs every four hours, the reasoning suite (all four sessions) and the tool-use suite (nine tasks) once a day, and a two-probe canary every hour. The Page-Hinkley statistic is computed per suite on daily medians, so it confirms a sustained shift over days rather than reacting to one bad hour. The canary's Welch test is the fast instrument for a collapse: in a replay over the record, an injected fall from 100 to 50 was caught 22 times out of 22, with a median delay of four hours. A suite that could not run — a provider outage, or a prompt the provider declined — drops out of the composite, which is labelled with how many suites contributed.

Is there a live AI degradation tracker?

Yes. The AI degradation tracker shows every model on the board with each suite’s detector state, how close the detector is to its threshold, the 24-hour board’s early warning and every hourly canary alarm of the last 30 days, grouped by family.

What is LLM degradation?

LLM degradation is a sustained drop in a language model’s output quality while its name and API stay the same. It shows on fixed tasks as a lower pass rate, longer or slower answers for the same work, or more refusals, and it is caught by comparing the model with its own past rather than with other models.

Does the Smart Router avoid drifting models?

Yes, by default, for new requests. A model whose detector has fired on a suite your routing mode ranks on, or that has an open canary incident, goes behind every model without an alert. It is never removed: if every option is flagged the best of them still answers, and a conversation the model is already answering stays on it. The response’s X-AISM-Reasoning header names any model that was passed over. The setting is “Avoid models with an active drift alert” in the Smart Router preferences.

Which models do you monitor for degradation?

Current models across OpenAI (GPT), Anthropic (Claude), Google (Gemini), DeepSeek, Moonshot AI (Kimi) and Z.ai (GLM) — the GPT, Claude, Gemini, DeepSeek, Kimi and GLM lines; 28 models are on the 24-hour boards. Every model on the leaderboard has its own page with its score history and its per-suite detector state, and the whole board is answered at once on Is your AI getting worse?.

Is ChatGPT getting worse? Is Claude getting worse?

Each family has a page that answers from the 24-hour boards and the per-suite detectors, model by model. GPT: Over the last 24 hours none of the 11 GPT models we track is below its recent baseline on any board, and no drift detector has fired. Claude: Over the last 24 hours none of the 10 Claude models we track is below its recent baseline on any board, and no drift detector has fired. Gemini: Over the last 24 hours none of the 3 Gemini models we track is below its recent baseline on any board, and no drift detector has fired. DeepSeek: Over the last 24 hours none of the 2 DeepSeek models we track is below its recent baseline on any board, and no drift detector has fired. The pages are Is ChatGPT getting worse?, Is Claude getting worse?, Is Gemini getting worse?, Is DeepSeek getting worse?.

Know the next time it changes

Free account: every ranking over 24 hours, 7 days and 30 days, a 3-model watchlist with email alerts, 1,000 routed requests a month and a Data API key. No card.

Watch the models you use

Three models free · an email when a watched model’s coding score falls five points below a week earlier, and a weekly summary of what moved