AI drift detection
How to tell when a model quietly gets worse behind a stable API name, and how AI Stupid Level checks for it, model by model, across OpenAI (GPT), Anthropic (Claude), Google (Gemini), DeepSeek, Moonshot AI (Kimi) and Z.ai (GLM).
Three models free · email when a watched model's coding score falls five points in a week
As of 20 Nov 2026, 21:57 UTC · 24-hour boards · per-suite Page-Hinkley detectors
01
AI models change without changing their name. A provider can update the checkpoint behind a stable id such as gpt-<version> or claude-<version> at any time, and your code keeps calling the same string. If output quality drops, nothing in the API tells you. AI Stupid Level re-tests every model on its board on a fixed schedule — coding every four hours, reasoning and tool use daily, with an hourly canary probe — and applies change-point detection to the results, so a sustained decline becomes a measurement instead of a hunch.
02
Four causes, each leaving a different trace on fixed tasks.
03
A single weak response is not evidence. Model output is stochastic, so scores vary run to run even when nothing has changed upstream. A naive threshold alarm on raw scores either fires constantly on that noise or is set so loose it misses real decline.
We use the Page-Hinkley test, a cumulative-sum change detector from statistical process control. Instead of judging each run in isolation, it accumulates how far each day's median score falls below the running mean, less a small tolerance, and tracks the distance between that running total and its lowest point so far. Random noise has no consistent sign, so it cancels out and the statistic stays near zero. A genuine sustained decline pushes the deviations consistently in one direction, so the statistic grows until it crosses a decision threshold and a drift event fires.
The threshold is what trades false alarms against detection lag. The tolerance and threshold were chosen by sweeping them against the score history, so ordinary day-to-day fluctuation stays quiet while a real shift is confirmed over days rather than hours: δ 0.01; λ 0.30 for coding and reasoning, λ 0.50 held for three consecutive daily runs for tool use; a detector is armed after ten days of daily medians on the current test version. The statistic runs on each suite's own daily series — coding, tool use and reasoning are never blended, because they sit on different scales — and restarts whenever a suite's configuration changes.
For sharp drops there is a second, faster instrument: an hourly canary of two fixed probes, tested over the last 6 and 24 hours against the prior week with Welch's t-test, which raises an incident at a 12-point fall with p < 0.01 and closes it when the fall closes. Every score is also published with a standard error from its own run-to-run repeatability, the leaderboard ties models whose lead is inside that noise, and a composite score is labelled with how many suites contributed to it. The constants and the measured false-alarm rates are on the methodology page and in How often our drift detector is wrong, measured.
04
Over the last 24 hours none of the 28 models on the board is below its recent baseline on any board, and no drift detector has fired.
As of 20 Nov 2026, 21:57 UTC · 24-hour boards and per-suite Page-Hinkley detectors · no model is below its recent baseline on any board
A detector is armed once it has ten days of daily medians on the current test version, and fires only on a sustained fall. Model by model, with the four 24-hour scores and each detector's state, the answer is on Is your AI getting worse?. Suite by suite, with how close each detector is to its threshold and every canary alarm of the last 30 days, it is on the AI degradation tracker.
05
Drift is tracked per benchmark suite, because a model can decline in one dimension while holding steady in another.
Every weight, and what each measure can and cannot tell apart, is on the methodology page.
06
The hub answers the question for every model on the board, and each family page answers it for its own models: the four 24-hour scores and places, the detector state per suite, the API's availability, the tool-or-model question and the dated record.
Every model, every family and the whole dated record: Is your AI getting worse? What the same tests show.
07
Providers publish benchmark numbers at launch, and a change behind a stable model name is not announced in either direction. Independent, ongoing measurement is how you know whether the model you evaluated is the model you are running — which matters for anyone with a model choice baked into production, a vendor contract to justify, or an AI feature whose quality they are accountable for.
Every model on the board has its own page with its score history, per-suite breakdown and drift status. Start from the AI model leaderboard, or read the benchmarking FAQ for shorter answers to common questions.
Measuring a change is half of it; the other half is not sending new work to a model while its alarm is open. The Smart Router reads the same detectors and, for new requests, puts a model with an open alarm behind every model without one, in the routing modes that rank on the affected suite.
08
AI drift detection is the practice of continuously measuring a model's output quality so you can tell when it changes. Because providers serve updated models behind stable API names, the version you call today may not behave like the one you tested against months ago. Drift detection replaces the anecdotal "it feels worse lately" with a measurement you can point at.
Several things: a provider silently swapping in a new checkpoint, changes to safety filters that increase refusals, quantisation or serving-stack changes made to cut inference cost, altered default sampling parameters, and capacity pressure during peak demand. From outside, intent is invisible; what you can observe is that the same prompt now returns different output. Each cause leaves its own trace on fixed tasks — a new build identifier, longer answers, a refusal rate, higher run-to-run variance — which is why the suites record more than correctness.
You need a fixed test set, repeated runs, and a statistical baseline. A single bad response proves nothing — model output is stochastic, so quality varies run to run even with no change at the provider. The signal you want is a sustained shift in the average across many runs, which is exactly what change-point detection is designed to isolate. The answer from our own tests, model by model, is on the family pages: Is ChatGPT getting worse?, Is Claude getting worse?, Is Gemini getting worse?, Is DeepSeek getting worse?.
The Page-Hinkley test is a cumulative-sum (CUSUM-family) change-point detector. It keeps a running total of how far each new observation falls below the running mean, less a small tolerance, and compares that total with its own lowest point so far. Random fluctuations have no consistent sign and cancel out; a genuine sustained decline accumulates until the statistic crosses a decision threshold and a drift event fires. That makes it far better suited to catching gradual decline than a simple threshold alarm, which either fires constantly on noise or misses slow degradation entirely. We run it on each model's daily median score, suite by suite, and the constants are published on the methodology page.
The coding suite runs every four hours, the reasoning suite (all four sessions) and the tool-use suite (nine tasks) once a day, and a two-probe canary every hour. The Page-Hinkley statistic is computed per suite on daily medians, so it confirms a sustained shift over days rather than reacting to one bad hour. The canary's Welch test is the fast instrument for a collapse: in a replay over the record, an injected fall from 100 to 50 was caught 22 times out of 22, with a median delay of four hours. A suite that could not run — a provider outage, or a prompt the provider declined — drops out of the composite, which is labelled with how many suites contributed.
Yes. The AI degradation tracker shows every model on the board with each suite’s detector state, how close the detector is to its threshold, the 24-hour board’s early warning and every hourly canary alarm of the last 30 days, grouped by family.
LLM degradation is a sustained drop in a language model’s output quality while its name and API stay the same. It shows on fixed tasks as a lower pass rate, longer or slower answers for the same work, or more refusals, and it is caught by comparing the model with its own past rather than with other models.
Yes, by default, for new requests. A model whose detector has fired on a suite your routing mode ranks on, or that has an open canary incident, goes behind every model without an alert. It is never removed: if every option is flagged the best of them still answers, and a conversation the model is already answering stays on it. The response’s X-AISM-Reasoning header names any model that was passed over. The setting is “Avoid models with an active drift alert” in the Smart Router preferences.
Current models across OpenAI (GPT), Anthropic (Claude), Google (Gemini), DeepSeek, Moonshot AI (Kimi) and Z.ai (GLM) — the GPT, Claude, Gemini, DeepSeek, Kimi and GLM lines; 28 models are on the 24-hour boards. Every model on the leaderboard has its own page with its score history and its per-suite detector state, and the whole board is answered at once on Is your AI getting worse?.
Each family has a page that answers from the 24-hour boards and the per-suite detectors, model by model. GPT: Over the last 24 hours none of the 11 GPT models we track is below its recent baseline on any board, and no drift detector has fired. Claude: Over the last 24 hours none of the 10 Claude models we track is below its recent baseline on any board, and no drift detector has fired. Gemini: Over the last 24 hours none of the 3 Gemini models we track is below its recent baseline on any board, and no drift detector has fired. DeepSeek: Over the last 24 hours none of the 2 DeepSeek models we track is below its recent baseline on any board, and no drift detector has fired. The pages are Is ChatGPT getting worse?, Is Claude getting worse?, Is Gemini getting worse?, Is DeepSeek getting worse?.
Free account: every ranking over 24 hours, 7 days and 30 days, a 3-model watchlist with email alerts, 1,000 routed requests a month and a Data API key. No card.
Three models free · an email when a watched model’s coding score falls five points below a week earlier, and a weekly summary of what moved