What 13 months of monitoring LLM APIs caught

1. Provider-side changes are real and observable. The record holds two build changes confirmed by a provider’s own identifiers, a requested model name silently redirected to a different served model, five isolated serving-speed steps of 28–51% with no change on our side, and four endpoint withdrawals.
2. None of them was a loss of correctness. On the nine tasks that ran all year, no model showed a sustained, model-specific fall in pass rate of five points or more, a result bounded by how close to ceiling those tasks sit.
3. The costliest changes were invisible to a correctness-dominated score. Two new builds doubled mean output length and raised latency by about 1.7× at unchanged pass rates, so the same work cost about twice as much and took 70% longer under an unchanged model name.
4. A fixed output budget turns verbosity into failure. When one model began answering 26% longer in March 2026, the share of its answers cut off at our 1,500-token cap rose from 23.7% to 39.5% and its pass rate fell from 93.1% to 87.2%.
5. Most apparent degradation was the instrument. Against roughly a dozen documented provider-side events, our earlier alerting layers raised 937 incidents, 492 of them since retracted, and a change-point scanner whose firings near the real events matched what chance predicts.

Two new builds, confirmed by the provider’s identifier

Between 16 June and 23 September 2026, the period with identifiers, only DeepSeek’s changed under a fixed requested name; it is the one vendor whose identifier encodes a build, and the one we measured continuously through July and August.

Our step detector found both changes independently, and no other isolated step in output length appears after March 2026. In each case output doubled and latency rose by 1.7× while correctness held or improved slightly: for a user paying per output token, the same work cost about twice as much and took 70% longer, under an unchanged model name. We checked our own side for the July event. The new build first appears at 02:10 UTC on 31 July; our two commits that day landed eight and eleven hours later and did not touch the DeepSeek request path.

The alerting layer of the time saw neither event for what it was. The composite score registered only the July event, and in the opposite direction, because the new build stopped failing the one task that had been collapsing a sweep’s score; the second left the composite flat. The change-point scanner fired on 40 of 86 days for one model and 47 of 86 for the other, and near the events about as often as chance predicted. The per-call record had captured both events precisely; the score built on it could not read them.

A model name redirected, 10 September 2026

From 10 September 2026, requests for deepseek-v4-flash were answered by a model identifying itself as deepseek-flash. The name we were paying to measure had been redirected to a different served model. Our coding corpus changed three days later, so the behavioural effect cannot be measured cleanly.

Verbosity that became failure: GLM-4.6, March 2026

Our coverage of GLM-4.6 has a gap from 17 February to 10 March 2026. When measurement resumed, its answers on the same tasks were 26% longer, 915 → 1,149 tokens. The share of answers reaching our 1,500-token output cap rose from 23.7% to 39.5%, and its pass rate fell from 93.1% to 87.2%; 97% of its failed answers after the change sat at the cap, against 86% before it. A user with no output cap would probably have seen a wordier model. Anyone running it under a fixed budget, as most production systems do, saw a model that failed one task in eight instead of one in fifteen. We count it as a provider-side change and as a degradation under a fixed budget: the budget is part of the model’s environment.

Five serving-speed steps with no change on our side

Before June 2026 no identifiers were recorded, so a provider-side change can only be inferred from an isolated behavioural step. Five such steps in latency survive the synchrony control and a reading of our own change history.

Serving latency moved by 28–51% in either direction, without announcement, five times with no change on our side. Two further latency changes, one in October 2025 on the day we changed every vendor’s output cap and one in April 2026 shared by two vendors within two days, cannot be separated from our own side or our network path and are not attributed. Our own change of 2 October 2025, a uniform 1,500-token cap and temperature for every vendor, moved models from three vendors by 31–51% on the same day; the synchrony control classified every one of those steps as ours.

Four endpoint withdrawals

The per-call record documents four withdrawals, each visible as a transition from successful calls to HTTP 404 under an unchanged request.

To an application that hard-codes a model identifier, a withdrawal is indistinguishable from an outage. The monitor’s own failure here was persistence: it retried dead endpoints for weeks and recorded each attempt as a failure of the model.

Across the 64 models with at least 100 runs, no model showed a sustained, model-specific step down in pass rate of five points or more (|t| ≥ 5) on the nine year-long tasks. At the relaxed threshold of three points, three steps appear: two on 2 October 2025, the day we imposed a uniform output cap and temperature on every vendor, which makes them ours; and one of about three points for GLM-4.6 around 27 December 2025.

This is a statement about sensitivity as much as about the models. Most models passed these tasks 98–100% of the time; GPT-4o passed 99.5% and Claude Opus 4.8 100%, so single-function tasks with well-known specifications could not tell a 2024 model from a 2026 one, let alone erosion within one model’s lifetime. A degradation that leaves easy tasks intact is invisible on them, and erosion at the top is what users of frontier models complain about. From September 2026 the scored coding corpus moved to defect repairs in small repositories graded by hidden tests. Of seventeen candidate tasks tried on the full 2026 fleet before any entered the corpus, sixteen sat at ceiling. On the corpus that resulted, the three tasks that discriminate, where the codebase supports an inference the complaint withholds, had fleet-wide pass rates of 12.7%, 63.6% and 67.9% from 13 September 2026, against 96–99% on the other four.

What the same tasks did show was change between generations. The 2025 generation of reasoning models spent a median of 1,490–1,584 output tokens and up to 22.5 seconds on tasks that 2024 models answered in 130–200 tokens and two to three seconds, without passing more often; for several of them 68–98% of failed runs were answers cut off at the 1,500-token cap. The adaptive-effort models of 2026 returned to 116–184 tokens and 1.6–3.4 seconds on the same tasks.

One widely discussed event sits just outside the record. A vendor-confirmed degradation of August–September 2025 peaked on 31 August; our measurement of that vendor’s models began on 8 September, after the peak, so the record can neither confirm nor refute it. Between 9 and 25 September 2025, Claude Sonnet 4 passed 99.5–100% of the year-long tasks on 15 of the 16 days with data; the exception, 13 September, was fleet-wide and therefore ours. As of 23 September 2026, every Page-Hinkley detector had just restarted on a new benchmark configuration and none had fired under its current design; its measured operating characteristics are in How often our drift detector is wrong, measured.

In September 2026 we published an article drawn from the composite-score record. Re-reading it against the per-call record, we withdrew three of its claims. In each case the figure had been a product of the instrument rather than of a model: score entries with no underlying calls, failures that were our own account, configuration or probe, or a count the per-call record could not reproduce. The article’s broader conclusion, that steps synchronised across vendors are instrument artefacts, stands and is confirmed in the raw record. Retracted detector incidents are kept in the database and flagged, never deleted; the detectors’ own correction record is in How often our drift detector is wrong, measured.

Also from the test record: How often our drift detector is wrong, measured. The constants behind the detectors are on the methodology page; the live verdict for each family, from the 24-hour boards, is on Is your AI getting worse?; every research page is listed on Research.