Research and reports from the test record

Every scheduled run since the coding suite began on 8 August 2025, with reasoning, tool use and the canary following in the autumn of 2025, on a fixed schedule: coding every four hours, reasoning and tool use daily, with an hourly canary probe. Each run stores what the model was asked, whether it passed, how long it took and how many output tokens it billed; since 16 June 2026 it also keeps the raw text the model returned and the provider’s own response identifier. To 23 September 2026 the record held 184,654 task-level coding runs, 64,268 tool-use sessions, 5,237 reasoning sessions and 63,650 canary probes. Score records that have no underlying calls are excluded.

No, by design: a public task can be trained on, and a record built on it would stop measuring the model. A research page describes a task by what it asks and how it is graded, never by its content, and names one only once it has left the scored corpus. The method is another matter: the methodology page and the public methodology PDF carry every weight, statistical test and drift constant the reports rely on. The validation harnesses run in our private backend; every constant is published.

It caught changes in how models answered more often than changes in whether they answered correctly: two new builds behind an unchanged model name that doubled output length and raised latency by about 1.7×, a requested model name redirected to a different served model, five serving-speed steps of 28–51%, four withdrawn endpoints, and one verbosity change that a fixed output budget turned into a six-point fall in pass rate. On the nine tasks that ran all year, no model lost five points or more of pass rate. The dates and values are in What 13 months of monitoring LLM APIs caught. The live answer for each family — is ChatGPT getting worse, is Claude getting worse — is on Is your AI getting worse?, from the 24-hour boards.

Measured three ways. In simulation with the production code at the fleet’s typical noise, a standard deviation of 1.5 points, the Page-Hinkley detector (a CUSUM-family test) raised 0 false alarms in 44,000 model-days while catching a sustained 5-point drop every time, in a median of 8 days. On bootstrapped real histories the fleet mean was 1.16 false alarms per 100 model-days, almost all of it from two models with noise above 7. The hourly canary, replayed over 1,753 real probe-times, raised 0 false alarms and caught an injected collapse from 100 to 50 in 22 of 22 series, in a median of 4 hours. The full grid is in How often our drift detector is wrong, measured.

Yes, and they stay on record. In September 2026, 446 canary incidents and 46 provider-wide incidents were retracted and 1,031 change points were flagged unreliable; the rows are kept in the database and flagged, never deleted. Three claims from an article of September 2026 were withdrawn when re-read against the per-call record. Each research page carries its own corrections, dated.

Each page ends with a plain-text citation — AI Stupid Level, the title, the kind of report, the publication date and the canonical link — and states its data window in the byline. Papers are dated and versioned; a correction is added to the page that made the claim, with its date, rather than silently rewriting it.