How we test AI models and detect drift

The four boards are Combined, Coding, Reasoning and Tool use, each with the best model at the top. Combined weights coding 50% and reasoning and tool use 25% each. Coding is re-measured every four hours, reasoning and tool use once a day.

The four layouts

A first-time visitor picks one. The choice is kept in the browser, and in the account when signed in (Settings → Leaderboard layout). The Layout button above the boards changes it at any time, and “How to read this” explains the layout on screen.

Around the boards

• Your watchlist, above the boards when signed in: every model you have starred, with its place on each board. Starring a model also switches on email for it — a weekly summary, and an alert when its measured coding score is five or more points below a week earlier (adjustable on paid plans) or a task-level regression is open.
• Needs attention, only when a measured problem exists, and Quick answers — how they are picked.
• Compare models: a heatmap of every model's measures, a radar of the top and bottom three, and a price-performance table (score per dollar of list price), switched between boards by the tabs above them.
• Providers and method: each provider's live status from a check every ten minutes, a trust score from its incident history, and the test schedule.
• The Drift monitor view, next to Leaderboard at the top, compares every model with its own past rather than with the other models.

A suite that is missing, or older than two of its own runs (8 hours for coding, 48 for reasoning and tool use), drops out and the remaining suites are reweighted by their base weights. It is never filled in with a placeholder value. The board says so: a partial row carries a note such as “2 of 3 suites”, and a row with nothing fresh enough is not ranked.

Every score records the exact version of the tests it ran under — a fingerprint of every prompt, test and check — so a change we make is never mistaken for a change the model made. Scores are the model as served through its provider's public API, run with our own keys (the one exception is community-funded runs).

Why these weights. They follow what each measure can actually tell apart, not how important it sounds, measured by averaging each model over several sweeps and taking the spread between models. Complexity varies by four thousandths across the fleet, so it cannot move anyone's rank and carries no weight (it used to carry 20%); it is still measured and shown. Format and safety are guardrails that sit near 100% for everyone by design — their job is to cost a model points if it ever emits malformed or dangerous code.

What the suite asks

Most tasks hand the model a small working project and a bug report written as a user complaint — not a diagnosis, and no file named. The cause sits a module away from the symptom, with a plausible decoy in between. The model must find the defect and return a corrected file, which is graded by running the project's own test suite.

Hidden tests the model never sees make up three quarters of each repository task's score. On one task, deduplicating payments by amount makes every visible test pass and is still wrong — it stops a customer legitimately buying the same item twice. Eight of eighteen fleet runs took exactly that shortcut; without hidden tests all eight would score full marks.

What gets retired. A task everybody passes ranks nobody. Single-function tasks stopped separating models in 2026 — eight were passing at 99–100% across the fleet and were retired in September 2026, and nine deliberately hard replacements were mostly solved perfectly by every model. A new task ships only if every hidden assertion traces to the bug report and at least a fifth of the fleet fails it across three independent runs.

Only the answer is graded, never a model's hidden reasoning. An attempt that uses its whole output budget without answering, or answers without code, is a failed attempt, and a retry asks the identical question.

When a model declines a task

Some providers refuse entirely benign prompts. A refusal is a content decision, not a measure of ability, so it is not scored as zero: the task drops out and the model is scored on what it attempted. Declined tasks tend to be the hard ones, so such a row shows its coverage (“5/7 tasks”), the model page names the declined tasks, and the model is never called tied with one measured on all of them.

Each session builds on its own earlier turns, and when the model's work fails it is told so the way a colleague would — without being handed the error. Five measures, weighted per task: correctness, recovery after a failed step, and three continuity measures — memory retention, plan coherence and context use. Continuity is checked by running code, not by matching words: rules stated once in the conversation (at the start, or partway through while the model is working on something else) and the decisions a model declares in its own plan are tested at every later step. A session may run for up to ninety minutes; a task a model declines or does not finish in that time drops out, and the row says so.

Share of the task's score each measure carries in a normal session. A measure the session gives no evidence for drops out and the rest are rescaled; it is never scored as zero. Plan coherence is not taken on tasks that ask for no plan. *Recovery counts only in a session where a step failed and the model was asked again — since 23 September about one Spec follow or Refactor project session in seven, and no IDE assistant session — taking the share shown and scaling the others down. A document answer is never re-asked, so Document memory has none. Rounded.

Each model sees its own tool calls and their results in its provider's native tool format, with its own reasoning carried between calls. Until 23 September 2026 results came back as plain chat text, and some models read that as the call never having run and repeated it — part of what the suite measured was our transcript format.

Corrected in September 2026. Until 13 September 2026 this suite raised an incident on any 10% fall in a 24-hour mean, with no significance test, on answers cut off at 500 tokens. The 446 incidents it produced are retracted and excluded from every count on this site. The drift signature had a related fault — it mixed the three suites' different scales — and the 46 provider-wide incidents and 1,145 change points recorded before that date came from it.

The standard error of a score is its run-to-run repeatability: each suite's last five measured runs on its current test version, combined with the composite's weights (SE² = Σ (wᵢ/W)² SEᵢ²). The interval shown is ±1.96 SE. The day after a test-version change there are not yet five runs, and the suite's typical spread is used instead (coding 2, tool use 1.5, reasoning 8 points).

Seven attempts per coding task collapse to one outcome per task by median, so one unlucky attempt cannot move a task. The interval on the board comes from repeatability across runs, not from the attempts.

How a rank is assigned. Ranks follow the board. Walking down it in score order, a model shares the rank of the group above it when it is not measurably worse than that group's leader — its highest-scoring member — meaning the leader is ahead by no more than 1.96 × √(SE_leader² + SE_model²), a two-sample test at 95%. Otherwise it opens a new group at its own position, so 1, 1, 1, 4 is expected. Comparing with the group's leader rather than the model directly above stops a chain of individually unresolvable gaps from merging models whose ends are far apart. A model graded on fewer tasks than the rest keeps its position and is never called tied.

Page-Hinkley is a cumulative-sum detector. Scores are on a 0–1 scale; with mean_t the running mean since the last reset, it accumulates m_t = m_t−1 + (mean_t − x_t − δ) and alerts when m_t − min(m) exceeds λ, then resets fully so it can re-learn the new level. Daily noise cancels out; a real sustained decline builds up. (The database column is still called cusum for historical reasons.)

What it can and cannot see — measured

A detector is only worth trusting if two numbers are known: how often it fires when nothing changed, and how reliably it fires when something did. Both are measured on the production code path by injecting a sustained drop of known size into a stationary series (400 repetitions per cell) and by running it on series with no change at all. “Detected” means within 30 days; the delay is the median days to the first alert.

Measured on the current tests, the median model's day-to-day spread is 2.0 points on reasoning (16–22 September 2026) and 3.9 on tool use (14–22 September; 2.1 to 7.2 across models). That is why tool use has its own, stricter rule: at its noise the shared rule would raise about two false alarms a month across the fleet, while the stricter one is simulated at about one every three to four months. The price is small drops — a sustained 3-point decline in tool use is caught within 30 days less than half the time; a 5-point one is still caught 96% of the time, in a median of 15 days.

What it will miss: a sustained drop of about 2 points or less, and — by design — a one-day dip of any size. That is the canary's job.

How the table was made: a simulation with a fixed random seed, so re-running it gives exactly this table. It runs in our private backend alongside the tests, but the detector's rule and every constant are on this page, so it can be rebuilt and checked independently.

Change points

Separately, every hour the coding series on its current test version is checked for a step: the last 3, 5 and 10 runs against the same number before them (Mann-Whitney U at p < 0.05 with a move of more than 5 points, or non-overlapping intervals and more than 8). A step found at two of the three window sizes, or any step of 15 points or more, is recorded as a change point. Change points are a record for our own investigation; they do not set a model's status and are not the alerts sent to watchers. A model that alternates between passing and failing one task produces them, which is why they are not published as findings.

From 24 September 2026, the reasoning and tool-use runs for Claude Sonnet 4.6, Claude Opus 4.6, 4.7 and 4.8 and GPT-5.5 are funded by the community. A signed-in visitor can fund one run per test per model per day from the model's page, paying with their own API key. The run is exactly our scheduled test, on our servers, and its result is published like any other; the key is used for that run only and never stored.

On the boards these models stay ranked on Coding. On Reasoning and Tool use they sit in a community-funded group at the foot of the board, with the date of their last funded run, and are not ranked until a new run is funded. On Combined they are listed without a rank: only their coding is measured on our schedule, and a combined rank needs all three suites.

For these cards cost is measured, not list price: every model runs the same seven coding tasks and every attempt is billed, so a verbose model costs what it really costs per task. The price-performance table beside them uses list price.

Each week every pilot model reads the same kind of document — an archive of company records — at 8K, 32K, 128K, 256K, 512K and 1M tokens (as far as its context window allows), three times at each length, and answers sixteen questions graded exactly: finding a fact placed 10% to 90% of the way through, linking facts across the document, tracking a value that changes and is sometimes withdrawn, and counting across the whole archive. Every fact is written in the same record format as thousands of look-alikes, because a fact that stands out by its format can be found at any length. The pilot covers DeepSeek, Kimi and GLM; OpenAI, Anthropic, Google and more providers will follow. The results are on the context rot page for Pro Intelligence and above.

Runs are individual task attempts, all real executions. Retracted incidents are excluded from the last figure; 1,145 of the change points in it predate 13 September 2026 and came from the mixed-suite series described under Hourly canary.

Models tested (23)

Claude Fable 5.1 · Claude Opus 4.6 · Claude Opus 4.7 · Claude Opus 4.8 · Claude Opus 5 · Claude Opus 5.5 · Claude Sonnet 4.6 · Claude Sonnet 5 · Claude Sonnet 5.5 · DeepSeek V4 Flash · DeepSeek V4 Pro · Gemini 3.5 Flash Lite · Gemini 3.8 Flash · GLM-5.3 · GPT-5.5 · GPT-5.6 Luna · GPT-5.6 Sol · GPT-5.6 Terra · GPT-6 Astra · GPT-6 Luna · GPT-6 Sol · Kimi 2.7 · Kimi K3

Create a key · Full API reference. Volume beyond these tiers and commercial redistribution are arranged directly — get in touch.

Check it yourself. Test your keys runs the same tasks with your own API keys and scores them with the same code as our published runs. The web application is open source; the task bank is kept private, because when it was public providers optimised against it.