For inference providers

Ranking criteria v1

How hosted entries are ordered on AI Stupid Level, and when one joins the main ranking.

01

How hosts are ordered

  1. The measured score decides. On each board, hosted entries are ordered by that board's score: the combined mean on Combined, and that suite's mean on Coding, Reasoning and Tool use. Two entries whose difference is within 1.96 times the square root of the sum of their squared standard errors are tied.
  2. Price breaks a statistical tie, and nothing else. Within a tie, the cheaper entry comes first. The price used is the highest price the host declared in the last 7 days, and a change counts only 72 hours after it is made. Price never moves an entry past one it is not tied with.
  3. The display name, alphabetically, only when score and price are equal.

Never used: offers, the agreement version, the approval date, profile fields, and payment, which does not exist. The directory of providers is alphabetical. On an open model's page, measured hosts come first by their combined mean, then the others alphabetically.

02

Joining the main ranking

A hosted entry joins the main ranking after two consecutive daily evaluations on which every criterion holds:

P1
Listing published by an approved company under the current agreement; standing good
P2
Measured for at least 14 days on the current task version
P3
The leaderboard is comparable for every suite of the board: every ranked model’s newest result of each suite is a measured run
P4
Fresh task versions in use: a new instance of each task per run. Until fresh versions exist, P4 fails for every listing
D1
Runs on the current version: coding at least 12, reasoning at least 6, tool use at least 6, and probes at least 72
D2
The newest run is within 2 cadences
D3
The standard error is at most twice the median standard error of the ranked rows on that board
D4
Completed runs are at least 90% of completed runs plus runs abandoned for credit, key, rate or reachability, over 14 days
D5
No capped trials over 14 days
I1
Spot checks, last full calendar month: at least 2 consistent checks, a 30-day minimum detectable effect of at most 3 coding points, and no confirmed inconsistency in the last 90 days
I2
No open integrity case that we have told you about
I3
The listing’s configuration unchanged for at least 7 days
S
Our switch for joining the main ranking is on

An entry leaves the main ranking when P1, P3, P4, D2, D3 or I2 fails on two consecutive evaluations, and at once when it is under review, paused for more than 48 hours or suspended. Each automatic departure or return comes with a statement of reasons.

03

What we publish, and when

Hosted scores appear only as daily means in whole points, one to two days after the day ends, for everyone, the provider included. Knowing when a run took place, or its per-task results on a fixed task set, would let a host work out our hidden tests.

04

Changes

A change to any criterion, threshold or parameter is announced 15 days before it applies. These criteria are Annex A of our Provider Terms and Listing Agreement.