For inference providers
How hosted entries are ordered on AI Stupid Level, and when one joins the main ranking.
01
Never used: offers, the agreement version, the approval date, profile fields, and payment, which does not exist. The directory of providers is alphabetical. On an open model's page, measured hosts come first by their combined mean, then the others alphabetically.
02
A hosted entry joins the main ranking after two consecutive daily evaluations on which every criterion holds:
An entry leaves the main ranking when P1, P3, P4, D2, D3 or I2 fails on two consecutive evaluations, and at once when it is under review, paused for more than 48 hours or suspended. Each automatic departure or return comes with a statement of reasons.
03
Hosted scores appear only as daily means in whole points, one to two days after the day ends, for everyone, the provider included. Knowing when a run took place, or its per-task results on a fixed task set, would let a host work out our hidden tests.
04
A change to any criterion, threshold or parameter is announced 15 days before it applies. These criteria are Annex A of our Provider Terms and Listing Agreement.