Live · 24 models · real API tests since August 2025
Is your AI worth what you’re paying? And is it still as good as last month?
AI Stupid Level has tested AI models on a fixed schedule since August 2025: every model on the same tasks, through each provider’s own API. Not votes, measurements. See which models get the job done, what a correct result really costs, and when a model changes. Our Smart Router puts it to work, sending every request to the model that is measurably best right now.
Building on AI? We benchmark the models on your own work, and keep it running. Free account: every ranking, a 3-model watchlist and 1,000 routed requests a month. No card.
Claude Opus 4.8Anthropic$24.22
GPT-6 LunaOpenAI$0.3081× less per correct fix, and GPT-6 Luna passed more often.
Same 6 bug-fix tasks for both, 23 Sep – 14 Oct, every attempt at the provider’s list price. Picked by rule: the model whose correct fixes cost the most, next to the cheapest that passed at least as often.
Measured and routed across 6 providers
- OpenAI
- Anthropic
Google
DeepSeek
Moonshot AI
Z.ai
The real price of AI
Price per token isn’t price per result.
Every model gets the same tasks, and every attempt is priced at the provider’s list price. Divide what the attempts cost by the ones that succeeded and you get what a correct result really costs: on our coding tasks, from $0.30 to $24.22 per 1,000 correct fixes. Pick the model you use and see what does as well for less.
6 bug fixes in real code repositories. A fix counts only when the hidden tests pass.
- Passed
- 56%
- Cost per 1,000 correct fixes
- $24.22
- Typical attempt
- 2.9 s
Did at least as well, for less
- GPT-6 Luna84% · $0.3081× less
- GPT-5.6 Luna89% · $0.6935× less
- Gemini 3.5 Flash Lite60% · $0.9127× less
The same tasks for every model, 23 Sep – 14 Oct. Every attempt is priced at the provider’s list price; cheapest first, bars to scale.
AI drift detection
When a model quietly changes, you see it — and the router moves on.
Providers change models without changing the name. We have tested every model on a fixed schedule since August 2025, so a change leaves its mark in the record: in quality, in speed and in what each answer costs. A few of them:
12 Aug 2026
2×longer answers
DeepSeek V4 Pro
A new build overnight. Answers twice as long (728 → 1,478 tokens) and 72% slower, under the same model name.
31 Jul 2026
2×cost per correct answer
DeepSeek V4 Flash
A new build overnight. Answers twice as long (510 → 1,045 tokens), 64% slower, and each correct answer cost almost twice as much at the same price per token.
March 2026
−6 ptspass rate
GLM-4.6
Answers about a quarter longer (907 → 1,149 tokens), and its pass rate on the same tasks fell from 94% to 87%.
Smart Router · next request
auto-reasoning stops picking the flagged model and routes to the next healthy one. The routing reason names who was passed over, and why.
Flagged models drop behind healthy ones for every strategy that depends on that suite.
Watchlist · email
An email when a model you watch drops 5 points below where it was a week ago. Set your own threshold on paid plans.
Plus a summary every Monday of what moved and what held.
Teams · webhook
A signed callback to your own endpoint when a watched model regresses or a project reaches a budget threshold.
Coding, reasoning and tool use each run their own Page-Hinkley detector against the model’s own baseline, tuned on real run-to-run noise.
Fixed probes run every hour; a sustained shift has to pass Welch’s t-test before it opens an incident.
Every flag points to the runs behind it. Full history and the diagnosis behind each change come with Pro.
For teams building on AI
Benchmark the models on your own work.
Public scores tell you which model is best on our tasks. An assessment tells you which one is best on yours — measured by the same harness, the same scoring rules and the same statistics.
01
Send up to 20 real tasks
Prompts, files and the answers you would accept. We confirm the scope within 2 business days.
02
We run 3 models of your choice
In the same sandbox as the public leaderboard, with repeated trials so a lucky run can’t decide the result.
03
Report in 7 days
Per-task results, cost and latency for each model, and our recommendation.
| Model | Passed | Per 1,000 correct | Typical attempt |
|---|---|---|---|
| Claude Opus 5.5 | 95% | $13.68 | 3.9 s |
| GPT-6 Luna | 84% | $0.30 | 3.8 s |
| Claude Opus 4.8 | 56% | $24.22 | 2.9 s |
What the numbers sayGPT-6 Luna passed more often than Claude Opus 4.8, and each correct fix cost 81× less. Claude Opus 5.5 passed the most attempts, at 46× the cost of GPT-6 Luna per correct fix. On your own tasks the answer can be different, and that is what an assessment measures.
Then keep it running
Your own benchmark, wired into your router.
Our team builds a custom benchmark around your workflow — your tasks, your acceptance rules, your formats — and runs it continuously alongside our public suites. It connects to your Smart Router, so your requests keep going to whichever model is best on your work, and you hear about it when that changes.
- Built by our team for your use case, from your real tasks.
- Runs on our schedule, with drift alerts on your own suite.
- Drives your router: the model that wins on your work gets your traffic.
Smart Router
Models change overnight. Your router should know.
Point any OpenAI-compatible client at ASL and every request goes to the model that is measurably best right now — for code, reasoning, tool use, speed or cost — ranked on the same results as this page. One endpoint, your own provider keys, zero markup on tokens.
- 0%
- markup on tokens. Providers bill you directly, at their own list price.
- 15
- auto routing modes, from
auto-codingtoauto-cheapest. - 10 min
- between health checks on every provider the router can pick.
- auto-codingFix the race condition in checkout.ts→ GPT-5.6 Luna · 200 · 7.9 s
- Model
auto-coding- Strategy
- best_coding
- Ranked on
- Coding suite: real repository bug fixes, every 4 hours
- Passed over
- none
- Routed to
- GPT-5.6 Luna · 7.9 s
GPT-5.6 LunaOpenAIrouted97Claude Opus 5.5Anthropicfallback96Claude Opus 4.7Anthropic95Claude Sonnet 5.5Anthropic94GPT-5.5OpenAI94GPT-6 AstraOpenAI94DeepSeek V4.1 FlashDeepSeekfallback93Best overall
best_overallThe highest combined score among the models you have keys for.
- Ranks on
- Combined score: coding 50%, reasoning 25%, tool use 25%
- Ordered by
- Quality (effective score)
- Always
- Only models you have a key for; drift-flagged models and open circuits go last
GPT-5.6 LunaOpenAIprimary96Claude Sonnet 5.5Anthropicfallback94Claude Opus 5.5Anthropic93DeepSeek V4.1 FlashDeepSeekfallback93Claude Fable 5.1Anthropic93GPT-5.6 TerraOpenAI92GPT-5.6 SolOpenAI92from openai import OpenAI client = OpenAI( base_url="https://aistupidlevel.info/v1", api_key="aism_…", # your ASL router key ) reply = client.chat.completions.create( model="auto-best", messages=[{"role": "user", "content": prompt}], stream=True, )
Response headers
HTTP/1.1 200 OK X-AISM-Model: gpt-5.6-luna X-AISM-Provider: openai X-AISM-Latency: 7853 X-AISM-Reasoning: Selected gpt-5.6-luna from openai for best overall performance (combined benchmark) (score: 96.0). Ranked #1 of 24 available models. Last updated 1h ago.
Also speaks Anthropic: native /v1/messages passthrough, plus /v1/embeddings. Works with any client that lets you set a base URL. Router documentation
Router internals
Built for agents in production.
A benchmark-driven pick is only half of a router. These are the details that decide whether it survives real traffic.
Your keys, your bill
Requests go out with your own OpenAI, Anthropic, Google, DeepSeek, Moonshot and Z.ai keys, so providers bill you directly. We add nothing to token prices — plans include a number of routed requests.
Drift-aware by default
When a model’s detector fires on the suite your strategy depends on — or it has an open canary incident — it drops behind healthy models. The routing reason names who was passed over.
Outages route around you
Each request carries up to 5 fallbacks: first from your own fallback order (a list of up to 10 models, skipping any that are unavailable), then the best models at other providers. A circuit opens after 5 straight failures and probes again after 60 seconds.
Tool loops that don’t derail
Native tool calls with the provider’s own call ids, each model’s reasoning carried between turns, and a loop that stays on the model that started it. Forced tool_choice and json_schema work everywhere — natively, or emulated and checked.
Long context, long thinking
Native streaming on every provider, request bodies up to 32 MB for 1M-token contexts, and a 15-minute timeout for models that think before they answer.
Budgets that hold
Hard and soft monthly budgets on every key and project, a maximum price per 1M tokens, a maximum latency, and provider or model exclusions — all checked before a request is sent.
X-AISM-ReasoningEvery pick explained
Each response names the model, the provider, the latency and why it won. Preview any routing setting against the real pipeline without spending a request.
Private by default
Prompt logging is off unless you turn it on. Logged prompts are scrubbed of secrets before they are encrypted, and so are your provider keys.
Learns from production
Real failures and your ratings through /v1/router/feedback adjust each model’s effective score, so a model that benchmarks well but fails in practice loses ground.
Benchmarks
Measured around the clock, not once a quarter.
The router is only as good as its data. Every model gets the same tasks, the same harness and the same scoring, on a fixed schedule — shown here in your time zone.
Methodology public. Test tasks private, so no lab can train on them.
On the board
Every model we measure.
24 models from 6 providers. Each has its own page with its score history, drift record and cost — open to everyone.
- GPT-5.5OpenAI
- GPT-5.6 LunaOpenAI
- GPT-5.6 SolOpenAI
- GPT-5.6 TerraOpenAI
- GPT-6 AstraOpenAI
- GPT-6 LunaOpenAI
- GPT-6 SolOpenAI
- GPT-6.1 SolOpenAI
- Claude Fable 5.1Anthropic
- Claude Opus 4.6Anthropic
- Claude Opus 4.7Anthropic
- Claude Opus 4.8Anthropic
- Claude Opus 5Anthropic
- Claude Opus 5.5Anthropic
- Claude Sonnet 4.6Anthropic
- Claude Sonnet 5Anthropic
- Claude Sonnet 5.5Anthropic
- Gemini 3.5 Flash LiteGoogle
- Gemini 3.8 FlashGoogle
- DeepSeek V4 ProDeepSeek
- DeepSeek V4.1 FlashDeepSeek
- Kimi 2.7Moonshot AI
- Kimi K3Moonshot AI
- GLM-5.3Z.ai
Provider status
Every provider, checked every 10 minutes.
A short completion goes to each provider with our own key, day and night. Failures are re-checked 15 seconds later and attributed, so the router only avoids real outages — and the status page never blames a vendor for our problem.
OpenAIOperational1.3 sAnthropicOperational1.2 sGoogleOperational649 msDeepSeekOperational988 msMoonshot AIOperational1.1 sZ.aiOperational1.6 sUnreachable twice while our side is healthy. Opens an incident; the router stops sending traffic.
A billing or quota answer for our own key. Logged, never shown as the provider’s outage.
A malformed or timed-out check of our own making. Re-run, not reported.
Several providers failing together, or our own network down. The router keeps routing.
Data API
Every score, as JSON.
Keyed access to scores, history, drift and incidents — for dashboards, CI gates and model reviews. Build against the free key at 10 requests a day; production tiers go up to 250,000 a day.
curl https://aistupidlevel.info/api/v1/models/292/history?period=7d \ -H "X-API-Key: $ASL_DATA_KEY"
{
"success": true,
"period": "7d",
"data": {
"dataPoints": 41,
"data": [
{ "timestamp": "2026-10-14T10:00:00Z", "score": 86,
"suite": "combined", "axes": { … } },
…Watchlist & alerts
Know before your users do.
Pick the models you depend on. We tell you when one of them slips — by email, in a weekly summary, or by webhook into your own tools.
- 3 models free, 20 on Pro and Developer, unlimited on Teams.
- Email when a watched model falls 5 points below where it was a week ago. Set your own thresholds on paid plans.
- A weekly summary every Monday of what moved and what held.
- Signed webhooks for regressions and budget thresholds, and a team watchlist that alerts everyone in the workspace.
Pricing
Start free. Pay when the router earns it.
Free
$0forever
No card required
The public evidence, plus a watchlist of three models with email alerts and a weekly summary.
Create free accountIncludes
- 3 tracked models, with email alerts
- 30 days of history
- Every category ranking
- Smart Router: 1,000 requests a month
- Data API: 10 requests a day
Pro Intelligence
$120/ year
$10 a month · save $24
Full history, the diagnosis behind every change, calibration and substitute analysis, exports and custom alerts.
Start free trialEverything in Free, plus
- 20 tracked models and custom alert thresholds
- Full history and the drift diagnosis behind every change
- Calibration and cheaper-substitute analysis
- Context-rot pilot: long-context accuracy up to 1M tokens
- Routing analytics and CSV/JSON exports
- Smart Router: 10,000 requests a month
- Data API: 10,000 requests a day
Developer
$250/ year
$20.83 a month · save $50
For running the Smart Router in production: ten times the requests, API monitoring and 30-day decision logs.
Start free trialEverything in Pro, plus
- Smart Router: 100,000 requests a month
- API monitoring: per-key logs, costs, prompt auditing and budget limits
- 30-day decision logs
- Data API: 25,000 requests a day
- A workspace with one project and a shared watchlist; viewers free
TeamsMost complete
$1,290/ year
$107.50 a month · save $258
Five editors, each with the full Teams plan, three projects with their own router keys, budgets and spend reports, plus webhooks, single sign-on and an audit trail.
Start free trialEverything in Developer, plus
- 5 editors, each with the full Teams plan
- 3 projects, each with its own router keys, rules, budget and spend report
- A team watchlist with alerts to everyone
- Smart Router: 1,000,000 requests a month
- 90-day decision logs
- Data API: 100,000 requests a day
- Webhooks for regressions and budgets
- Single sign-on (OIDC or SAML), SCIM and an audit trail
EnterpriseContracted scope, unlimited seats and requests, 365-day decision logs and invoicing. 250,000 Data API calls a day.
Talk to usPast your router allowance? Top up credits from $5 — 2,000 requests per $1, no expiry. Providers bill your tokens directly; we never add to them. Compare every plan
FAQ
Questions people actually ask.
Still unsure? The methodology explains every number on the site, and the full FAQ covers the rest.
Are AI models really getting worse over time?
Sometimes — and usually without notice. Providers change weights, safety layers and serving infrastructure behind the same model name. AI Stupid Level re-runs the same tasks on every model on a fixed schedule and compares each one with its own history, so a drop shows up as data instead of a hunch.
Is ChatGPT getting worse? Is Claude getting worse?
Check the live scores. Every OpenAI and Anthropic model we track is re-tested on coding every 4 hours and on reasoning and tool use every day, and a model is flagged only when a change holds up statistically — not after one bad run.
What does “cost per correct fix” mean?
What we paid the provider, at its list price, for every attempt at a task, divided by the attempts that passed the hidden tests. A model that is cheap per token but fails often can cost more per correct result than a pricier one that gets it right, so this is the number that tells you what the work really costs.
Which AI model is best for coding right now?
It changes, which is the point. The leaderboard ranks every model we track on real repository bug fixes every 4 hours, and the router’s auto-coding mode sends your request to the current leader among the models you have keys for.
How is the Smart Router different from other LLM routers?
It routes on measured quality, not just price and uptime. Rankings come from our own continuous benchmarks, drift-flagged models drop behind healthy ones, and every response explains its pick in an X-AISM-Reasoning header. You keep your own provider keys and pay providers directly.
Can you test the models on our own tasks?
Yes. A workload assessment runs up to 20 of your real tasks on three models in our harness, with a report in 7 days. Our team can also build a custom benchmark around your workflow, run it continuously alongside our public suites, and connect it to your Smart Router, so your requests keep going to the model that is best on your work.
Do you mark up tokens or resell inference?
No. Requests are sent with your own provider keys and billed to you by the provider at their list price. Plans include a monthly number of routed requests; past that you can top up credits or move up a plan.
Which providers and models do you cover?
OpenAI, Anthropic, Google, DeepSeek, Moonshot AI (Kimi) and Z.ai (GLM). The router handles text, code and tool calls. It does not route images or audio, because we do not measure them and will not route what we do not measure.
Why do I need an account to see the leaderboard?
A free account costs nothing and includes every category ranking over the last 24 hours, 7 days and 30 days, a three-model watchlist with email alerts, 1,000 routed requests a month and a Data API key. The newest run, as it lands, is part of the paid plans. Model pages, the methodology and the status page stay open to everyone.
Stop guessing which model is best today.
Create a free account to see every ranking, watch three models and route your first 1,000 requests — with your own keys.