score: rate on a rubric
Each model reads a chatbot's reply to a prompt and rates how helpful it is on a scale of 5 levels, with a probability for each level. All models get the same 300 replies, 5 times each, and we check every answer against human labels. The replies come from HelpSteer2.
Score vs price
Up and to the left is better. The line joins the best value at each price.
System One LLM
Automation at 95% accuracy
Share of questions each model can answer alone and stay 95% right. It answers only above its confidence cut-off; the rest go to a person.
No model reaches 95% accuracy at any confidence on this task. Keep a person in the loop.
Ranking
System One LLM Baseline
No model clearly beats guessing yet: the top rows can't be told apart from answering with the label base rates.
| Model | Wins | 95% range of the Decision Score | ||||||
|---|---|---|---|---|---|---|---|---|
| 1 | Jev | 3 | 9.2 | 41.3% | 19.7 | $0.036 | 670 ms | |
Accuracy41.3%95% range 35.9–46.9% Calibration gap19.7 ptslower is better · 0.3% of answers at confidence 1.00 Automation at 95%—never reaches 95% accuracy Speed478 msmedian · p95 670 ms Stability2.3%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupai-sdk experimental_evaluate ai@7.0.106 · native probabilities via Vercel AI Gateway · reasoning — · temperature — · 300 items × 5 runs · 2026-09-18 · served as typesafe-ai/jevmodel pagehelpsteer2 run log | ||||||||
| =2 | GLM-5.3 | 2 | 7.8 | 43.0% | 12.9 | $0.78 | 3.6 s | |
Accuracy43.0%95% range 38.6–47.9% Calibration gap12.9 ptslower is better · 0.0% of answers at confidence 1.00 Automation at 95%—never reaches 95% accuracy Speed2.2 smedian · p95 3.6 s Stability31.3%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Novita (fp8) · reasoning low (mandatory) · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as z-ai/glm-5.3model pagehelpsteer2 run log | ||||||||
| =2 | Mercury 2.5 | 2 | −5.5 | 41.7% | 31.5 | $0.034 | 1.2 s | |
Accuracy41.7%95% range 37.3–46.2% Calibration gap31.5 ptslower is better · 0.1% of answers at confidence 1.00 Automation at 95%—never reaches 95% accuracy Speed618 msmedian · p95 1.2 s Stability38.7%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Inception · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as inception/mercury-2.5model pagehelpsteer2 run log | ||||||||
| 4 | Mistral Medium 3.5 | 1 | −13.7 | 43.9% | 33.6 | $1.3 | 1.2 s | |
Accuracy43.9%95% range 38.5–49.3% Calibration gap33.6 ptslower is better · 2.1% of answers at confidence 1.00 Automation at 95%—never reaches 95% accuracy Speed706 msmedian · p95 1.2 s Stability3.7%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Mistral · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as mistralai/mistral-medium-3-5model pagehelpsteer2 run log | ||||||||
| =5 | Gemini 3.8 Flash | 0 | 4.6 | 42.4% | 22.6 | $1.3 | 4.0 s | |
Accuracy42.4%95% range 36.9–47.5% Calibration gap22.6 ptslower is better · 0.1% of answers at confidence 1.00 Automation at 95%—never reaches 95% accuracy Speed1.9 smedian · p95 4.0 s Stability11.7%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Google AI Studio · reasoning low (mandatory) · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as google/gemini-3.8-flashmodel pagehelpsteer2 run log | ||||||||
| — | Label prior | — | 0.0 | 41.7% | 0.0 | — | — | |
Accuracy41.7%95% range 36.0–47.0% Calibration gap0.0 ptslower is better · 0.0% of answers at confidence 1.00 Automation at 95%—no probabilities Speed—median · p95 — Stability0.0%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupconstant probabilities · reasoning — · temperature — · 300 items × 5 runsmodel page | ||||||||
| =5 | Qwen3.8 Flash | 0 | −1.4 | 36.1% | 25.8 | $0.13 | 2.6 s | |
Accuracy36.1%95% range 31.5–40.7% Calibration gap25.8 ptslower is better · 0.0% of answers at confidence 1.00 Automation at 95%—never reaches 95% accuracy Speed1.4 smedian · p95 2.6 s Stability26.0%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Alibaba · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as qwen/qwen3.8-flashmodel pagehelpsteer2 run log | ||||||||
| =5 | DeepSeek V4.1 Flash | 0 | −19.0 | 34.7% | 33.5 | $0.13 | 1.2 s | |
Accuracy34.7%95% range 30.5–38.7% Calibration gap33.5 ptslower is better · 1.5% of answers at confidence 1.00 Automation at 95%—never reaches 95% accuracy Speed912 msmedian · p95 1.2 s Stability34.3%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · DeepSeek · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as deepseek/deepseek-v4.1-flashmodel pagehelpsteer2 run log | ||||||||
Wins: a model wins a column when it is one of the two best in it: the highest score, the highest accuracy, the smallest calibration gap, the lowest price or the fastest speed. Ties count for everyone tied. The table is ordered by wins, then by score; models with as many wins share a place, so =3 means tied for third.
Release 2026-09-18 · suite 0.1.0 · board.json · permalink · explore every question · how we score