noul: yes or no
Each model reads the abstract of a medical study and answers a yes-or-no question about it, saying how likely yes is. All models get the same 300 questions, 5 times each, and we check every answer against human labels. The questions come from PubMedQA.
Score vs price
Up and to the left is better. The line joins the best value at each price.
System One LLM Top 2 for that metric (score, cost, speed)
Automation at 95% accuracy
Share of questions each model can answer alone and stay 95% right. It answers only above its confidence cut-off; the rest go to a person.
Ranking
System One LLM Baseline Top 2 for that metric (score, cost, speed)
| Model | Wins | 95% range of the Decision Score | ||||||
|---|---|---|---|---|---|---|---|---|
| 1 | Jev | 4 | 69.0 | 91.3% | 5.0 | $0.029 | 653 ms | |
Accuracy91.3%95% range 87.9–94.3% Calibration gap5.0 ptslower is better · 0.0% of answers at confidence 1.00 Automation at 95%86%of decisions, acting at confidence ≥ 0.75 (right 95.0%)shared gate ≥ 0.91: acts on 49%, right 98.6% Speed438 msmedian · p95 653 ms Stability0.0%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupai-sdk experimental_evaluate ai@7.0.106 · native probabilities via Vercel AI Gateway · reasoning — · temperature — · 300 items × 5 runs · 2026-09-18 · served as typesafe-ai/jevmodel pagepubmedqa run log | ||||||||
| 2 | Gemini 3.8 Flash | 3 | 73.0 | 92.5% | 2.0 | $0.8 | 5.5 s | |
Accuracy92.5%95% range 89.3–95.3% Calibration gap2.0 ptslower is better · 0.8% of answers at confidence 1.00 Automation at 95%79%of decisions, acting at confidence ≥ 0.86 (right 96.6%)shared gate ≥ 0.91: acts on 73%, right 97.1% Speed1.9 smedian · p95 5.5 s Stability1.7%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Google AI Studio · reasoning low (mandatory) · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as google/gemini-3.8-flashmodel pagepubmedqa run log | ||||||||
| =3 | GLM-5.3 | 1 | 60.6 | 88.7% | 2.0 | $0.51 | 2.9 s | |
Accuracy88.7%95% range 85.3–91.6% Calibration gap2.0 ptslower is better · 0.3% of answers at confidence 1.00 Automation at 95%45%of decisions, acting at confidence ≥ 0.91 (right 96.7%)shared gate ≥ 0.91: acts on 45%, right 96.7% Speed1.8 smedian · p95 2.9 s Stability5.3%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Novita (fp8) · reasoning low (mandatory) · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as z-ai/glm-5.3model pagepubmedqa run log | ||||||||
| =3 | Mistral Medium 3.5 | 1 | 58.0 | 88.8% | 5.2 | $0.87 | 917 ms | |
Accuracy88.8%95% range 85.3–92.2% Calibration gap5.2 ptslower is better · 14.2% of answers at confidence 1.00 Automation at 95%65%of decisions, acting at confidence ≥ 0.91 (right 95.5%)shared gate ≥ 0.91: acts on 65%, right 95.5% Speed533 msmedian · p95 917 ms Stability1.0%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Mistral · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as mistralai/mistral-medium-3-5model pagepubmedqa run log | ||||||||
| =3 | Mercury 2.5 | 1 | 55.7 | 87.1% | 4.7 | $0.023 | 1.0 s | |
Accuracy87.1%95% range 84.0–90.1% Calibration gap4.7 ptslower is better · 1.1% of answers at confidence 1.00 Automation at 95%19%of decisions, acting at confidence ≥ 0.99 (right 95.8%)shared gate ≥ 0.91: acts on 72%, right 92.5% Speed584 msmedian · p95 1.0 s Stability10.0%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Inception · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as inception/mercury-2.5model pagepubmedqa run log | ||||||||
| =6 | Qwen3.8 Flash | 0 | 62.4 | 89.7% | 2.5 | $0.086 | 3.5 s | |
Accuracy89.7%95% range 86.3–92.7% Calibration gap2.5 ptslower is better · 0.7% of answers at confidence 1.00 Automation at 95%—never reaches 95% accuracyshared gate ≥ 0.91: acts on 72%, right 93.9% Speed1.1 smedian · p95 3.5 s Stability4.7%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Alibaba · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as qwen/qwen3.8-flashmodel pagepubmedqa run log | ||||||||
| =6 | DeepSeek V4.1 Flash | 0 | 47.5 | 83.7% | 6.2 | $0.078 | 1.1 s | |
Accuracy83.7%95% range 79.9–87.3% Calibration gap6.2 ptslower is better · 19.5% of answers at confidence 1.00 Automation at 95%—never reaches 95% accuracyshared gate ≥ 0.91: acts on 54%, right 93.0% Speed838 msmedian · p95 1.1 s Stability6.7%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · DeepSeek · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as deepseek/deepseek-v4.1-flashmodel pagepubmedqa run log | ||||||||
| — | Label prior | — | 0.0 | 62.0% | 0.0 | — | — | |
Accuracy62.0%95% range 56.7–67.3% Calibration gap0.0 ptslower is better · 0.0% of answers at confidence 1.00 Automation at 95%—no probabilitiesshared gate ≥ 0.91: acts on 0% Speed—median · p95 — Stability0.0%of answers change on an identical rerun Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupconstant probabilities · reasoning — · temperature — · 300 items × 5 runsmodel page | ||||||||
Wins: a model wins a column when it is one of the two best in it: the highest score, the highest accuracy, the smallest calibration gap, the lowest price or the fastest speed. Ties count for everyone tied. The table is ordered by wins, then by score; models with as many wins share a place, so =3 means tied for third.
Release 2026-09-18 · suite 0.1.0 · board.json · permalink · explore every question · how we score