Skip to content

LLM · Alibaba

Qwen3.8 Flash

Ranks =6 of 7 on noul, =5 of 7 on choice, =5 of 7 on score. Verbalized probabilities through the Jevals adapter, no reasoning, via OpenRouter · Alibaba.

Compare with Jev →

noul

62.4Decision Score
Rank 2 of 8

Accuracy
89.7%
Calibration gap
2.5
Hand-off at 95%
—
$ per 1k
$0.086
Speed p95
3.5 s
10said 52%, right 0% (1 answers)said 65%, right 70% (30 answers)said 73%, right 80% (10 answers)said 85%, right 79% (332 answers)said 95%, right 94% (1127 answers)Said vs right. Squares on the diagonal mean honest confidence; bigger squares hold more answers.
choice

62.4Decision Score
Rank =4 of 8

Accuracy
76.8%
Calibration gap
9.5
Hand-off at 95%
—
$ per 1k
$0.12
Speed p95
3.1 s
10said 35%, right 43% (14 answers)said 45%, right 36% (76 answers)said 50%, right 33% (3 answers)said 64%, right 50% (123 answers)said 75%, right 50% (4 answers)said 85%, right 66% (435 answers)said 95%, right 91% (843 answers)Said vs right. Squares on the diagonal mean honest confidence; bigger squares hold more answers.
score

−1.4Decision Score
Rank =3 of 8

Accuracy
36.1%
Calibration gap
25.8
Hand-off at 95%
—
$ per 1k
$0.13
Speed p95
2.6 s
10said 44%, right 23% (232 answers)said 50%, right 27% (353 answers)said 62%, right 34% (594 answers)said 73%, right 48% (46 answers)said 85%, right 62% (151 answers)said 95%, right 60% (124 answers)Said vs right. Squares on the diagonal mean honest confidence; bigger squares hold more answers.

Details

noul
Accuracy89.7%95% range 86.3–92.7%
Calibration gap2.5 ptslower is better · 0.7% of answers at confidence 1.00
10said 52%, right 0% (1 answers)said 65%, right 70% (30 answers)said 73%, right 80% (10 answers)said 85%, right 79% (332 answers)said 95%, right 94% (1127 answers)
Automation at 95%—never reaches 95% accuracyshared gate ≥ 0.91: acts on 72%, right 93.9%
Speed1.1 smedian · p95 3.5 s
Stability4.7%of answers change on an identical rerun
Valid output100.0%0.0% refused · 1,500 of 1,500 answered
Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Alibaba · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as qwen/qwen3.8-flashpubmedqa run log
choice
Accuracy76.8%95% range 72.6–80.9%
Calibration gap9.5 ptslower is better · 0.0% of answers at confidence 1.00
10said 35%, right 43% (14 answers)said 45%, right 36% (76 answers)said 50%, right 33% (3 answers)said 64%, right 50% (123 answers)said 75%, right 50% (4 answers)said 85%, right 66% (435 answers)said 95%, right 91% (843 answers)
Automation at 95%—never reaches 95% accuracyshared gate ≥ 0.96: acts on 2%, right 100.0%
Speed1.7 smedian · p95 3.1 s
Stability9.0%of answers change on an identical rerun · 22.1% change when options are reordered
Valid output99.9%0.0% refused · 1,498 of 1,500 answered
Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Alibaba · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as qwen/qwen3.8-flashbanking77 run log
score
Accuracy36.1%95% range 31.5–40.7%
Calibration gap25.8 ptslower is better · 0.0% of answers at confidence 1.00
10said 44%, right 23% (232 answers)said 50%, right 27% (353 answers)said 62%, right 34% (594 answers)said 73%, right 48% (46 answers)said 85%, right 62% (151 answers)said 95%, right 60% (124 answers)
Automation at 95%—never reaches 95% accuracy
Speed1.4 smedian · p95 2.6 s
Stability26.0%of answers change on an identical rerun
Valid output100.0%0.0% refused · 1,500 of 1,500 answered
Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Alibaba · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as qwen/qwen3.8-flashhelpsteer2 run log
Model id
qwen/qwen3.8-flash
Served as
qwen/qwen3.8-flash
Adapter
jevals-llm 0383a0e3e592 · the prompt
Host
OpenRouter · Alibaba
Price as of
2026-09-18
Harness
4da1b6d6bc57, 0401f0112fa8
Release
2026-09-18

See every question Qwen3.8 Flash answered →