Skip to content

LLM · DeepSeek

DeepSeek V4.1 Flash

Ranks =6 of 7 on noul, 3 of 7 on choice, =5 of 7 on score. Verbalized probabilities through the Jevals adapter, no reasoning, via OpenRouter · DeepSeek.

Compare with Jev →

noul

47.5Decision Score
Rank 7 of 8

Accuracy
83.7%
Calibration gap
6.2
Hand-off at 95%
—
$ per 1k
$0.078
Speed p95
1.1 s
10said 52%, right 14% (7 answers)said 62%, right 41% (51 answers)said 71%, right 66% (145 answers)said 84%, right 72% (304 answers)said 96%, right 92% (993 answers)Said vs right. Squares on the diagonal mean honest confidence; bigger squares hold more answers.
choice

63.9Decision Score
Rank =4 of 8

Accuracy
75.9%
Calibration gap
2.8
Hand-off at 95%
39%
$ per 1k
$0.14
Speed p95
1.3 s
10said 18%, right 0% (2 answers)said 26%, right 0% (3 answers)said 35%, right 22% (50 answers)said 44%, right 44% (109 answers)said 55%, right 50% (155 answers)said 63%, right 66% (183 answers)said 73%, right 69% (216 answers)said 84%, right 88% (219 answers)said 95%, right 96% (561 answers)Said vs right. Squares on the diagonal mean honest confidence; bigger squares hold more answers.
score

−19.0Decision Score
Rank 7 of 8

Accuracy
34.7%
Calibration gap
33.5
Hand-off at 95%
—
$ per 1k
$0.13
Speed p95
1.2 s
10said 34%, right 9% (11 answers)said 44%, right 16% (102 answers)said 54%, right 22% (351 answers)said 62%, right 28% (346 answers)said 73%, right 34% (267 answers)said 85%, right 54% (261 answers)said 95%, right 60% (162 answers)Said vs right. Squares on the diagonal mean honest confidence; bigger squares hold more answers.

Details

noul
Accuracy83.7%95% range 79.9–87.3%
Calibration gap6.2 ptslower is better · 19.5% of answers at confidence 1.00
10said 52%, right 14% (7 answers)said 62%, right 41% (51 answers)said 71%, right 66% (145 answers)said 84%, right 72% (304 answers)said 96%, right 92% (993 answers)
Automation at 95%—never reaches 95% accuracyshared gate ≥ 0.91: acts on 54%, right 93.0%
Speed838 msmedian · p95 1.1 s
Stability6.7%of answers change on an identical rerun
Valid output100.0%0.0% refused · 1,500 of 1,500 answered
Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · DeepSeek · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as deepseek/deepseek-v4.1-flashpubmedqa run log
choice
Accuracy75.9%95% range 71.9–80.1%
Calibration gap2.8 ptslower is better · 0.6% of answers at confidence 1.00
10said 18%, right 0% (2 answers)said 26%, right 0% (3 answers)said 35%, right 22% (50 answers)said 44%, right 44% (109 answers)said 55%, right 50% (155 answers)said 63%, right 66% (183 answers)said 73%, right 69% (216 answers)said 84%, right 88% (219 answers)said 95%, right 96% (561 answers)
Automation at 95%39%of decisions, acting at confidence ≥ 0.86 (right 95.8%)shared gate ≥ 0.96: acts on 15%, right 99.1%
Speed999 msmedian · p95 1.3 s
Stability10.7%of answers change on an identical rerun · 30.2% change when options are reordered
Valid output99.9%0.0% refused · 1,498 of 1,500 answered
Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · DeepSeek · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as deepseek/deepseek-v4.1-flashbanking77 run log
score
Accuracy34.7%95% range 30.5–38.7%
Calibration gap33.5 ptslower is better · 1.5% of answers at confidence 1.00
10said 34%, right 9% (11 answers)said 44%, right 16% (102 answers)said 54%, right 22% (351 answers)said 62%, right 28% (346 answers)said 73%, right 34% (267 answers)said 85%, right 54% (261 answers)said 95%, right 60% (162 answers)
Automation at 95%—never reaches 95% accuracy
Speed912 msmedian · p95 1.2 s
Stability34.3%of answers change on an identical rerun
Valid output100.0%0.0% refused · 1,500 of 1,500 answered
Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · DeepSeek · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as deepseek/deepseek-v4.1-flashhelpsteer2 run log
Model id
deepseek/deepseek-v4.1-flash
Served as
deepseek/deepseek-v4.1-flash
Adapter
jevals-llm 0383a0e3e592 · the prompt
Host
OpenRouter · DeepSeek
Price as of
2026-09-18
Harness
4da1b6d6bc57
Release
2026-09-18

See every question DeepSeek V4.1 Flash answered →