Skip to content

Compare · release 2026-09-18

Jev vs Gemini 3.8 Flash

Jev vs Gemini 3.8 Flash, scored against human labels. Decision Score: noul 69.0 vs 73.0, choice 67.8 vs 74.1 and score 9.2 vs 4.6. Release 2026-09-18.

Jev is a System One model: it answers with probabilities and writes no text. Gemini 3.8 Flash is an LLM, answering the same questions through an adapter. How Jev differs from an LLM

noul on PubMedQA

Jev Gemini 3.8 Flash
Decision Score69.095% range 59.8 to 76.573.095% range 62.1 to 82.2
Ranktied for rank 1 of 8tied for rank 1 of 8
Accuracy91.3%92.5%
Calibration gap5.0 pts2.0 pts
Hand-off at 95%86%79%
$ per 1k decisions$0.029$0.8
Speed p95653 ms5.5 s

Question by question (right = most of 5 runs right), out of 300: both right on 270, only Jev on 4, only Gemini 3.8 Flash on 9, neither on 17. Same usual pick on 96%. See where they differ

choice on Banking77

Jev Gemini 3.8 Flash
Decision Score67.895% range 61.3 to 74.574.195% range 68.4 to 79.8
Ranktied for rank 2 of 8rank 1 of 8
Accuracy79.7%84.6%
Calibration gap9.8 pts3.2 pts
Hand-off at 95%54%45%
$ per 1k decisions$0.043$1.4
Speed p95693 ms3.9 s

Question by question (right = most of 5 runs right), out of 300: both right on 233, only Jev on 5, only Gemini 3.8 Flash on 22, neither on 40. Same usual pick on 89%. See where they differ

score on HelpSteer2 helpfulness

Jev Gemini 3.8 Flash
Decision Score9.295% range −4.4 to 21.04.695% range −12.2 to 19.0
Ranktied for rank 1 of 8tied for rank 1 of 8
Accuracy41.3%42.4%
Calibration gap19.7 pts22.6 pts
Hand-off at 95%——
$ per 1k decisions$0.036$1.3
Speed p95670 ms4.0 s

Question by question (right = most of 5 runs right), out of 300: both right on 82, only Jev on 42, only Gemini 3.8 Flash on 47, neither on 129. Same usual pick on 56%. See where they differ

Jev · Gemini 3.8 Flash · all models · how we score · board.json