compare · release 2026-09-18
Jev vs Gemini 3.8 Flash
Jev vs Gemini 3.8 Flash, scored against human labels. Decision Score: noul 69.0 vs 73.0, choice 67.8 vs 74.1 and score 9.2 vs 4.6. Release 2026-09-18.
noul → if · PubMedQA
| Jev | Gemini 3.8 Flash | |
|---|---|---|
| Decision Score | 69.095% range 59.8 to 76.5 | 73.095% range 62.1 to 82.2 |
| Rank | tied for rank 1 of 8 | tied for rank 1 of 8 |
| Accuracy | 91.3% | 92.5% |
| Calibration gap | 5.0 pts | 2.0 pts |
| Hand-off at 95% | 86% | 79% |
| $ per 1k decisions | $0.029 | $0.8 |
| Speed p95 | 653 ms | 5.5 s |
Question by question (right = most of 5 runs right), out of 300: both right on 270, only Jev on 4, only Gemini 3.8 Flash on 9, neither on 17. Same usual pick on 96%. See where they differ
choice → match · Banking77
| Jev | Gemini 3.8 Flash | |
|---|---|---|
| Decision Score | 67.895% range 61.3 to 74.5 | 74.195% range 68.4 to 79.8 |
| Rank | tied for rank 2 of 8 | rank 1 of 8 |
| Accuracy | 79.7% | 84.6% |
| Calibration gap | 9.8 pts | 3.2 pts |
| Hand-off at 95% | 54% | 45% |
| $ per 1k decisions | $0.043 | $1.4 |
| Speed p95 | 693 ms | 3.9 s |
Question by question (right = most of 5 runs right), out of 300: both right on 233, only Jev on 5, only Gemini 3.8 Flash on 22, neither on 40. Same usual pick on 89%. See where they differ
score → sort · HelpSteer2 helpfulness
| Jev | Gemini 3.8 Flash | |
|---|---|---|
| Decision Score | 9.295% range −4.4 to 21.0 | 4.695% range −12.2 to 19.0 |
| Rank | tied for rank 1 of 8 | tied for rank 1 of 8 |
| Accuracy | 41.3% | 42.4% |
| Calibration gap | 19.7 pts | 22.6 pts |
| Hand-off at 95% | — | — |
| $ per 1k decisions | $0.036 | $1.3 |
| Speed p95 | 670 ms | 4.0 s |
Question by question (right = most of 5 runs right), out of 300: both right on 82, only Jev on 42, only Gemini 3.8 Flash on 47, neither on 129. Same usual pick on 56%. See where they differ
Jev · Gemini 3.8 Flash · all models · how we score · board.json