# Jev vs Gemini 3.8 Flash

Jev vs Gemini 3.8 Flash, scored against human labels. Decision Score: noul 69.0 vs 73.0, choice 67.8 vs 74.1 and score 9.2 vs 4.6. Release 2026-09-18.

## noul (PubMedQA)

|  | Jev | Gemini 3.8 Flash |
| --- | --- | --- |
| Decision Score (95% range) | 69.0 (59.8 to 76.5) | 73.0 (62.1 to 82.2) |
| Rank | tied for rank 1 of 8 | tied for rank 1 of 8 |
| Accuracy | 91.3% | 92.5% |
| Calibration gap (ECE, pts) | 5.0 | 2.0 |
| Hand-off at 95% | 86% | 79% |
| $ per 1k decisions | $0.029 | $0.8 |
| Speed p95 | 653 ms | 5.5 s |

Question by question (right = most of 5 runs right), out of 300: both right on 270, only Jev on 4, only Gemini 3.8 Flash on 9, neither on 17. Same usual pick on 96%.

## choice (Banking77)

|  | Jev | Gemini 3.8 Flash |
| --- | --- | --- |
| Decision Score (95% range) | 67.8 (61.3 to 74.5) | 74.1 (68.4 to 79.8) |
| Rank | tied for rank 2 of 8 | rank 1 of 8 |
| Accuracy | 79.7% | 84.6% |
| Calibration gap (ECE, pts) | 9.8 | 3.2 |
| Hand-off at 95% | 54% | 45% |
| $ per 1k decisions | $0.043 | $1.4 |
| Speed p95 | 693 ms | 3.9 s |

Question by question (right = most of 5 runs right), out of 300: both right on 233, only Jev on 5, only Gemini 3.8 Flash on 22, neither on 40. Same usual pick on 89%.

## score (HelpSteer2 helpfulness)

|  | Jev | Gemini 3.8 Flash |
| --- | --- | --- |
| Decision Score (95% range) | 9.2 (−4.4 to 21.0) | 4.6 (−12.2 to 19.0) |
| Rank | tied for rank 1 of 8 | tied for rank 1 of 8 |
| Accuracy | 41.3% | 42.4% |
| Calibration gap (ECE, pts) | 19.7 | 22.6 |
| Hand-off at 95% | — | — |
| $ per 1k decisions | $0.036 | $1.3 |
| Speed p95 | 670 ms | 4.0 s |

Question by question (right = most of 5 runs right), out of 300: both right on 82, only Jev on 42, only Gemini 3.8 Flash on 47, neither on 129. Same usual pick on 56%.

Decision Score: 100 = perfect, 0 = guessing the label base rates, below 0 = worse than that. Ranks: 1 + the number of rows significantly better (paired item bootstrap, 95%); rows that cannot be told apart share a rank. Hand-off at 95%: the share of decisions a model can take alone, at its own confidence threshold, while staying at least 95% right.

Source: https://jevals.com/compare/jev-vs-gemini-3.8-flash/ · release 2026-09-18, suite 0.1.0 · data (CC-BY-4.0): https://jevals.com/data/releases/2026-09-18/board.json · cite as: Jevals (jevals.com), release 2026-09-18, suite 0.1.0. CC-BY-4.0.