Compare · release 2026-09-18
Jev vs DeepSeek V4.1 Flash
Jev vs DeepSeek V4.1 Flash, scored against human labels. Decision Score: noul 69.0 vs 47.5, choice 67.8 vs 63.9 and score 9.2 vs −19.0. Release 2026-09-18.
noul on PubMedQA
| Jev | DeepSeek V4.1 Flash | |
|---|---|---|
| Decision Score | 69.095% range 59.8 to 76.5 | 47.595% range 34.7 to 59.1 |
| Rank | tied for rank 1 of 8 | rank 7 of 8 |
| Accuracy | 91.3% | 83.7% |
| Calibration gap | 5.0 pts | 6.2 pts |
| Hand-off at 95% | 86% | — |
| $ per 1k decisions | $0.029 | $0.078 |
| Speed p95 | 653 ms | 1.1 s |
Question by question (right = most of 5 runs right), out of 300: both right on 249, only Jev on 25, only DeepSeek V4.1 Flash on 4, neither on 22. Same usual pick on 90%. See where they differ
choice on Banking77
| Jev | DeepSeek V4.1 Flash | |
|---|---|---|
| Decision Score | 67.895% range 61.3 to 74.5 | 63.995% range 58.5 to 69.6 |
| Rank | tied for rank 2 of 8 | tied for rank 4 of 8 |
| Accuracy | 79.7% | 75.9% |
| Calibration gap | 9.8 pts | 2.8 pts |
| Hand-off at 95% | 54% | 39% |
| $ per 1k decisions | $0.043 | $0.14 |
| Speed p95 | 693 ms | 1.3 s |
Question by question (right = most of 5 runs right), out of 300: both right on 221, only Jev on 17, only DeepSeek V4.1 Flash on 14, neither on 48. Same usual pick on 88%. See where they differ
score on HelpSteer2 helpfulness
| Jev | DeepSeek V4.1 Flash | |
|---|---|---|
| Decision Score | 9.295% range −4.4 to 21.0 | −19.095% range −36.0 to −3.3 |
| Rank | tied for rank 1 of 8 | rank 7 of 8 |
| Accuracy | 41.3% | 34.7% |
| Calibration gap | 19.7 pts | 33.5 pts |
| Hand-off at 95% | — | — |
| $ per 1k decisions | $0.036 | $0.13 |
| Speed p95 | 670 ms | 1.2 s |
Question by question (right = most of 5 runs right), out of 300: both right on 74, only Jev on 50, only DeepSeek V4.1 Flash on 26, neither on 150. Same usual pick on 61%. See where they differ
Jev · DeepSeek V4.1 Flash · all models · how we score · board.json