compare · release 2026-09-18
Jev vs Mistral Medium 3.5
Jev vs Mistral Medium 3.5, scored against human labels. Decision Score: noul 69.0 vs 58.0, choice 67.8 vs 59.5 and score 9.2 vs −13.7. Release 2026-09-18.
noul → if · PubMedQA
| Jev | Mistral Medium 3.5 | |
|---|---|---|
| Decision Score | 69.095% range 59.8 to 76.5 | 58.095% range 45.3 to 69.8 |
| Rank | tied for rank 1 of 8 | tied for rank 3 of 8 |
| Accuracy | 91.3% | 88.8% |
| Calibration gap | 5.0 pts | 5.2 pts |
| Hand-off at 95% | 86% | 65% |
| $ per 1k decisions | $0.029 | $0.87 |
| Speed p95 | 653 ms | 917 ms |
Question by question (right = most of 5 runs right), out of 300: both right on 256, only Jev on 18, only Mistral Medium 3.5 on 10, neither on 16. Same usual pick on 91%. See where they differ
choice → match · Banking77
| Jev | Mistral Medium 3.5 | |
|---|---|---|
| Decision Score | 67.895% range 61.3 to 74.5 | 59.595% range 54.3 to 64.8 |
| Rank | tied for rank 2 of 8 | rank 5 of 8 |
| Accuracy | 79.7% | 74.5% |
| Calibration gap | 9.8 pts | 8.1 pts |
| Hand-off at 95% | 54% | 11% |
| $ per 1k decisions | $0.043 | $1.5 |
| Speed p95 | 693 ms | 1.4 s |
Question by question (right = most of 5 runs right), out of 300: both right on 212, only Jev on 26, only Mistral Medium 3.5 on 12, neither on 50. Same usual pick on 83%. See where they differ
score → sort · HelpSteer2 helpfulness
| Jev | Mistral Medium 3.5 | |
|---|---|---|
| Decision Score | 9.295% range −4.4 to 21.0 | −13.795% range −28.3 to −0.1 |
| Rank | tied for rank 1 of 8 | rank 5 of 8 |
| Accuracy | 41.3% | 43.9% |
| Calibration gap | 19.7 pts | 33.6 pts |
| Hand-off at 95% | — | — |
| $ per 1k decisions | $0.036 | $1.3 |
| Speed p95 | 670 ms | 1.2 s |
Question by question (right = most of 5 runs right), out of 300: both right on 63, only Jev on 61, only Mistral Medium 3.5 on 69, neither on 107. Same usual pick on 42%. See where they differ
Jev · Mistral Medium 3.5 · all models · how we score · board.json