compare · release 2026-09-18
Jev vs Mercury 2.5
Jev vs Mercury 2.5, scored against human labels. Decision Score: noul 69.0 vs 55.7, choice 67.8 vs 54.0 and score 9.2 vs −5.5. Release 2026-09-18.
noul → if · PubMedQA
| Jev | Mercury 2.5 | |
|---|---|---|
| Decision Score | 69.095% range 59.8 to 76.5 | 55.795% range 44.8 to 65.4 |
| Rank | tied for rank 1 of 8 | tied for rank 3 of 8 |
| Accuracy | 91.3% | 87.1% |
| Calibration gap | 5.0 pts | 4.7 pts |
| Hand-off at 95% | 86% | 19% |
| $ per 1k decisions | $0.029 | $0.023 |
| Speed p95 | 653 ms | 1.0 s |
Question by question (right = most of 5 runs right), out of 300: both right on 257, only Jev on 17, only Mercury 2.5 on 7, neither on 19. Same usual pick on 92%. See where they differ
choice → match · Banking77
| Jev | Mercury 2.5 | |
|---|---|---|
| Decision Score | 67.895% range 61.3 to 74.5 | 54.095% range 47.8 to 60.6 |
| Rank | tied for rank 2 of 8 | rank 7 of 8 |
| Accuracy | 79.7% | 70.3% |
| Calibration gap | 9.8 pts | 10.4 pts |
| Hand-off at 95% | 54% | — |
| $ per 1k decisions | $0.043 | $0.036 |
| Speed p95 | 693 ms | 1.5 s |
Question by question (right = most of 5 runs right), out of 300: both right on 209, only Jev on 29, only Mercury 2.5 on 7, neither on 55. Same usual pick on 83%. See where they differ
score → sort · HelpSteer2 helpfulness
| Jev | Mercury 2.5 | |
|---|---|---|
| Decision Score | 9.295% range −4.4 to 21.0 | −5.595% range −16.2 to 4.0 |
| Rank | tied for rank 1 of 8 | tied for rank 3 of 8 |
| Accuracy | 41.3% | 41.7% |
| Calibration gap | 19.7 pts | 31.5 pts |
| Hand-off at 95% | — | — |
| $ per 1k decisions | $0.036 | $0.034 |
| Speed p95 | 670 ms | 1.2 s |
Question by question (right = most of 5 runs right), out of 300: both right on 67, only Jev on 57, only Mercury 2.5 on 61, neither on 115. Same usual pick on 48%. See where they differ
Jev · Mercury 2.5 · all models · how we score · board.json