# choice (pick one of N) benchmark: Jev vs 6 LLMs

Gemini 3.8 Flash (74.1) ranks first in Decision Score on Banking77. Jev (67.8, tied for rank 2 of 8): $0.043 per 1,000 decisions, p95 693 ms. Release 2026-09-18.

Pick one option from a list. The model returns a probability for every option, like a match over its cases. The test: “Which intent does this banking customer's message express?” (Banking77, 77 options, 300 questions × 5 runs, human labels, https://huggingface.co/datasets/mteb/banking77).

| Rank | Model | Type | Decision Score (95% range) | Accuracy | Calibration gap (ECE, pts) | Hand-off at 95% | $ per 1k decisions | Speed p95 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | [Gemini 3.8 Flash](https://jevals.com/models/gemini-3.8-flash/) | LLM, Google, reasoning low | 74.1 (68.4 to 79.8) | 84.6% | 3.2 | 45% | $1.4 | 3.9 s |
| =2 | [Jev](https://jevals.com/models/jev/) | System One, TypeSafe AI | 67.8 (61.3 to 74.5) | 79.7% | 9.8 | 54% | $0.043 | 693 ms |
| =2 | [GLM-5.3](https://jevals.com/models/glm-5.3/) | LLM, Z.ai, reasoning low | 66.8 (62.0 to 71.7) | 78.9% | 6.2 | 38% | $0.72 | 3.2 s |
| =4 | [DeepSeek V4.1 Flash](https://jevals.com/models/deepseek-v4.1-flash/) | LLM, DeepSeek, no reasoning | 63.9 (58.5 to 69.6) | 75.9% | 2.8 | 39% | $0.14 | 1.3 s |
| =4 | [Qwen3.8 Flash](https://jevals.com/models/qwen3.8-flash/) | LLM, Alibaba, no reasoning | 62.4 (55.9 to 68.9) | 76.8% | 9.5 | — | $0.12 | 3.1 s |
| 5 | [Mistral Medium 3.5](https://jevals.com/models/mistral-medium-3.5/) | LLM, Mistral AI, no reasoning | 59.5 (54.3 to 64.8) | 74.5% | 8.1 | 11% | $1.5 | 1.4 s |
| 7 | [Mercury 2.5](https://jevals.com/models/mercury-2.5/) | LLM, Inception, no reasoning | 54.0 (47.8 to 60.6) | 70.3% | 10.4 | — | $0.036 | 1.5 s |
| 8 | [Label prior](https://jevals.com/models/label-prior/) | baseline | 0.0 (0.0 to 0.0) | 1.3% | 0.0 | — | — | — |

Decision Score: 100 = perfect, 0 = guessing the label base rates, below 0 = worse than that. Ranks: 1 + the number of rows significantly better (paired item bootstrap, 95%); rows that cannot be told apart share a rank. Hand-off at 95%: the share of decisions a model can take alone, at its own confidence threshold, while staying at least 95% right. "=" before a rank marks a tie. Shared confidence gate for this board: 0.96.

Source: https://jevals.com/choice/ · release 2026-09-18, suite 0.1.0 · data (CC-BY-4.0): https://jevals.com/data/releases/2026-09-18/board.json · cite as: Jevals (jevals.com), release 2026-09-18, suite 0.1.0. CC-BY-4.0.