Skip to content

findings · release 2026-09-18

On nouls, Jev is statistically tied with the best of six LLMs at 1/28 of the price

suite 0.1.0 · 300 questions × 5 runs per task · human labels · run from a laptop at UTC+3

Jevals scores models that answer typed questions with probabilities. This first release puts Jev and six of the newest fast and mid-priced LLMs on the same three tasks, one per question type, with the same instructions and option order for every model.

What we found

  1. tied #1, 1/28 the priceNoul (PubMedQA, yes/no): Gemini 3.8 Flash (Decision Score 73.0) and Jev (69.0) share first place: the 95% range of their paired difference, −2.5 to +10.4 points, includes zero. Jev costs $0.03 per 1,000 decisions against $0.80, and its p95 latency is 0.65 s against 5.5 s. noul board
  2. 86% hand-offActing only when confident: Jev's probabilities work as a gate. Giving each model its own threshold for 95% accuracy, Jev can take 86% of noul decisions alone, against 79% for Gemini 3.8 Flash. On choice it takes 54%, ahead of Gemini 3.8 Flash (45%), DeepSeek V4.1 Flash (39%) and GLM-5.3 (38%). GPT-5.6 Luna, whose run was stopped after the choice task and which is not listed, reached 64% there. At the shared choice gate (0.96) Jev acts on 59% of decisions and is right on 94.3% of those; the LLMs state 0.96 or more on at most 15%, partly because they favour round values such as 0.95.
  3. #2 on choiceChoice (Banking77, 77 intents): Gemini 3.8 Flash leads at 74.1 and is significantly ahead of Jev. Jev (67.8, 79.7% accuracy) and GLM-5.3 (66.8) share second place; Jev costs $0.04 per 1,000 decisions against $0.72 for GLM-5.3 and $1.37 for Gemini. choice board
  4. nobody beats guessingScore (HelpSteer2 helpfulness, 0–4): no model clearly beats answering with the label base rates. Jev (9.2), GLM-5.3 (7.8) and Gemini 3.8 Flash (4.6) are indistinguishable from it. DeepSeek V4.1 Flash (−19.0) and Mistral Medium 3.5 (−13.7) score below it (95% ranges below zero; Mistral's only just), with calibration gaps above 33 points. score board

Measured against what was claimed

  • Jev's Banking77 accuracy here, 79.7%, matches the 80.3% a Vercel engineer reported on a different 385-item sample (vercel/ai#20846).
  • TypeSafe quotes 70–500 ms end to end, and about 100 ms for most queries in its docs; Vercel measured medians of 132–272 ms. From this laptop at UTC+3, Jev's median was 0.44–0.48 s and its p95 0.65–0.69 s, one question per request. Jev is served from one US region, so this includes the trip.
  • Price: $0.042 per million input tokens, output free. Measured: $0.03–0.04 per 1,000 decisions.

Read these numbers with care

  • One public dataset per question type, so "noul" here means PubMedQA. Calibration in particular is known to vary by task.
  • No frontier LLM is listed yet: one full run of GPT-6 Astra (about $39) or Claude Fable 5.1 (about $62) costs more than this release's whole $25 budget. "The best of six" means the six listed.
  • LLM probabilities are stated by the model in its reply (verbalized), not read from token probabilities.
  • Latency is measured from one residential connection; compare rows with each other.
  • The Gateway does not report which Jev version answered; TypeSafe lists one, jev-1.13.0.

Every number above can be recomputed from the published per-decision logs: see methodology and board.json (CC-BY-4.0). This release cost $15.07 in measured API spend, plus $3.98 on runs that were started and then stopped; the changelog lists them and their completed results.