Skip to content

You are looking at the frozen release 2026-09-18. See the latest boards

evals for models
that decide

Jev is the first System One model: it writes no text, it answers typed questions with probabilities, in one fast pass. Jevals checks how good those answers are, against human labels, next to the LLMs that try to decide the same way.

3 question types7 models31,500 scored decisions

Jevals is independent. It is not affiliated with TypeSafe AI or Jev and does not represent them.

three questions, three boards

System One LLM top 2 for that metric (score, cost, speed)

noul→ if

A yes/no question. The model returns P(yes), and your code acts on it with an if.

if (p_yes > 0.9) approve()

PubMedQA · 300 questions

  1. Jev69.0
  2. Gemini 3.8 Flash73.0
  3. GLM-5.360.6
open the noul board →
choice→ match

Pick one option from a list. The model returns a probability for every option, like a match over its cases.

switch (intent) { case … }

Banking77 · 300 questions

  1. Jev67.8
  2. Gemini 3.8 Flash74.1
  3. DeepSeek V4.1 Flash63.9
open the choice board →
score→ sort

Place the state on an ordered rubric. The model returns a probability for every level, ready to sort by.

replies.sort(by(score))

HelpSteer2 helpfulness · 300 questions

  1. Jev9.2
  2. GLM-5.37.8
  3. Mercury 2.5−5.5

No model clearly beats guessing yet.

open the score board →

Decision Score: 100 = perfect, 0 = guessing the label base rates. Bars on a 0–100 scale.

how a System One model decides

  1. state{ "question": "…", "abstract": "…" }any text or JSON
  2. questionnoul: "Does the abstract support yes?"typed: noul, choice or score
  3. answerP(yes) = 0.93probabilities, no text
  4. your codeif (0.93 > 0.9) act() else askHuman()acts only when confident

Jevals asks every model the same questions, 5 times each, and scores every answer against human labels. LLMs, the System Two counterpart that thinks in text, answer through an adapter that asks them for the same probabilities. How we score

findings · release 2026-09-18On nouls, Jev is statistically tied with the best of six LLMs at 1/28 of the priceread the findings →
coming next

GPT-6 Astra · Claude Fable 5.1 · Grok 4.6 · Qwen3.8 Max · Kimi K3 · decider-2b · openjev · GLiClass · TF-IDF + logistic regression · logprob-based LLM rows

Every row is run by us at list price. How we list models