methodology · suite 0.1.0

How Jevals scores a decision

Four ideas cover everything on the boards. The full recipe follows, precise enough to recompute every number from the published logs.

Glossary

System One model

A model that answers typed questions with probabilities in one parallel pass and writes no text. Jev, from TypeSafe AI, is the first. The name borrows Kahneman's fast, intuitive System 1.

System Two

The slow, deliberate counterpart: LLMs that reason in text. On Jevals they answer the same questions through an adapter.

State

The input a question is about: any text or JSON.

Primitive

One of the three typed questions a System One model answers.

noul

Yes or no, answered with P(yes). Short for Bernoulli. In code, an if.

choice

Pick one of up to 255 options, with a probability per option. In code, a match or switch.

score

Place the state on an ordered rubric of 2 to 10 levels, with a probability per level. In code, something to sort by.

Criteria

The descriptions of the options or levels that come with a question.

Confidence

How sure an answer is: here, the probability of the pick.

Gate

A confidence threshold: act at or above it, escalate below it.

Adapter

The prompt that makes an LLM answer a typed question with a probability for each answer, so it can be scored like a System One model.

Jaggedness

TypeSafe's word for the documented weak spots of a Jev version, such as counting, dates, negation and distracting text in the state.

The full recipe

What is measured

Jevals scores decisions, not text. A system reads a state and answers one typed question with a probability distribution over the allowed answers. There are three question types, the three primitives of Jev's interface:

• Noul (noul, yes/no): returns P(yes).
• Choice: pick one of K options. Returns a probability per option.
• Score: place the state on an ordered rubric of levels. Returns a probability per level.

Each primitive has its own board and its own ranking. There is no overall index across primitives. Every label is ground truth from a public human-labelled dataset. No model grades another model.

Tasks

Suite 0.1.0 has one task per primitive. Each task is a fixed sample of 300 items drawn by proportional allocation (largest remainder) over the split's natural label distribution, after dropping items whose state is longer than 6,000 Unicode code points, with a fixed seed. Every item is answered 5 times. Item ids, upstream row indices, labels, class counts and the dataset revision are in each task's suite file, linked below.

The state is a JSON object built only from the whitelisted fields listed above, so no field that reveals the label reaches any system. Item text is not republished here; each item links to its upstream row. The state hash (state_sha256, SHA-256 of the UTF-8 bytes of JSON.stringify(state)) is checked before every paid run.

Systems

• Jev is called through Vercel AI Gateway (typesafe-ai/jev) with the AI SDK's experimental_evaluate, SDK retries disabled. Its native probabilities are used as returned. The Gateway does not report which Jev version answered; on 2026-09-18 TypeSafe lists one version, jev-1.13.0, behind its jev-latest alias (models). A new Jev version means a rerun in a new release.
• LLMs are called through OpenRouter with one adapter for every model family: the same prompt, the same options, prompt-only JSON (no structured-output mode, which hosts support unevenly), one pinned host per model with fallbacks disabled, provider default temperature, and the lowest reasoning setting the host allows unless the row says otherwise. The adapter asks for a verbalized probability distribution.
• Baselines: the label prior answers every item with the base rates of the evaluated items; it defines 0 on the Decision Score scale.

Every system gets one question per request (batch size 1), zero-shot, with identical instructions and criteria. Choice options are presented in a seeded random order that is identical for every system: repeats 0 and 1 share one order, repeats 2, 3 and 4 each get a new one. Score levels and yes/no are never reordered.

The LLM adapter prompt

Prompt hash 0383a0e3e592. Placeholders in braces are filled per item.

You are answering one typed decision question about a state.

STATE (JSON):
{state}

QUESTION: {instructions}

{options_heading}
{options}

Give a probability for {what}
Reply with only this JSON object and nothing else:
{"probabilities": {"<option>": <probability between 0 and 1>}}

For questions with more than 10 options the adapter asks for the 5 most likely options with probabilities; the remaining mass is spread evenly over the unlisted options. For 10 or fewer options it asks for every option.

Probability vectors

Every answer becomes a probability vector over the task's options before scoring.

• Jev's values are rounded to 2 decimals on the wire. Vectors that sum to 0.99 are renormalized.
• An answer is malformed if it is not a JSON object with a probability map, names an unknown or duplicate option (names must match exactly, including case and punctuation), has a value that is not a finite number in [0, 1], or sums to 0. For LLM replies it is also malformed if it lists more than 5 options when asked for the 5 most likely, has other fields after the probability map, or is surrounded by other text containing braces.
• If the listed values sum to more than 1 they are divided by their sum. If they sum to less than 1, the remainder is spread evenly over unlisted options (top-5 mode) or the vector is divided by its sum (full mode).
• The system's pick is the most probable option. For choice, ties go to the option the system listed first (for Jev, its own choice field); for score, ties go to the lower level. A yes/no answer of exactly 0.5 has no pick and counts as wrong.

Decision Score

Decision Score = 100 × (1 − L_system / L_prior)

L is the mean per-item loss. Each item's loss is the average over its 5 repeats of the multiclass Brier score, Σ_k(p_k − y_k)², for choice and yes/no, and of the ranked probability score over cumulative level probabilities, Σ_k<K(P_k − Y_k)² / (K − 1), for score. L_prior is the same loss for the label prior on the same items.

100 = perfect. 0 = no better than answering with the label base rates. Below 0 = worse than that. Negative scores are shown, not clamped; the chart floors at −10 and marks rows below it. Both losses are proper scoring rules: they reward being right and being honest about uncertainty together, and cannot be gamed by overconfidence. A system with no probabilities (one-hot answers) is scored as one-hot. With more than one task in a tab, the tab score is the plain mean of the task scores.

Accuracy

The share of decisions whose pick equals the label, over all items and repeats. Score uses the most probable level (exact match). Refused and malformed answers count as wrong.

Calibration gap (ECE)

Expected calibration error on the top label: confidence is the probability of the pick (for yes/no, the larger of P(yes) and P(no)). Decisions go into 10 equal-width bins by min(9, floor(round(100·c) / 10)), so a confidence of exactly 1.00 lands in the last bin. ECE = Σ_b (n_b/N) · |accuracy_b − mean confidence_b|, shown in points (5.8 means 0.058). Lower is better. Rows without probabilities show a dash. ECE never sets the Decision Score rank: calibration comparisons only mean something at similar accuracy. It is one of the five columns a model can win in the board order (below). The row detail also shows the share of answers at confidence 1.00 and a 95% interval for ECE.

95% ranges and ranks

Intervals are 95% percentile intervals from an item-cluster bootstrap: 2,000 seeded resamples of items with replacement, where all repeats of an item move together, and the prior's loss is recomputed on each resample. Every row in a tab uses the same resamples, so differences between rows are paired.

Rank = 1 + the number of rows that are significantly better, where row j is significantly better than row i if the 95% interval of DS_j − DS_i over the shared resamples lies wholly above 0. Rows that cannot be told apart share a rank. The pairwise tests are not adjusted for multiple comparisons.

Board order. A model wins a column when it is in that column's top two, ties included: the two highest Decision Scores (none when the label prior ties for first), the two highest accuracies, the two lowest calibration gaps, the two lowest prices and the two lowest p95 times. Boards list models by number of wins, then by Decision Score, and models with as many wins share a place. The label prior has no wins and no place. The Decision Score rank stays in the data (rank_ub) and on model pages.

Confidence gate and hand-off

Jev-type systems are meant to be used with a gate: act when confident, escalate otherwise. For each primitive the gate t is the smallest confidence on the 0.01 grid at which the pooled error of all non-baseline decisions with confidence ≥ t is at most 5%, with at least 100 such decisions. If no such t exists the gate is empty. The gate is computed once per suite version from the first release and then frozen. Each row's detail shows its coverage (share of its decisions at or above t) and its accuracy on those decisions. Rows without probabilities (one-hot answers) have no coverage. One shared threshold favours systems whose stated confidence sits above it; LLMs that state round values such as 0.95 fall just short of a 0.96 gate, so compare coverage at matched accuracy too. Current gates: choice 0.96 · score none · noul 0.91.

Hand-off at 95% gives each system its own threshold instead: the lowest confidence t on the 0.01 grid at which its decisions with confidence ≥ t (at least 100 of them) are at least 95% correct. Its hand-off share is those decisions over all its decisions, refused and malformed ones included. The threshold is chosen on the same items it is measured on, so the share is optimistic in the same way for every system. Rows whose accuracy never reaches 95% at any threshold show a dash. Board rows also show coverage at the shared gate.

Flip rates

Repeat flip rate: share of items whose pick differs between repeats 0 and 1, which are identical requests. It measures nondeterminism. Order flip rate (choice only): share of items whose pick is not the same across the four distinct option orders (repeats 0, 2, 3, 4). It mixes option-order sensitivity with nondeterminism; compare it with the repeat flip rate. Refused and malformed answers are excluded.

Cost

$ per 1k decisions = total cost of all calls, including malformed-output retries and reasoning tokens, divided by the number of decisions, times 1,000. Cost is always logged token usage × the list price snapshot stored in the run header (input and output price per token for the pinned host; Jev: $0.042 per million input tokens, output free). The one discarded warm-up call at the start of each run is charged to that run. All systems are called one question per request; batching several questions per call is cheaper for every system and is not shown in the main column.

Latency

p95 is the 95th percentile of end-to-end time from sending the request to having a parsed, validated answer, across all answered decisions, nearest-rank. It includes malformed-output retries but not transport-error backoff. Requests run at concurrency 4 after one discarded warm-up call. All rows are measured from the same machine, stated on every board as run from; that is a residential connection, not a datacenter, so compare rows with each other rather than with vendor claims. Each run's time window is in its row detail.

Refusals, malformed answers and transport failures

• Malformed output is retried up to 2 times; the retries count in cost and latency.
• A refusal (a provider refusal field or a content-filter stop) is not retried.
• After retries, a refused or malformed answer is scored as the uniform distribution and a wrong pick. It counts in the Decision Score and accuracy, and is excluded from ECE, flip rates and the gate. Rows report schema-valid rate, refusal rate and the number answered.
• Transport failures (HTTP errors, provider errors, a 60 s timeout, a truncated response) are retried with backoff and never scored. A decision that still fails is not written, and the run resumes it later. A board cannot be built while any (item, repeat) is missing.

Versions and snapshots

Every board is a release with a permanent URL under /r/<release>/. A patch changes no score. A minor version changes how stored logs are graded and regrades them without new calls. Adding, removing or re-sampling a task is a new suite version whose scores are not comparable with the previous one; the old release stays frozen. Launch suite is 0.1.0; 1.0.0 is reserved for the full suite. Field names reserved for later case-level (workflow) scoring: case_id, action_set_label, adjudicator_ids, adjudication_date.

Known limits

• All items come from public datasets that are probably in LLM pretraining data. There is no private held-out set yet; one needs authored items and will come with them.
• One dataset per primitive in suite 0.1.0, so each tab is one task. Results may not transfer to other tasks; calibration in particular is known to be task-dependent.
• LLM probabilities are verbalized (the model states them). They tend to cluster on round values. Logprob-based rows are not in this version.
• Latency is measured from one residential location; Jev is served from one US region.
• Labels are taken as published. Public datasets have label noise.
• In adapter prompt v1 the yes/no options read - yes: Yes: the passages support…. One retired model copied the whole line as the option name; none of the listed models did (0 malformed yes/no answers). The next adapter version quotes option names.

Verify it yourself

Each board page links its board.json and the run log behind every row, and the public jevals-data repository collects them with the suite files. A run log's first line describes the system, its prices and the harness version; every other line is one decision: the item, the repeat, the true label, the answer with its stated probabilities, its cost and its latency. The formulas on this page turn those lines into every number on the board. If your numbers differ from ours, open an issue.