Fast decision models in agent loops

Reading right,
acting wrong.

System One models pick an agent's next action in a single pass, with no reasoning to inspect. When they choose wrongly, did they misread what the agent has seen, or read it correctly and still act wrongly? We ask both questions of the same state, separately, across 18 systems — and for strong models the answer is almost always the second.

Yubo Li · Ramayya Krishnan · Rema Padman Carnegie Mellon University

Read the findings → How it works Paper on arXiv ↗

Each square is one system's 19,663 paired test states. Width: the share read correctly (every factual probe right). Height: the share acted on correctly. Orange = read right, acted wrong.

Leaderboard

Every system answers the same frozen requests: identical states, questions, options and option order. Sort by any column; click a row for its domains and serving details. The reasoning model is shown for reference and not ranked.

read right, acted rightread right, acted wrongmisread, acted rightmisread, acted wrong

Findings & insights

Six things the leaderboard doesn't show.

Each chart is live: hover for exact values, switch domains or systems, and compare. Numbers are pooled over the paired test states unless a study says otherwise.

Most wrong actions follow correct reading.

For the six most accurate systems, including the reasoning model, 73–85% of decision errors happen on states where every measured fact was answered correctly. The more accurate the system, the larger that share: stronger models mainly stop misreading, not stop choosing wrongly.

A wrong action is weak evidence of misreading. Measuring reading separately is the only way to tell.

Accuracy vs. errors after correct reading

Each mark is a system (shape: type). Systems below 0.5 accuracy or with low coverage are faded; their errors are mostly input-limit refusals.

The errors gather on the same states.

If facts-right, action-wrong errors were noise, they would spread across states. Instead, all 14 competent fast systems fail this way on the same 141 states, where a permutation null expects none. Each system is right on 74–94% of states no other system fails, but on only 1–16% of states all the others fail.

These are properties of states, not quirks of models — and a reasoning model resolves fewer than half of them.

States by how many systems fail them with every fact right

Bars: observed paired test states; ticks: within-cell permutation null (mean of 200).

Accuracy on states that k other systems fail

Thin lines: each competent system, leaving itself out. Dashed: the reasoning reference.

When the facts provably suffice, models fail to compose them.

On generated states where four facts determine the action, systems read every fact (97–100%) and many decide perfectly from the facts alone — yet lose up to 45 points when they must do both from the raw trace. Their own fact answers, appended, recover 15–28 points; composing the action in code reaches 0.96–1.00.

Two mechanisms: failing to consolidate scattered evidence (precedence errors) and capture by true but irrelevant cues (distractor errors).

From reading to composing, per system

Decision accuracy on 1,200 generated test states under each input; the line spans raw trace → facts only (the composition gap). Hover a row.

What the raw-trace errors are

On real states, the missing piece is the next sub-goal.

Appending correct answers to the benchmark's own probes adds at most 7 points for all but two systems. On ALFWorld, probes designed for the decision show JEV reads the state almost perfectly; stating the next sub-goal adds 16 points, any block of facts at most 5. Extra information is not neutral: an incomplete summary can pull a model away from a trace it otherwise reads well.

The bottleneck moves from reading to planning — a job for a planner, training, or deliberation.

Change in decision accuracy when information is added

Points relative to the decision asked alone; solid cells: 95% bootstrap interval excludes zero. Hover for intervals.

Errors after misreading are rarer but costlier.

Replaying every wrong decision in the simulators with exact lookahead: most cost at most one extra step. Where the two kinds differ, errors after a misread fact are the expensive ones — in TextWorld, 20% are unrecoverable against 6% after correct reading.

Between 5% and 42% of a system's "wrong" simulator decisions are in fact also optimal.

Cost of wrong decisions by exact lookahead

Calibration differs more than accuracy.

System One models return a distribution, so confidence could decide when to escalate. Expected calibration error ranges from 2.6% to 26.6% among systems above 0.5 accuracy; Gemma's likelihood readout is almost always near-certain, while several fine-tuned models are strongly underconfident.

Whether confidence can route uncertain states depends on the readout, not on accuracy.

Reliability

Accuracy against mean confidence in 15 equal-width bins (marker area: share of decisions). Below the diagonal: overconfident. GPT-6.1's verbalized probabilities are not comparable and are omitted.

Method

Ask the same state twice.

A System One model exposes only a choice. So we pair every next-action question with factual probes about the same state, each answered alone: no question or answer is ever added to another's input. Step through a real ALFWorld state below.

Four outcomes, carefully read

Crossing "all measured facts correct" (U) with "action correct" (D) gives four outcomes. Each licenses less than it seems.

Outcome	Directly observed	Not licensed
U=1 · D=1	Measured facts and action both correct	That all relevant facts were understood
U=1 · D=0	All measured facts correct; action wrong	That the failure lies purely in deciding
U=0 · D=1	Some probe wrong; action correct	That the model used a shortcut, or that the wrong fact mattered
U=0 · D=0	Some probe wrong; action wrong	That the probe error caused the wrong action

When can probes support the diagnosis?

A facts-right, action-wrong outcome points at the decision only if the probes capture what the decision needs. That can be checked without any model: group the states that share the same correct probe answers; the probes are sufficient on the group if some action is correct in all of them.

⋂_{s ∈ S} G(s) ≠ ∅

If two states share every correct probe answer but have disjoint gold sets, answering every probe correctly cannot determine the action.

The studies

The paired measurement runs on 19,663 test states. Three follow-up studies then add information to the decision and measure the change on the same states.

B · 1,200 generated states

Composition ladder

Four rule facts provably determine the action. Ask the decision from the raw trace, with the correct facts, with irrelevant facts, and from the facts alone.

Are the facts that are read also used?
Trace rewrites isolate the mechanism

See finding 03 →

C · 2,783 real states

Probe-then-decide

Ask the decision alone, after the benchmark's probe questions, after the model's own answers, and after the correct answers.

Does knowing the benchmark facts fix it?
Own answers: repairs vs. breaks

See finding 04 →

D · 644 ALFWorld states

Decision-oriented probes

About 26 narrow probes per state, derived from the trace for the decision: location, inventory, sightings, progress, and the next sub-goal.

Facts, state digest or sub-goal?
Correct values vs. the model's own

See finding 04 →

Where the paired states come from

Simulators with exact ground truth, customer-service policies, tool use, web navigation and coding traces. Every admissible action is in context; gold labels use only what the agent has seen.

Paired test states per domain (19,663 in total; 65,228 benchmark records including text-understanding primitives and controls).

Explore

Go looking.

Rankings shift by domain, cost and latency span three orders of magnitude, and removing the history tells you which domains need it. Pick a lens.

Systems × domains

Metric

Click a domain to sort. Hatched: fewer than ten errors (share not shown) or not applicable.

Accuracy against cost and speed

Log scale. The step line is the efficient frontier of the fast systems. Hosted costs are measured billing; self-hosted costs assume $2.50 per H100-hour on the latency panel (an upper bound for batched serving).

Remove the history

Decision accuracy with the full trace (ring) and with the executed history removed (dot). Where earlier observations matter, accuracy collapses: the models do read the trace.

Longer traces

ALFWorld decision accuracy by trace length: solid, the needed object was already seen; dashed, the agent must still explore.

Paper