Decisions, Scores, and Predictive Distributions
What should a language model report when another system
needs to make a decision?
Schematic report roles. Text generation may still occur inside the producer.
Language models increasingly supply the predictions on which other systems act. The important question is what a report means—and whether it supports the decision that follows.
We connect classification, reward and quality scoring, numerical prediction, and probabilistic forecasting through three questions.
A ranking score, an outcome probability, and a teacher’s quantile make different claims.
Preserving order, enforcing a constraint, and improving task performance are different achievements.
A compact report may require repeated generation—or reuse work done earlier.
Follow what reaches the consumer, then how it is produced. These roles can overlap: one system may expose quantiles, a median, and a thresholded decision.
A selected class, candidate, route, or action under a specified rule.
Select a model before an answer is generated.
Values used to compare, rank, or threshold candidates against a criterion.
Rank candidate responses by reward.
A point or numerical property defined by a target, units, and loss.
Predict a median price for absolute error.
A predictive law, samples, or specified uncertainty summaries for a named target.
Retain price quantiles for a tail-sensitive decision.
A cross-cutting boundary: combine local reports into spans, fields, or constrained assignments.
Across the four rolesFinite quantiles do not specify a full density. A teacher’s generation process and observed outcomes are different prediction targets.
Taxonomy in the paper
Concrete comparisons reveal what a report’s format alone cannot tell us.
On held-out values in a number look-up task, xVal achieves lower numerical error (MSE 0.0035 vs. 0.09), while FP15 more often reproduces the value to three significant figures (34.5% vs. 1.5%). The same evaluated predictions support different rankings.
A fitted probe moves teacher sampling offline. RELISH reuses one backbone encoding within a request. CAMEL reflects selectively. Other scalar interfaces aggregate repeated online generations.
Retain the information needed for today’s decision—and for the changes you expect to make later.
Scroll table to compare
| The consumer needs to… | Useful report | What must be checked |
|---|---|---|
| Select, route, or abstain | A final decision, or a score with a validated rule. | Fix candidates, costs, and thresholds. A margin is not automatically a correctness probability. |
| Rank or combine candidates | Comparable scores; retain component values if weights may change. | Ordering alone does not validate probability claims or weighted sums. Preserve rubric identities and scales. |
| Predict a number | A loss-matched point: a mean, median, or specified quantile. | Match the target and units. Squared error calls for a mean; absolute error admits a median. |
| Answer uncertainty or changing-loss queries | Required quantiles, a predictive law, or adequate retained samples. | Name the prediction target. Teacher uncertainty differs from outcome uncertainty; joint events need dependence information. |
| Return a valid structured object | A completed object, or local reports plus constraints. | Check legality and semantic quality separately. Include solver, parser, and assembly costs. |
A guide to information requirements, not a ranking of estimators. Richer reports need not be more accurate or cheaper to estimate.
Full selection guideWhen losses, constraints, or teacher processes change,
what must be revalidated?
The full taxonomy, configuration-level comparisons, statistical foundations, and evidence-linked selection guide.
@unpublished{li2026beyond,
title = {Beyond Natural-Language Interfaces:
Decisions, Scores, and Predictive Distributions},
author = {Li, Yubo and Krishnan, Ramayya and
Padman, Rema},
year = {2026},
note = {Survey manuscript},
url = {https://yubol-bobo.github.io/beyond-nl-survey/}
}This research was supported in part by the National Institute of Standards and Technology under Federal Award ID 60NANB24D231 and by Carnegie Mellon University’s AI Measurement Science and Engineering Center (AIMSEC). We also gratefully acknowledge support from OpenAI’s Researcher Access Program, which provided API credits used in this research.