Reporting ≠ capability
An incentive for truthful confidence need not favor the policy with the highest average accuracy.
Objectives · Section 3A RESEARCH SURVEY·CARNEGIE MELLON UNIVERSITY
Calibration-Aware
Reinforcement Learningfor Large Language Models
A Survey of Objectives, Optimization, and Decision-Making
Carnegie Mellon University
01Probability targetsWhat does the number mean?
02ObjectivesWhat does the reward elicit?
03OptimizationWhat does the update preserve?
04Decision-makingWhen is the report useful?
01 / OVERVIEW
Confidence is a number.
Its meaning is a contract.
Reinforcement learning can change both what a language model says and the confidence it expresses. This survey follows probability claims from their targets and training objectives through implemented updates to downstream decisions.
We bring together calibration-aware RL and its supervised and post-hoc comparators, asking when learned reports retain the statistical meaning that decisions require.
14learning families in Table 2
52source records in Appendix B
106references across the survey
Score ≠ update
Normalization, token masks, finite rollouts, and direct reward gradients shape the learning rule actually used.
Optimization · Section 4Ranking ≠ decision value
Good ordering can support selection at fixed coverage while failing at a transferred probability threshold.
See the decision example
The same number can make different claims. The event and information set define the probability target. Each step from feedback to decision introduces conditions to check.View full size
02 / THE METHOD LANDSCAPE
Compare what is learned.
The scored object, learning intervention, and decision semantics matter as much as the optimizer’s name.
Eight representative families from the survey. This is a conceptual comparison, not a benchmark leaderboard.
Scroll horizontally to compare all four columns
| Family / examples | What is scored | Learning intervention | Decision implication |
|---|---|---|---|
| External eventsForecasting | Event probabilities, scored against resolved outcomes. | Update the reporting policy while the event definition stays fixed. | Assess reliability and proper scores on held-out events. |
| Fixed-answer confidenceRewarding Doubt | Supplied-answer correctness, using a clipped log-score reward. | Generate confidence with the answer held as input. | Isolates reporting from answer choice during training. |
| Joint answer and confidenceRLCR · VPD | Selected-answer confidence; correctness and report-error rewards. | Update answers and confidence together. | Reporting and capability interact. Vector output may receive scalar supervision. |
| Relative orderingRLSR | Correctness order through rank-weighted selective-risk rewards. | Change confidence ordering. | Supports fixed-coverage selection; the probability scale remains unidentified. |
| Selective actionsTruthRL | Answer/abstain behavior under ternary utilities. | Update the action policy under the chosen reward. | Evaluate utility and coverage. Action likelihood is not correctness confidence. |
| Predictive distributionsDAR | A finite set of generated numerical outcomes, scored with empirical CRPS. | Leave-one-out credit and group-standardized RL. | Distinguish the scored finite set from its generating distribution. |
| Supervised confidence†ConfTuner | Fixed-answer correctness through semantic report error. | Directly fit confidence tokens. | Tests whether rollout RL adds value beyond supervised reporting. |
| Post-hoc calibration†Temperature scaling | Fixed-logit class probabilities under held-out likelihood loss. | Fit a scalar temperature with the generator fixed. | Rescales probabilities and preserves class argmax; needs representative calibration data. |
† Comparator without online rollout-reward training. Examples link to their primary papers.
All 52 source records · Appendix B03 / A CLOSER LOOK
Same ranking.
A different decision.
A strictly increasing transformation preserves which answers rank highest. It can still change which answers cross a fixed probability threshold—and reverse their utility.
A threshold carries a probability assumption. At threshold 0.8 with payoffs +1 / −4 / 0, truthful reports yield utility +0.125 per input; sharpened scores yield −0.125. Selecting a new threshold of 0.95 restores the original accepted set.View full size
An exact analytic construction, not an LLM experiment. Four equally weighted contexts have correctness rates (0.1, 0.4, 0.6, 0.9). Both reporters have AUROC 0.825. The example isolates the difference between ranking, calibration, and decision utility; it does not establish an empirical advantage for a training method.
04 / PAPER & RESOURCES
The full argument.
With the details intact.
Read the synthesis, mathematical analyses, method inventory, and implementation qualifications in one manuscript.
59 pages · Manuscript dated 21 September 2026
Cite this survey
@misc{li2026calibrationaware,
title = {Calibration-Aware Reinforcement
Learning for Large Language Models:
A Survey of Objectives, Optimization,
and Decision-Making},
author = {Li, Yubo and Miao, Yidi and
Krishnan, Ramayya and Padman, Rema},
year = {2026},
url = {https://yubol-bobo.github.io/rlcd-survey/}
}Acknowledgments
This research was supported in part by the National Institute of Standards and Technology under Federal Award ID 60NANB24D231 and by Carnegie Mellon University’s AI Measurement Science and Engineering Center (AIMSEC). We also gratefully acknowledge support from OpenAI’s Researcher Access Program, which provided API credits used in this research.