Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator

Beier Luo, Shuoyuan Wang, Sharon Li, Hongxin Wei

Department of Statistics and Data Science, Southern University of Science and Technology;
Department of Computer Sciences, University of Wisconsin-Madison

Problem

Post-trained language models (PoLMs) are essential for aligning large language models with human preferences, but they often suffer from over-confidence—assigning high confidence to both correct and incorrect outputs. This undermines reliability, especially in high-stakes domains like medicine and security. While pre-trained language models (PLMs) tend to be well-calibrated, post-training disrupts this property. Traditional calibration methods like temperature scaling require labeled data, which is scarce and expensive to obtain for many real-world tasks. This creates a critical gap: how to calibrate PoLMs effectively without access to labeled data?

Reliability diagram evaluation for pre-trained vs. post-trained models across four modern LLM architectures on MMLU [Hendrycks et al., 2021a]. The post-trained models are trained by multiple post-training techniques, including SFT, RLHF, and DPO. More reliability diagrams of various post-training methods are provided in Appendix A.

Method

We propose Disagreement-Aware Confidence Alignment (DACA), an unsupervised post-hoc calibration method that leverages the well-calibrated nature of pre-trained models (PLMs) to calibrate post-trained models (PoLMs) using only unlabeled data. Instead of aligning confidence across all examples, DACA selectively uses agreement examples—those where the PLM and PoLM produce the same prediction. This avoids the under-confidence issue caused by prediction disagreement. Formally, given logits from the PoLM $ g(x) $ and softmax outputs from the PLM $ p(x) $, DACA optimizes the temperature $ \tau $ by minimizing KL divergence only on agreement samples:

\[ \mathcal{L}_{\text{DACA}}(\tau) = \mathbb{E}_{x \sim \mathcal{D}} \left[ \mathbb{1}\{\arg\max_i f_i(x) = \arg\max_i g_i(x)\} \cdot D_{\text{KL}}\left( \sigma(f(x)) \parallel \sigma(g(x)/\tau) \right) \right] \]

This ensures that confidence alignment occurs only when the PLM’s confidence is a reliable proxy for correctness.

Under-confidence issue of naive confidence alignment. ( a ): Reliability diagram for Yi-1.59B-Chat on the computer security and college chemistry subjects of MMLU [Hendrycks et al., 2021a]. Results of more models are presented in Appendix D. ( b ): Temperature values of Yi-1.5-9B-Chat under different training epochs when trained separately on the disagreement and agreement sets and the whole dataset. The training process is performed on the computer security and the college chemistry subject of MMLU.

Theory

We show theoretically that naive confidence alignment—aligning PLM and PoLM confidence on all unlabeled data—can lead to under-confidence. When the two models disagree, the PLM’s confidence reflects its own accuracy, not the PoLM’s. Since post-training often improves accuracy, the PoLM may be correct even when the PLM is not, causing the PLM’s confidence to underestimate the PoLM’s true correctness likelihood. This leads to an inflated temperature $ \tau $ during optimization, resulting in under-confidence. Proposition 3.3 shows that on disagreement examples where $ \sigma_c(f(x)) < 1/k $, the gradient of the KL divergence w.r.t. $ \tau $ remains positive, pushing $ \tau $ to larger values. DACA eliminates this effect by excluding disagreement examples, enabling more accurate and stable temperature estimation.

Experiments

We evaluate DACA across diverse LLMs—including Llama-3, Gemma-3, Qwen2.5, Yi-1.5, GPT-4o, and DeepSeek-V3—on standard benchmarks like MMLU, MedMCQA, and MathQA. DACA consistently improves calibration, reducing ECE by up to 15.08% across models. On MMLU, it reduces the ECE of Gemma-3-12B-Instruct from 23.68% to 8.60%, outperforming supervised temperature scaling (9.75%). It also excels on MedMCQA, cutting GPT-4o’s ECE from 21.23% to 6.99% using a smaller pre-trained model. DACA outperforms baselines like CAPE and verbalization-based elicitation methods, and remains effective across model sizes and architectures. It is robust to different post-training techniques (SFT, RLHF, DPO), improving calibration across all variants of Llama-3.1-8B.

Average calibration performance across 57 MMLU subjects for several contemporary PoLMs. "Vanilla" refers to the uncalibrated model. † indicates calibration methods with access to labels. Best results are shown in bold, and the second-best results are presented in italics. Detailed results for a broader range of LLMs are available in the Appendix D.2.

ECE comparison between our methods and baselines on MedMCQA across varying contemporary LLM families and parameter sizes.

Applications

DACA extends beyond multiple-choice QA to open-ended tasks like TruthfulQA, where confidence is estimated via self-evaluation (P(True)). DACA significantly improves calibration, reducing ECE from 30.955% to 5.244% on Qwen2.5-32B-Instruct. It also enhances selective classification: by better aligning confidence with accuracy, DACA enables safer prediction filtering. As shown, accuracy on high-confidence predictions improves markedly under DACA, especially at higher thresholds, making it ideal for decision-critical applications where reliability trumps coverage.

ECE of DACA with different LLMs for the open-ended TruthfulQA benchmark. The lower ECE indicates better calibration performance. Detailed results with more models are provided in Appendix D.

Selective classification accuracy on MedMCQA across different models. Accuracy is reported on subsets of examples with confidence scores above thresholds ranging from 0.55 to 0.95.

@article{luo2024pretrained, title={Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator}, author={Luo, Beier and Wang, Shuoyuan and Li, Sharon and Wei, Hongxin}, journal={arXiv preprint arXiv:XXXX.XXXXX}, year={2024} }