When are likely answers right? On Sequence Probability and Correctness in LLMs
Johannes Zenn, Jonas Geiping
Read on arXiv →
Key claim
Higher sequence probability doesn't always lead to better accuracy.
In plain English
Imagine you're building a chatbot that needs to generate accurate responses based on user prompts. You might think that if you can make the model more confident in its answers, it will produce better results. This is where decoding methods come in; they adjust how the model generates text to favor more likely outputs. However, there's a catch: just because a response has a high probability doesn't mean it's correct. This is a common issue when trying to improve the accuracy of generated text, known as the misalignment of sequence probability and correctness. Builders often face situations where tweaking the model's settings or the way it generates text doesn't lead to better answers, which is frustrating and counterproductive. This is what's called the failure of decoding decisions to improve accuracy. What this paper explores is the relationship between the probability of a generated sequence and its correctness across different methods and settings. They found that while higher sequence probability can indicate correctness for specific pairs of prompts and answers, simply increasing that probability through changes in methods or hyperparameters doesn't guarantee better accuracy. This means that builders need to be cautious when relying on probability scores to judge the quality of generated responses. The practical takeaway is that understanding when decoding methods can improve correctness is essential for developing more reliable language models.
The paper provides a meaningful extension by quantifying the relationship between sequence probability and correctness across various decoding methods.
The findings are supported by empirical analysis across multiple benchmarks and conditions, demonstrating solid evaluation practices.
Deep reliability assessment
The methodology supports a nuanced empirical claim: sequence log-probability can correlate with correctness across prompt-answer pairs within a dataset, but that signal often fails when used for actual decoding choices across methods, hyperparameters, or repeated samples for the same prompt. What would be overclaimed is any universal rule like “higher probability means better answer” or “probability-only reranking is a reliable verifier” outside the tested model families, benchmarks, and decoding regimes.
Reproducibility
No code repository or release link is mentioned in the provided paper text. The experiments use identifiable model families, including Qwen2.5, Qwen3, and Olmo3, and six benchmarks, with MATH500 explicitly shown in Figure 1, but full reproduction would require the paper's complete experimental details and decoding hyperparameters.
Key figure
Figure 1 visualizes the relationship between log-probability and correctness at three granularities: across decoding methods and hyperparameters, across prompt-answer pairs within a dataset, and across repeated responses to a single prompt.