When are likely answers right? On Sequence Probability and Correctness in LLMs
Johannes Zenn, Jonas Geiping
Read on arXiv →
Key claim
Higher sequence probability doesn't always lead to better accuracy.
This paper reveals that while higher sequence probability often predicts correctness for specific prompt-answer pairs, it doesn't reliably improve accuracy when changing decoding methods or hyperparameters. This insight is crucial for builders looking to enhance model performance through decoding strategies.
In plain English
Imagine you're building a chatbot that needs to generate accurate responses based on user prompts. You might think that if you can make the model more confident in its answers, it will produce better results. This is where decoding methods come in; they adjust how the model generates text to favor more likely outputs. However, there's a catch: just because a response has a high probability doesn't mean it's correct. This is a common issue when trying to improve the accuracy of generated text, known as the misalignment of sequence probability and correctness. Builders often face situations where tweaking the model's settings or the way it generates text doesn't lead to better answers, which is frustrating and counterproductive. This is what's called the failure of decoding decisions to improve accuracy. What this paper explores is the relationship between the probability of a generated sequence and its correctness across different methods and settings. They found that while higher sequence probability can indicate correctness for specific pairs of prompts and answers, simply increasing that probability through changes in methods or hyperparameters doesn't guarantee better accuracy. This means that builders need to be cautious when relying on probability scores to judge the quality of generated responses. The practical takeaway is that understanding when decoding methods can improve correctness is essential for developing more reliable language models.
The paper provides a meaningful extension by quantifying the relationship between sequence probability and correctness across various decoding methods.
The findings are supported by empirical analysis across multiple benchmarks and conditions, demonstrating solid evaluation practices.
Deep reliability assessment
The methodology supports a nuanced empirical claim: sequence log-probability can correlate with correctness across prompt-answer pairs within a dataset, but that signal often fails when used for actual decoding choices across methods, hyperparameters, or repeated samples for the same prompt. What would be overclaimed is any universal rule like “higher probability means better answer” or “probability-only reranking is a reliable verifier” outside the tested model families, benchmarks, and decoding regimes.
Reproducibility
No code repository or release link is mentioned in the provided paper text. The experiments use identifiable model families, including Qwen2.5, Qwen3, and Olmo3, and six benchmarks, with MATH500 explicitly shown in Figure 1, but full reproduction would require the paper's complete experimental details and decoding hyperparameters.
Discussion questions
- 1.The paper finds that log-probability predicts correctness across different prompt-answer pairs in a dataset, but not reliably among multiple answers to the same prompt. If you were building self-consistency or reranking into a product, does that make probability scores basically unusable, or still useful as one weak feature?
- 2.They frame many decoding methods as shifting mass toward higher-probability outputs, then show that increasing probability via hyperparameters does not reliably improve accuracy. Has anyone here seen lower temperature, beam search, or best-of-N make outputs worse in production despite looking more confident or polished?
- 3.The within-dataset correlation seems to depend more on model family than decoding method. Should we interpret that as a property of the model's calibration, the benchmark distribution, or the answer format? What experiment would convince you one way or the other?
- 4.The paper tests several local, mode-seeking, and power-distribution decoding methods, but the practical baseline for many teams is not pure probability reranking; it is an LLM judge, a verifier, unit tests, or domain-specific scoring. Is probability-only decoding a fair target to study, or is it too weak compared with what builders actually deploy?
- 5.If higher-probability answers are more likely to be correct across a dataset but not within a prompt's candidate answers, what does that imply for verifier-free self-distillation? Would you still train on the highest-probability samples, or would that risk amplifying fluent-but-wrong modes?
Key figure
Figure 1 visualizes the relationship between log-probability and correctness at three granularities: across decoding methods and hyperparameters, across prompt-answer pairs within a dataset, and across repeated responses to a single prompt.