The Calibration Channel Determines the Bayes-Error Proxy: An Exact Law for Temperature-Induced Distortion

Imagine you're building a machine learning model that predicts whether an email is spam or not. You want to ensure that the model's confidence in its predictions is accurate, as this affects how you handle uncertain cases. Currently, many practitioners use soft labels — probabilities that indicate how likely an email is spam — to gauge performance. However, if these probabilities aren't perfectly calibrated, they can lead to misleading conclusions about the model's reliability, which is what's called the Bayes-error estimator. This can result in overconfidence or underconfidence in the model's predictions, ultimately affecting decision-making. To address this, researchers have explored calibration techniques, like temperature scaling, which adjusts the predicted probabilities to improve their accuracy. This paper dives deeper into how temperature scaling can distort the error estimates derived from these probabilities. By establishing a clear relationship between the temperature parameter and the classifier's margin distribution, the authors reveal that you can manipulate the reported error estimates significantly, even if the actual classification performance remains unchanged. This insight emphasizes the importance of understanding the calibration process and suggests that relying solely on proxy values without considering their derivation can be misleading. Compared to previous work, this research provides a more nuanced view of calibration, showing that the temperature that minimizes expected calibration error does not align with stable proxy values. For builders, this means that when designing systems that rely on probabilistic outputs, it's crucial to consider how calibration methods can impact the interpretation of those outputs.

The paper provides a new understanding of how temperature scaling affects error estimation in binary classification.

The empirical results across multiple datasets support the theoretical claims made in the paper.