TransTS: an adaptive post-hoc method for probability calibration under label noise

Probability calibration aims to align the classifier’s confidence with the true likelihood of correctness (i.e., empirical frequencies). Existing calibration methods perform well when labels are clean. However, when labels are corrupted, calibration deterioration is almost inevitable. In this paper, we study probability calibration under label noise. Specifically, we first observe that existing calibration methods struggle to maintain calibration quality in the presence of label noise. Second, we reveal that label noise leads to calibration deterioration due to the failure of the calibration equation, which further results in over-calibration for temperature scaling (TS). To address this issue, we propose adaptive transitional temperature scaling (TransTS), which adaptively scales the logits according to the noise level. TransTS constructs a consistent calibrator that can converge to its counterpart trained on clean data, and we provide theoretical justifications for this property. As a general post-hoc method, TransTS can be easily integrated with any pre-trained model. Results on a variety of experimental cases show that TransTS outperforms five built-in methods and eleven post-hoc methods, as well as several widely used learning with noisy label (LNL) methods.