Understanding and mitigating the label noise in pre-training on downstream tasks
Published in International Conference on Learning Representations (ICLR 2024), 2024
Pre-training on large-scale datasets has become the foundation of modern deep learning, but these datasets often contain significant label noise. This work explores how label noise in pre-training data affects model performance on downstream tasks. We provide theoretical and empirical analysis of the noise propagation mechanism and propose mitigation strategies that can improve the robustness of pre-trained models when transferred to clean downstream tasks.
Recommended citation: @inproceedings{chen2024understanding, title={Understanding and Mitigating the Label Noise in Pre-Training on Downstream Tasks}, author={Chen, Hao and Wang, Jindong and Shah, Ankit and Tao, Ran and Wei, Hongxin and Xie, Xing and Sugiyama, Masashi and Raj, Bhiksha}, booktitle={International Conference on Learning Representations (ICLR)}, year={2024} }
Download Paper