Dropout can Simulate Exponential Number of Models for Sample Selection Techniques
This work addresses noisy label training for machine learning practitioners by offering a more convenient and effective method, though it is incremental as it builds on existing sample selection techniques.
The paper tackles the problem of training with noisy labels by leveraging Dropout to simulate an exponential number of models for sample selection, leading to improved results compared to existing two-model approaches.
Following Coteaching, generally in the literature, two models are used in sample selection based approaches for training with noisy labels. Meanwhile, it is also well known that Dropout when present in a network trains an ensemble of sub-networks. We show how to leverage this property of Dropout to train an exponential number of shared models, by training a single model with Dropout. We show how we can modify existing two model-based sample selection methodologies to use an exponential number of shared models. Not only is it more convenient to use a single model with Dropout, but this approach also combines the natural benefits of Dropout with that of training an exponential number of models, leading to improved results.