DeepMoM: Robust Deep Learning With Median-of-Means
It addresses a common but understudied issue of data quality in deep learning, offering a robust alternative to standard loss functions for practitioners dealing with noisy datasets.
The paper tackles the problem of random data corruption in deep learning, which arises from measurement errors and unreliable sources, by introducing a robust training approach based on median-of-means principles, demonstrating strong practical performance.
Data used in deep learning is notoriously problematic. For example, data are usually combined from diverse sources, rarely cleaned and vetted thoroughly, and sometimes corrupted on purpose. Intentional corruption that targets the weak spots of algorithms has been studied extensively under the label of "adversarial attacks." In contrast, the arguably much more common case of corruption that reflects the limited quality of data has been studied much less. Such "random" corruptions are due to measurement errors, unreliable sources, convenience sampling, and so forth. These kinds of corruption are common in deep learning, because data are rarely collected according to strict protocols -- in strong contrast to the formalized data collection in some parts of classical statistics. This paper concerns such corruption. We introduce an approach motivated by very recent insights into median-of-means and Le Cam's principle, we show that the approach can be readily implemented, and we demonstrate that it performs very well in practice. In conclusion, we believe that our approach is a very promising alternative to standard parameter training based on least-squares and cross-entropy loss.