Statistical Characteristics of Deep Representations: An Empirical Investigation
This work addresses the problem of understanding and optimizing deep learning representations for researchers, but it is incremental as it builds on existing regularization methods without establishing a consistent relationship.
The study investigated how eight representation regularization methods, including two new rank regularizers, affect statistical characteristics like correlation, sparsity, and rank in deep learning, finding that these can be manipulated during training to improve baseline performance through fine-tuning, but no direct link between performance and statistical characteristics was observed.
In this study, the effects of eight representation regularization methods are investigated, including two newly developed rank regularizers (RR). The investigation shows that the statistical characteristics of representations such as correlation, sparsity, and rank can be manipulated as intended, during training. Furthermore, it is possible to improve the baseline performance simply by trying all the representation regularizers and fine-tuning the strength of their effects. In contrast to performance improvement, no consistent relationship between performance and statistical characteristics was observable. The results indicate that manipulation of statistical characteristics can be helpful for improving performance, but only indirectly through its influence on learning dynamics or its tuning effects.