Model Complexity of Deep Learning: A Survey
It addresses the fundamental problem of model complexity for researchers and practitioners in deep learning, but is incremental as it synthesizes existing work without new methods or results.
The paper provides a systematic survey of model complexity in deep learning, categorizing it into expressive capacity and effective model complexity, and reviews studies across factors like model framework and size, with applications in generalization and model selection.
Model complexity is a fundamental problem in deep learning. In this paper we conduct a systematic overview of the latest studies on model complexity in deep learning. Model complexity of deep learning can be categorized into expressive capacity and effective model complexity. We review the existing studies on those two categories along four important factors, including model framework, model size, optimization process and data complexity. We also discuss the applications of deep learning model complexity including understanding model generalization, model optimization, and model selection and design. We conclude by proposing several interesting future directions.