The Shape of Learning Curves: a Review
This is an incremental review paper that synthesizes existing literature on learning curves for researchers and practitioners who use them for model selection, data efficiency predictions, and computational optimization.
This review paper examines the diverse shapes of learning curves in machine learning, finding that while many follow well-behaved patterns like power laws or exponentials, some display ill-behaved patterns where performance worsens with more data, and concludes that no universal model exists.
Learning curves provide insight into the dependence of a learner's generalization performance on the training set size. This important tool can be used for model selection, to predict the effect of more training data, and to reduce the computational complexity of model training and hyperparameter tuning. This review recounts the origins of the term, provides a formal definition of the learning curve, and briefly covers basics such as its estimation. Our main contribution is a comprehensive overview of the literature regarding the shape of learning curves. We discuss empirical and theoretical evidence that supports well-behaved curves that often have the shape of a power law or an exponential. We consider the learning curves of Gaussian processes, the complex shapes they can display, and the factors influencing them. We draw specific attention to examples of learning curves that are ill-behaved, showing worse learning performance with more training data. To wrap up, we point out various open problems that warrant deeper empirical and theoretical investigation. All in all, our review underscores that learning curves are surprisingly diverse and no universal model can be identified.