To go deep or wide in learning?
This addresses computational overheads in deep learning for AI practitioners by offering a more efficient alternative, though it appears incremental as it builds on kernel methods.
The paper tackles the trade-off between deep multi-layer models and problem-specific feature extraction by proposing wide learning, a single-layer approach using arc-cosine kernels with infinite width. It shows that this method outperforms both single-layer and deep finite-width architectures on some benchmark datasets.
To achieve acceptable performance for AI tasks, one can either use sophisticated feature extraction methods as the first layer in a two-layered supervised learning model, or learn the features directly using a deep (multi-layered) model. While the first approach is very problem-specific, the second approach has computational overheads in learning multiple layers and fine-tuning of the model. In this paper, we propose an approach called wide learning based on arc-cosine kernels, that learns a single layer of infinite width. We propose exact and inexact learning strategies for wide learning and show that wide learning with single layer outperforms single layer as well as deep architectures of finite width for some benchmark datasets.