Search Spaces for Neural Model Training
This work addresses the challenge of training efficiency for neural models, offering a method to accelerate both training and inference, though it appears incremental in its approach.
The paper tackles the problem of neural model training inefficiency by proposing that adding weights creates wider search spaces for optimization, and demonstrates that augmenting these spaces can train sparse models achieving competitive scores across multiple deep learning workloads.
While larger neural models are pushing the boundaries of what deep learning can do, often more weights are needed to train models rather than to run inference for tasks. This paper seeks to understand this behavior using search spaces -- adding weights creates extra degrees of freedom that form new paths for optimization (or wider search spaces) rendering neural model training more effective. We then show how we can augment search spaces to train sparse models attaining competitive scores across dozens of deep learning workloads. They are also are tolerant of structures targeting current hardware, opening avenues for training and inference acceleration. Our work encourages research to explore beyond massive neural models being used today.