Exploring Low Rank Training of Deep Neural Networks
This work addresses efficiency in training for the machine learning community, but it appears incremental as it builds on prior low-rank training methods.
The paper tackles the problem of training deep neural networks more efficiently by using low-rank factorized layers, and through extensive ablations on models like GPT2, it provides evidence that falsifies common beliefs in the field, hinting at new research opportunities.
Training deep neural networks in low rank, i.e. with factorised layers, is of particular interest to the community: it offers efficiency over unfactorised training in terms of both memory consumption and training time. Prior work has focused on low rank approximations of pre-trained networks and training in low rank space with additional objectives, offering various ad hoc explanations for chosen practice. We analyse techniques that work well in practice, and through extensive ablations on models such as GPT2 we provide evidence falsifying common beliefs in the field, hinting in the process at exciting research opportunities that still need answering.