Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
This addresses the challenge of parameter-efficient training for low-rank pre-tuning in deep learning, offering a novel method that is incremental in improving low-rank optimization techniques.
The paper tackles the problem of low-rank pre-training for large deep-learning models, where existing methods struggle to maintain low-rank structure and performance, by proposing Q3R, a quadratic reweighted rank regularizer that enables training with prescribed low target ranks, achieving comparable predictive performance to dense models with small computational overhead, as demonstrated by truncating 60% and 80% of parameters in a ViT-Tiny model with only ~1.3% and ~4% accuracy drops on CIFAR-10.
Parameter-efficient training, based on low-rank optimization, has become a highly successful tool for fine-tuning large deep-learning models. However, these methods fail at low-rank pre-training tasks where maintaining the low-rank structure and the objective remains a challenging task. We propose the Quadratic Reweighted Rank Regularizer dubbed Q3R, which leads to a novel low-rank inducing training strategy inspired by the iteratively reweighted least squares (IRLS) framework. Q3R is based on a quadratic regularizer term which majorizes a smoothed log determinant serving as rank surrogate objective. Unlike other low-rank training techniques, Q3R is able to train weight matrices with prescribed, low target ranks of models that achieve comparable predictive performance as dense models, with small computational overhead, while remaining fully compatible with existing architectures. For example, we demonstrated one experiment where we are able to truncate $60\%$ and $80\%$ of the parameters of a ViT-Tiny model with $~1.3\%$ and $~4\%$ accuracy drop in CIFAR-10 performance respectively. The efficacy of Q3R is confirmed on Transformers across both image and language tasks, including for low-rank fine-tuning.