Dynamic Layer Tying for Parameter-Efficient Transformers
This addresses parameter efficiency for transformer models, but it is incremental as it builds on existing weight-sharing techniques.
The paper tackles the problem of reducing trainable parameters in deep transformer networks by using Reinforcement Learning to dynamically tie layers during training, resulting in modest perplexity improvements and up to an order of magnitude reduction in memory consumption.
In the pursuit of reducing the number of trainable parameters in deep transformer networks, we employ Reinforcement Learning to dynamically select layers during training and tie them together. Every few iterations, the RL agent is asked whether to train each layer $i$ independently or to copy the weights of a previous layer $j<i$. This facilitates weight sharing, reduces the number of trainable parameters, and also serves as an effective regularization technique. Experimental evaluations validate that our model modestly outperforms the baseline transformer model with regard to perplexity and drastically reduces the number of trainable parameters. In particular, the memory consumption during training is up to one order of magnitude less than the conventional training method.