Memory-efficient Stochastic methods for Memory-based Transformers
This work addresses memory constraints for researchers and practitioners using memory-based transformers in long-range context applications, representing an incremental improvement with specific efficiency gains.
The paper tackles the memory inefficiency problem in training memory-based transformers by proposing a two-phase training mechanism and a novel regularization technique, achieving better performance than Transformer-XL on character-level language modeling with similar parameters and on word-level language modeling with 20% fewer parameters, while also reducing score standard deviation by around 30% on GLUE tasks with BERT.
Training Memory-based transformers can require a large amount of memory and can be quite inefficient. We propose a novel two-phase training mechanism and a novel regularization technique to improve the training efficiency of memory-based transformers, which are often used for long-range context problems. For our experiments, we consider transformer-XL as our baseline model which is one of memorybased transformer models. We show that our resultant model, Skip Cross-head TransformerXL, outperforms the baseline on character level language modeling task with similar parameters and outperforms the baseline on word level language modelling task with almost 20% fewer parameters. Our proposed methods do not require any additional memory. We also demonstrate the effectiveness of our regularization mechanism on BERT which shows similar performance with reduction in standard deviation of scores of around 30% on multiple GLUE tasks.