Intra-Layer Recurrence in Transformers for Language Modeling
This work addresses parameter efficiency in transformer models for natural language processing, but it appears incremental as it builds on existing recurrent transformer methods.
The paper tackles the problem of high parameter counts in deep transformer models by proposing Intra-Layer Recurrence (ILR), a method that applies recurrence selectively to individual layers, and finds that allocating more iterations to earlier layers yields optimal results.
Transformer models have established new benchmarks in natural language processing; however, their increasing depth results in substantial growth in parameter counts. While existing recurrent transformer methods address this issue by reprocessing layers multiple times, they often apply recurrence indiscriminately across entire blocks of layers. In this work, we investigate Intra-Layer Recurrence (ILR), a more targeted approach that applies recurrence selectively to individual layers within a single forward pass. Our experiments show that allocating more iterations to earlier layers yields optimal results. These findings suggest that ILR offers a promising direction for optimizing recurrent structures in transformer architectures.