Value Residual Learning
This work addresses a critical challenge in deep learning for AI practitioners by improving efficiency in Transformer models, though it is incremental as it builds on existing residual methods.
The paper tackles the problem of information propagation in deep Transformer networks by introducing ResFormer, which uses value residual connections to preserve token-level information, achieving equivalent validation loss with 16.11% fewer parameters and 20.3% less training data compared to standard Transformers.
While Transformer models have achieved remarkable success in various domains, the effectiveness of information propagation through deep networks remains a critical challenge. Standard hidden state residuals often fail to adequately preserve initial token-level information in deeper layers. This paper introduces ResFormer, a novel architecture that enhances information flow by incorporating value residual connections in addition to hidden state residuals. And a variant is SVFormer, where all layers share the first layer's value embedding. Comprehensive empirical evidence demonstrates ResFormer achieves equivalent validation loss with 16.11\% fewer model parameters and 20.3\% less training data compared to Transformer, while maintaining similar memory usage and computational cost. Besides, SVFormer reduces KV cache size by nearly half with only a small performance penalty and can be integrated with other KV-efficient methods, yielding further reductions in KV cache, with performance influenced by sequence length and cumulative learning rate.