BlackGoose Rimer: Harnessing RWKV-7 as a Simple yet Superior Replacement for Transformers in Large-Scale Time Series Modeling
This addresses the problem of computational efficiency and performance in large-scale time series modeling for researchers and practitioners, though it appears incremental as it builds on existing architectures like RWKV-7 and Timer.
The paper tackled the challenge of scaling time series models to large datasets by proposing BlackGoose Rimer, which integrates RWKV-7 into a transformer-based model, achieving performance improvements of 1.13 to 43.3x and a 4.5x reduction in training time with fewer parameters.
Time series models face significant challenges in scaling to handle large and complex datasets, akin to the scaling achieved by large language models (LLMs). The unique characteristics of time series data and the computational demands of model scaling necessitate innovative approaches. While researchers have explored various architectures such as Transformers, LSTMs, and GRUs to address these challenges, we propose a novel solution using RWKV-7, which incorporates meta-learning into its state update mechanism. By integrating RWKV-7's time mix and channel mix components into the transformer-based time series model Timer, we achieve a substantial performance improvement of approximately 1.13 to 43.3x and a 4.5x reduction in training time with 1/23 parameters, all while utilizing fewer parameters. Our code and model weights are publicly available for further research and development at https://github.com/Alic-Li/BlackGoose_Rimer.