Reservoir Transformers
This work addresses the problem of improving transformer efficiency and performance for researchers and practitioners working on natural language processing tasks, offering an incremental improvement.
This paper demonstrates that transformers can achieve impressive performance even when some layers are randomly initialized and never updated. By interspersing non-linear 'reservoir' layers with regular transformer layers, the authors show improvements in wall-clock compute time until convergence and overall performance on machine translation and language modeling tasks.
We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. Inspired by old and well-established ideas in machine learning, we explore a variety of non-linear "reservoir" layers interspersed with regular transformer layers, and show improvements in wall-clock compute time until convergence, as well as overall performance, on various machine translation and (masked) language modelling tasks.