Homeostasis and Sparsity in Transformer
This work addresses performance enhancement for transformer-based models in tasks like machine translation, though it appears incremental by combining existing ideas from neuroscience with standard architectures.
The paper tackles the problem of improving transformer performance by integrating homeostasis mechanisms like RFB-kWTA and 'Smart' Inhibition into the attention mechanism and output, achieving a BLEU score of 0.3062 on the Multi30K dataset, outperforming classical and dropout-based transformers.
The transformer architecture has become an integral part of the field of modern neural networks, playing a crucial role in a variety of tasks, such as text generation, machine translation, image and audio processing, among others. There is also an alternative approach to building intelligent systems, proposed by Jeff Hawkins and inspired by the processes occurring in the neocortex. In our article we want to combine some of these ideas and to propose the use of homeostasis mechanisms, such as RFB-kWTA and "Smart" Inhibition, in the attention mechanism of the transformer and at the output of the transformer block, as well as conducting an experiment involving the introduction of sparse distributed representations of the transformer at various points. RFB-kWTA utilizes statistics of layer activations across time to adjust the entire layer, enhancing the values of rare activations while reducing those of frequent ones. "Smart" Inhibition also uses activation statistics to sample sparsity masks, with rarer activation times are more likely to be activated. Our proposed mechanisms significantly outperform the classical transformer 0.2768 BLEU and a model that only makes use of dropout in the attention mechanism and output of the transformer block 0.3007 BLEU, achieving a score of 0.3062 on the Multi30K dataset.