Avey-B
This work addresses the need for compact, efficient encoders in industrial NLP under compute and memory constraints, representing an incremental improvement by adapting an existing model with architectural tweaks.
The paper tackled the problem of creating efficient bidirectional encoders for NLP by reformulating the autoregressive Avey model into an encoder-only architecture with innovations like decoupled parameters and neural compression, achieving consistent outperformance over four Transformer-based encoders on token-classification and information-retrieval benchmarks with better scaling to long contexts.
Compact pretrained bidirectional encoders remain the backbone of industrial NLP under tight compute and memory budgets. Their effectiveness stems from self-attention's ability to deliver high-quality bidirectional contextualization with sequence-level parallelism, as popularized by BERT-style architectures. Recently, Avey was introduced as an autoregressive, attention-free alternative that naturally admits an encoder-only adaptation. In this paper, we reformulate Avey for the encoder-only paradigm and propose several innovations to its architecture, including decoupled static and dynamic parameterizations, stability-oriented normalization, and neural compression. Results show that this reformulated architecture compares favorably to four widely used Transformer-based encoders, consistently outperforming them on standard token-classification and information-retrieval benchmarks while scaling more efficiently to long contexts.