Supernova: Achieving More with Less in Transformer Architectures
This work addresses the computational and data efficiency challenges in large language models, offering a more resource-efficient alternative to the prevailing scaling paradigm.
The paper tackles the problem of achieving high performance in transformer architectures with fewer parameters and less training data, resulting in a 650M-parameter model that achieves 90% of the performance of 1B-parameter models while using 35% fewer parameters and requiring only 100B training tokens.
We present Supernova, a 650M-parameter decoder-only transformer that demonstrates how careful architectural design and tokenization innovation can achieve the performance of larger models while maintaining computational efficiency. Our architecture combines Rotary Positional Embeddings (RoPE), Grouped Query Attention (GQA) with a 3:1 compression ratio, RMSNorm for computational efficiency, and SwiGLU activation functions. A critical innovation is our custom 128,000-vocabulary byte-level BPE tokenizer, which achieves state-of-the-art compression performance. Through detailed analysis, we show that Supernova achieves 90% of the performance of 1B-parameter models while using 35% fewer parameters and requiring only 100B training tokens--an order of magnitude less than competing models. Our findings challenge the prevailing scaling paradigm, demonstrating that architectural efficiency and tokenization quality can compensate for reduced parameter counts.