Spelling Bee Embeddings for Language Modeling
This addresses efficiency and performance issues in language modeling for AI researchers and practitioners, though it appears incremental as a simple modification to embeddings.
The paper tackles the problem of improving language model performance by modifying token embeddings to incorporate spelling information, resulting in models that achieve equivalent test loss with approximately 8% less compute and data across scaling studies from 40M to 800M parameters.
We introduce a simple modification to the embedding layer. The key change is to infuse token embeddings with information about their spelling. Models trained with these embeddings improve not only on spelling, but also across standard benchmarks. We conduct scaling studies for models with 40M to 800M parameters, which suggest that the improvements are equivalent to needing about 8% less compute and data to achieve the same test loss.