Tiny Transformers Excel at Sentence Compression
This work addresses the problem of high memory usage in language models for AI researchers, though it appears incremental as it builds on existing transformer architectures.
The paper tackles the inefficiency of token embeddings in large language models by showing that 1-3-layer transformers can compress standard English sentences into a single 3-kilobyte token, implying small networks can learn to construct valid sentences.
It is staggering that words of the English language, which are on average represented by 5--6 bytes of ASCII, require as much as 24 kilobytes when served to large language models. We show that there is room for more information in every token embedding. We demonstrate that 1--3-layer transformers are capable of encoding and subsequently decoding standard English sentences into as little as a single 3-kilobyte token. Our work implies that even small networks can learn to construct valid English sentences and suggests the possibility of optimising large language models by moving from sub-word token embeddings towards larger fragments of text.