Synergy: End-to-end Concept Model
This work addresses the need for more robust and flexible language model pipelines by demonstrating the feasibility of tokenizer-free architectures, which is an incremental advancement in the field.
The paper tackles the problem of tokenizer-free language modeling by introducing Synergy, an end-to-end model that learns to tokenize bytes and achieves comparable performance to Byte-level Byte Pair Encoder tokenizers with fewer concept tokens, while showing advantages over Llama3 at the same scale and dataset size.
In this paper, we present Synergy, a language model that bridges different levels of abstraction in an end-to-end fashion through a learned routing mechanism. Focusing on low-level linguistic abstraction, we trained our model as a byte-level language model. Our model spontaneously learns to tokenize bytes, producing fewer concept tokens than Byte-level Byte Pair Encoder (BBPE) tokenizers while keeping comparable performance. By comparing with Llama3, we observed an advantage of Synergy under the same model scale and training dataset size. Further studies show that the middle part (the higher abstraction part) of our model performs better when positional encodings are removed, suggesting the emergence of position-independent concepts. These findings demonstrate the feasibility of tokenizer-free architectures, paving the way for more robust and flexible pipelines.