xVal: A Continuous Numerical Tokenization for Scientific Language Models
This addresses the challenge of processing scientific datasets with LLMs, potentially enabling foundation models for science, though it appears incremental as it builds on existing tokenization strategies.
The paper tackled the problem of LLMs' poor handling of numerically-dense scientific data due to discrete tokenization, introducing xVal as a continuous numerical tokenization method that generally outperforms other strategies in out-of-distribution generalization and computational efficiency.
Due in part to their discontinuous and discrete default encodings for numbers, Large Language Models (LLMs) have not yet been commonly used to process numerically-dense scientific datasets. Rendering datasets as text, however, could help aggregate diverse and multi-modal scientific data into a single training corpus, thereby potentially facilitating the development of foundation models for science. In this work, we introduce xVal, a strategy for continuously tokenizing numbers within language models that results in a more appropriate inductive bias for scientific applications. By training specially-modified language models from scratch on a variety of scientific datasets formatted as text, we find that xVal generally outperforms other common numerical tokenization strategies on metrics including out-of-distribution generalization and computational efficiency.