The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models
This addresses a critical bottleneck for using LLMs as universal assistants in domains like mobile health sensing, though it is incremental as it builds on known tokenization issues.
The paper identifies that large language models (LLMs) struggle with tokenizing numerical/temporal data from sources like wearables or health records, treating consecutive values as separate tokens and ignoring temporal relationships. It presents a case study showing popular LLMs tokenize such data incorrectly and highlights potential solutions like prompt tuning with embedding layers or multimodal adapters.
Large Language Models (LLMs) have demonstrated remarkable generalization across diverse tasks, leading individuals to increasingly use them as personal assistants and universal computing engines. Nevertheless, a notable obstacle emerges when feeding numerical/temporal data into these models, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. Here, we discuss recent works that employ LLMs for human-centric tasks such as in mobile health sensing and present a case study showing that popular LLMs tokenize temporal data incorrectly. To address that, we highlight potential solutions such as prompt tuning with lightweight embedding layers as well as multimodal adapters, that can help bridge this "modality gap". While the capability of language models to generalize to other modalities with minimal or no finetuning is exciting, this paper underscores the fact that their outputs cannot be meaningful if they stumble over input nuances.