Words That Make Language Models Perceive
This work addresses the challenge of enhancing multimodal perception in language models for AI researchers, though it is incremental as it builds on existing prompting techniques.
The study tackled the problem of aligning text-only large language models with sensory modalities by using explicit sensory prompts, resulting in reliable activation of modality-appropriate representations without actual sensory input.
Large language models (LLMs) trained purely on text ostensibly lack any direct perceptual experience, yet their internal representations are implicitly shaped by multimodal regularities encoded in language. We test the hypothesis that explicit sensory prompting can surface this latent structure, bringing a text-only LLM into closer representational alignment with specialist vision and audio encoders. When a sensory prompt tells the model to 'see' or 'hear', it cues the model to resolve its next-token predictions as if they were conditioned on latent visual or auditory evidence that is never actually supplied. Our findings reveal that lightweight prompt engineering can reliably activate modality-appropriate representations in purely text-trained LLMs.