What if Othello-Playing Language Models Could See?
This work addresses the symbol grounding problem in language models for AI researchers, offering incremental insights into multi-modal efficiency in a controlled testbed.
The paper tackled the problem of whether multi-modal learning improves language models' world understanding by introducing VISOTHELLO, a model trained on both move sequences and board images for Othello, and found that it enhances performance, robustness, and cross-modal alignment compared to text-only approaches.
Language models are often said to face a symbol grounding problem. While some have argued the problem can be solved without resort to other modalities, many have speculated that grounded learning is more efficient. We explore this question in Othello, a simplified, rule-based world that offers a controlled and interpretable testbed for studying world understanding. Building on prior work, we introduce VISOTHELLO, a multi-modal model trained jointly on move sequences and board images. Using the Othello rule understanding task, we examine whether multi-modal learning provides advantages over text-only approaches. We further evaluate robustness under semantically irrelevant perturbations and analyze the consistency of cross-modal alignment. Our results suggest that multi-modal training not only improves performance and robustness but also promotes convergence toward shared internal representations across different model architectures.