When 2D Tasks Meet 1D Serialization: On Serialization Friction in Structured Tasks
For researchers working on structured tasks with LLMs, this work highlights the limitations of 1D serialization and suggests that preserving 2D layout is a promising direction, though the findings are on a small synthetic testbed.
The paper investigates 'serialization friction'—the representational burden when LLMs process 2D structured tasks (matrix transpose, Game of Life, LU decomposition) as 1D token sequences. A vision-augmented pathway that preserves 2D layout consistently outperforms the text-only serialized pathway, with the gap widening at larger dimensions and error patterns becoming spatially structured.
Large language models (LLMs) conventionally process structured inputs as 1D token sequences. While natural for prose, such linearization may introduce additional representational burden for tasks whose computation depends directly on explicit 2D structure, because row--column alignment and local neighborhoods are no longer directly expressed in the input. We study this setting, which we refer to as serialization friction, on a small diagnostic testbed of synthetic tasks with explicit 2D structure: matrix transpose, Conway's Game of Life, and LU decomposition. To examine this question, we compare a text-only language pathway over serialized inputs with a vision-augmented pathway, built on the same language backbone, that receives the same underlying content rendered in task-faithful 2D layout, yielding a system-level comparison between two end-to-end input pathways. Across the tasks and settings we study, the visual pathway consistently outperforms the textual pathway; the gap often widens at larger dimensions, and error patterns under serialization become increasingly spatially structured. These findings indicate that the relationship between input representation and model performance on such tasks warrants further investigation, and suggest that preserving task-relevant 2D layout is a promising direction for structured 2D tasks.