Large Language Models Understand Layout
This addresses the problem of layout understanding in LLMs for researchers and developers, though it is incremental as it builds on existing model capabilities.
The paper demonstrates that large language models (LLMs) can process text layouts using spatial markers, showing a performance drop when these markers are removed, and finds this ability is enhanced by pretraining on coding data and instruction-tuning, with potential benefits for visual question-answering systems.
Large language models (LLMs) demonstrate extraordinary abilities in a wide range of natural language processing (NLP) tasks. In this paper, we show that, beyond text understanding capability, LLMs are capable of processing text layouts that are denoted by spatial markers. They are able to answer questions that require explicit spatial perceiving and reasoning, while a drastic performance drop is observed when the spatial markers from the original data are excluded. We perform a series of experiments with the GPT-3.5, Baichuan2, Llama2 and ChatGLM3 models on various types of layout-sensitive datasets for further analysis. The experimental results reveal that the layout understanding ability of LLMs is mainly introduced by the coding data for pretraining, which is further enhanced at the instruction-tuning stage. In addition, layout understanding can be enhanced by integrating low-cost, auto-generated data approached by a novel text game. Finally, we show that layout understanding ability is beneficial for building efficient visual question-answering (VQA) systems.