Are Layout-Infused Language Models Robust to Layout Distribution Shifts? A Case Study with Scientific Documents
This work addresses the problem of layout distribution shifts for researchers and practitioners using layout-infused models on scientific documents, highlighting an incremental evaluation methodology.
The study tested the robustness of layout-infused language models to layout distribution shifts in scientific document structure recovery, finding that performance degrades by up to 20 F1 points, and simple training strategies reduce degradation by over 35% relative F1 but fail to reach in-distribution levels.
Recent work has shown that infusing layout features into language models (LMs) improves processing of visually-rich documents such as scientific papers. Layout-infused LMs are often evaluated on documents with familiar layout features (e.g., papers from the same publisher), but in practice models encounter documents with unfamiliar distributions of layout features, such as new combinations of text sizes and styles, or new spatial configurations of textual elements. In this work we test whether layout-infused LMs are robust to layout distribution shifts. As a case study we use the task of scientific document structure recovery, segmenting a scientific paper into its structural categories (e.g., "title", "caption", "reference"). To emulate distribution shifts that occur in practice we re-partition the GROTOAP2 dataset. We find that under layout distribution shifts model performance degrades by up to 20 F1. Simple training strategies, such as increasing training diversity, can reduce this degradation by over 35% relative F1; however, models fail to reach in-distribution performance in any tested out-of-distribution conditions. This work highlights the need to consider layout distribution shifts during model evaluation, and presents a methodology for conducting such evaluations.