Aggregated Structural Representation with Large Language Models for Human-Centric Layout Generation
This work addresses the challenge of generating human-centric layouts for mobile applications, offering an incremental improvement over existing graph-based and vision-based methods by combining them with LLMs.
The paper tackles the problem of automated layout generation for mobile devices by proposing an Aggregation Structural Representation (ASR) module that integrates graph networks with large language models to preserve structural information and enhance generative capability, achieving strong performance on the RICO dataset as demonstrated by mIoU metrics and a user study.
Time consumption and the complexity of manual layout design make automated layout generation a critical task, especially for multiple applications across different mobile devices. Existing graph-based layout generation approaches suffer from limited generative capability, often resulting in unreasonable and incompatible outputs. Meanwhile, vision based generative models tend to overlook the original structural information, leading to component intersections and overlaps. To address these challenges, we propose an Aggregation Structural Representation (ASR) module that integrates graph networks with large language models (LLMs) to preserve structural information while enhancing generative capability. This novel pipeline utilizes graph features as hierarchical prior knowledge, replacing the traditional Vision Transformer (ViT) module in multimodal large language models (MLLM) to predict full layout information for the first time. Moreover, the intermediate graph matrix used as input for the LLM is human editable, enabling progressive, human centric design generation. A comprehensive evaluation on the RICO dataset demonstrates the strong performance of ASR, both quantitatively using mean Intersection over Union (mIoU), and qualitatively through a crowdsourced user study. Additionally, sampling on relational features ensures diverse layout generation, further enhancing the adaptability and creativity of the proposed approach.