BrickNet: Graph-Backed Generative Brick Assembly
This work addresses the challenge of generating physically valid assembly sequences for diverse LEGO bricks, enabling more complex and realistic brick assembly generation.
BrickNet introduces a graph-based program representation to generate LEGO build sequences for thousands of part types, overcoming the physical constraint violations that plague direct pose prediction. The method is trained on a new dataset of over 100,000 human-designed LDraw objects and achieves improved physical grounding.
We train a language model to generate LEGO-brick build sequences. While prior work has been restricted to discrete, voxel-like towers, we consider a much broader set of pieces, encompassing thousands of part types with diverse connection semantics. To enable this, we first collect a large-scale dataset of over 100,000 human-designed LDraw brick objects and scenes. The complexity of our setting makes it challenging to autoregressively assemble structures that satisfy physical constraints. When predicting block pose directly, build sequences quickly become invalid after a small number of steps. Although pieces are placed in 3D space, it is the spatial relationships of the parts which define the whole. With this in mind, we design a graph-based program representation that parametrizes structure through connectivity, improving the physical grounding of generated sequences. To enable future applications, we make our dataset and models available for research purposes. https://kulits.github.io/BrickNet