Mining Attribute Subspaces for Efficient Fine-tuning of 3D Foundation Models
For practitioners fine-tuning 3D foundation models, this work provides a method to identify and leverage disentangled attribute subspaces, potentially reducing computational cost while improving performance.
This paper investigates whether LoRA subspaces in 3D foundation models are associated with distinct attribute variations (texture, geometry, camera motion, lighting) and whether they are disentangled. The authors propose a method to extract these subspaces from synthetic data and show they are approximately orthogonal, enabling a reduced LoRA subspace that improves fine-tuning efficiency and prediction accuracy on real downstream tasks.
With the emergence of 3D foundation models, there is growing interest in fine-tuning them for downstream tasks, where LoRA is the dominant fine-tuning paradigm. As 3D datasets exhibit distinct variations in texture, geometry, camera motion, and lighting, there are interesting fundamental questions: 1) Are there LoRA subspaces associated with each type of variation? 2) Are these subspaces disentangled (i.e., orthogonal to each other)? 3) How do we compute them effectively? This paper provides answers to all these questions. We introduce a robust approach that generates synthetic datasets with controlled variations, fine-tunes a LoRA adapter on each dataset, and extracts a LoRA sub-space associated with each type of variation. We show that these subspaces are approximately disentangled. Integrating them leads to a reduced LoRA subspace that enables efficient LoRA fine-tuning with improved prediction accuracy for downstream tasks. In particular, we show that such a reduced LoRA subspace, despite being derived entirely from synthetic data, generalizes to real datasets. An ablation study validates the effectiveness of the choices in our approach.