MoST: Efficient Monarch Sparse Tuning for 3D Representation Learning
This addresses the need for efficient fine-tuning in 3D point cloud tasks, offering a novel approach that is incremental in improving parameter efficiency while maintaining high performance.
The paper tackles the problem of parameter-efficient fine-tuning for 3D representation learning by introducing MoST, a reparameterization-based method that uses sparse Point Monarch matrices to capture local geometric features without inference overhead, achieving state-of-the-art results such as 97.5% accuracy on ScanObjectNN and 96.2% on ModelNet40 classification.
We introduce Monarch Sparse Tuning (MoST), the first reparameterization-based parameter-efficient fine-tuning (PEFT) method tailored for 3D representation learning. Unlike existing adapter-based and prompt-tuning 3D PEFT methods, MoST introduces no additional inference overhead and is compatible with many 3D representation learning backbones. At its core, we present a new family of structured matrices for 3D point clouds, Point Monarch, which can capture local geometric features of irregular points while offering high expressiveness. MoST reparameterizes the dense update weight matrices as our sparse Point Monarch matrices, significantly reducing parameters while retaining strong performance. Experiments on various backbones show that MoST is simple, effective, and highly generalizable. It captures local features in point clouds, achieving state-of-the-art results on multiple benchmarks, e.g., 97.5% acc. on ScanObjectNN (PB_50_RS) and 96.2% on ModelNet40 classification, while it can also combine with other matrix decompositions (e.g., Low-rank, Kronecker) to further reduce parameters.