Self-supervised Learning by View Synthesis
This work addresses the need for better initialization methods for vision transformers in 3D vision tasks, offering a novel self-supervised approach that is incremental in improving representation learning.
The paper tackles the problem of self-supervised learning for vision transformers by introducing view-synthesis autoencoders (VSA), which use multi-view data to synthesize images from different poses, resulting in significant outperformance in linear probing and competitive fine-tuning on 3D classification datasets like ModelNet40, ShapeNet Core55, and ScanObjectNN.
We present view-synthesis autoencoders (VSA) in this paper, which is a self-supervised learning framework designed for vision transformers. Different from traditional 2D pretraining methods, VSA can be pre-trained with multi-view data. In each iteration, the input to VSA is one view (or multiple views) of a 3D object and the output is a synthesized image in another target pose. The decoder of VSA has several cross-attention blocks, which use the source view as value, source pose as key, and target pose as query. They achieve cross-attention to synthesize the target view. This simple approach realizes large-angle view synthesis and learns spatial invariant representation, where the latter is decent initialization for transformers on downstream tasks, such as 3D classification on ModelNet40, ShapeNet Core55, and ScanObjectNN. VSA outperforms existing methods significantly for linear probing and is competitive for fine-tuning. The code will be made publicly available.