itKD: Interchange Transfer-based Knowledge Distillation for 3D Object Detection
This addresses the need for lightweight models in 3D object detection, though it is incremental as it builds on existing knowledge distillation methods.
The paper tackles the problem of improving computational efficiency in point-cloud based 3D object detection by proposing an autoencoder-style framework with interchange transfer-based knowledge distillation, achieving superior performance on datasets like Waymo and nuScenes.
Point-cloud based 3D object detectors recently have achieved remarkable progress. However, most studies are limited to the development of network architectures for improving only their accuracy without consideration of the computational efficiency. In this paper, we first propose an autoencoder-style framework comprising channel-wise compression and decompression via interchange transfer-based knowledge distillation. To learn the map-view feature of a teacher network, the features from teacher and student networks are independently passed through the shared autoencoder; here, we use a compressed representation loss that binds the channel-wised compression knowledge from both student and teacher networks as a kind of regularization. The decompressed features are transferred in opposite directions to reduce the gap in the interchange reconstructions. Lastly, we present an head attention loss to match the 3D object detection information drawn by the multi-head self-attention mechanism. Through extensive experiments, we verify that our method can train the lightweight model that is well-aligned with the 3D point cloud detection task and we demonstrate its superiority using the well-known public datasets; e.g., Waymo and nuScenes.