A3D: Adaptive 3D Networks for Video Action Recognition
This work addresses the problem of deploying video action recognition models on devices with variable computational constraints, such as edge devices, for practitioners and researchers.
This paper introduces A3D, an adaptive 3D network for video action recognition that can operate under various computational constraints after a single training. It optimizes configurations by balancing network width and spatio-temporal resolution, achieving significant performance boosts over baselines under the same computational constraints.
This paper presents A3D, an adaptive 3D network that can infer at a wide range of computational constraints with one-time training. Instead of training multiple models in a grid-search manner, it generates good configurations by trading off between network width and spatio-temporal resolution. Furthermore, the computation cost can be adapted after the model is deployed to meet variable constraints, for example, on edge devices. Even under the same computational constraints, the performance of our adaptive networks can be significantly boosted over the baseline counterparts by the mutual training along three dimensions. When a multiple pathway framework, e.g. SlowFast, is adopted, our adaptive method encourages a better trade-off between pathways than manual designs. Extensive experiments on the Kinetics dataset show the effectiveness of the proposed framework. The performance gain is also verified to transfer well between datasets and tasks. Code will be made available.