Alignment-Uniformity aware Representation Learning for Zero-shot Video Classification
This addresses the problem of limited generalization in zero-shot video classification for researchers and practitioners, offering a novel approach that is incremental but with strong specific gains.
The paper tackles zero-shot video classification by enhancing generalization to unseen classes through an end-to-end framework that preserves alignment and uniformity in representations, achieving relative improvements of 28.1% on UCF101 and 27.0% on HMDB51 compared to state-of-the-art methods.
Most methods tackle zero-shot video classification by aligning visual-semantic representations within seen classes, which limits generalization to unseen classes. To enhance model generalizability, this paper presents an end-to-end framework that preserves alignment and uniformity properties for representations on both seen and unseen classes. Specifically, we formulate a supervised contrastive loss to simultaneously align visual-semantic features (i.e., alignment) and encourage the learned features to distribute uniformly (i.e., uniformity). Unlike existing methods that only consider the alignment, we propose uniformity to preserve maximal-info of existing features, which improves the probability that unobserved features fall around observed data. Further, we synthesize features of unseen classes by proposing a class generator that interpolates and extrapolates the features of seen classes. Besides, we introduce two metrics, closeness and dispersion, to quantify the two properties and serve as new measurements of model generalizability. Experiments show that our method significantly outperforms SoTA by relative improvements of 28.1% on UCF101 and 27.0% on HMDB51. Code is available.