Performance Variability in Zero-Shot Classification
This addresses the reliability issue for researchers and practitioners using zero-shot classification, but it is incremental as it builds on existing methods.
The paper tackles the problem of performance instability in zero-shot classification under different class partitions, showing strong variability experimentally and proposing ensemble learning as a mitigation strategy.
Zero-shot classification (ZSC) is the task of learning predictors for classes not seen during training. Although the different methods in the literature are evaluated using the same class splits, little is known about their stability under different class partitions. In this work we show experimentally that ZSC performance exhibits strong variability under changing training setups. We propose the use ensemble learning as an attempt to mitigate this phenomena.