TextManiA: Enriching Visual Feature by Text-driven Manifold Augmentation
This work addresses data scarcity and class imbalance in visual recognition tasks, offering a method to leverage pre-trained language models for augmentation, but it appears incremental as it builds on existing language and visual models.
The paper tackles the problem of enriching visual feature spaces using text-driven manifold augmentation, particularly in scarce or imbalanced data scenarios, and demonstrates that TextManiA improves performance in these settings, showing compatibility with label mix-based approaches.
We propose TextManiA, a text-driven manifold augmentation method that semantically enriches visual feature spaces, regardless of class distribution. TextManiA augments visual data with intra-class semantic perturbation by exploiting easy-to-understand visually mimetic words, i.e., attributes. This work is built on an interesting hypothesis that general language models, e.g., BERT and GPT, encompass visual information to some extent, even without training on visual training data. Given the hypothesis, TextManiA transfers pre-trained text representation obtained from a well-established large language encoder to a target visual feature space being learned. Our extensive analysis hints that the language encoder indeed encompasses visual information at least useful to augment visual representation. Our experiments demonstrate that TextManiA is particularly powerful in scarce samples with class imbalance as well as even distribution. We also show compatibility with the label mix-based approaches in evenly distributed scarce data.