Pitfalls of Conditional Batch Normalization for Contextual Multi-Modal Learning
This work highlights a critical pitfall in a popular method for contextual learning, which is important for researchers in multi-modal AI to avoid performance degradation.
The paper investigates Conditional Batch Normalization (CBN) for multi-modal learning, finding that it can degrade visual feature learning and encourage shortcut learning, with experiments showing CBN learns minimal visual features on bird classification and partial features on histology datasets.
Humans have perfected the art of learning from multiple modalities through sensory organs. Despite their impressive predictive performance on a single modality, neural networks cannot reach human level accuracy with respect to multiple modalities. This is a particularly challenging task due to variations in the structure of respective modalities. Conditional Batch Normalization (CBN) is a popular method that was proposed to learn contextual features to aid deep learning tasks. This technique uses auxiliary data to improve representational power by learning affine transformations for convolutional neural networks. Despite the boost in performance observed by using CBN layers, our work reveals that the visual features learned by introducing auxiliary data via CBN deteriorates. We perform comprehensive experiments to evaluate the brittleness of CBN networks to various datasets, suggesting that learning from visual features alone could often be superior for generalization. We evaluate CBN models on natural images for bird classification and histology images for cancer type classification. We observe that the CBN network learns close to no visual features on the bird classification dataset and partial visual features on the histology dataset. Our extensive experiments reveal that CBN may encourage shortcut learning between the auxiliary data and labels.