Exploring new ways: Enforcing representational dissimilarity to learn new features and reduce error consistency
This work addresses the issue of error consistency in machine learning ensembles, which is incremental as it builds on prior decorrelation methods by focusing on intermediate representations.
The paper tackles the problem of correlated predictions and common failure modes in ensembles of independently trained models by promoting dissimilar intermediate representations during training, resulting in less correlated outputs and slightly lower error consistency, which leads to higher ensemble accuracy.
Independently trained machine learning models tend to learn similar features. Given an ensemble of independently trained models, this results in correlated predictions and common failure modes. Previous attempts focusing on decorrelation of output predictions or logits yielded mixed results, particularly due to their reduction in model accuracy caused by conflicting optimization objectives. In this paper, we propose the novel idea of utilizing methods of the representational similarity field to promote dissimilarity during training instead of measuring similarity of trained models. To this end, we promote intermediate representations to be dissimilar at different depths between architectures, with the goal of learning robust ensembles with disjoint failure modes. We show that highly dissimilar intermediate representations result in less correlated output predictions and slightly lower error consistency, resulting in higher ensemble accuracy. With this, we shine first light on the connection between intermediate representations and their impact on the output predictions.