ModSelect: Automatic Modality Selection for Synthetic-to-Real Domain Generalization
This addresses modality selection for multimodal systems in domain generalization, though it appears incremental as it builds on existing domain adaptation approaches.
The paper tackles the problem of selecting robust modalities for cross-domain activity recognition by proposing ModSelect, an unsupervised method that uses correlation between unimodal classifier predictions and domain discrepancy to select modalities. The method consistently improved performance on a synthetic-to-real domain adaptation benchmark, narrowing the domain gap.
Modality selection is an important step when designing multimodal systems, especially in the case of cross-domain activity recognition as certain modalities are more robust to domain shift than others. However, selecting only the modalities which have a positive contribution requires a systematic approach. We tackle this problem by proposing an unsupervised modality selection method (ModSelect), which does not require any ground-truth labels. We determine the correlation between the predictions of multiple unimodal classifiers and the domain discrepancy between their embeddings. Then, we systematically compute modality selection thresholds, which select only modalities with a high correlation and low domain discrepancy. We show in our experiments that our method ModSelect chooses only modalities with positive contributions and consistently improves the performance on a Synthetic-to-Real domain adaptation benchmark, narrowing the domain gap.