Soyoung Choi

Seong-Hyeon Hwang, Soyoung Choi, Steven Euijong Whang

Multimodal models often over-rely on dominant modalities, failing to achieve optimal performance. While prior work focuses on modifying training objectives or optimization procedures, data-centric solutions remain underexplored. We propose MIDAS, a novel data augmentation strategy that generates misaligned samples with semantically inconsistent cross-modal information, labeled using unimodal confidence scores to compel learning from contradictory signals. However, this confidence-based labeling can still favor the more confident modality. To address this within our misaligned samples, we introduce weak-modality weighting, which dynamically increases the loss weight of the least confident modality, thereby helping the model fully utilize weaker modality. Furthermore, when misaligned features exhibit greater similarity to the aligned features, these misaligned samples pose a greater challenge, thereby enabling the model to better distinguish between classes. To leverage this, we propose hard-sample weighting, which prioritizes such semantically ambiguous misaligned samples. Experiments on multiple multimodal classification benchmarks demonstrate that MIDAS significantly outperforms related baselines in addressing modality imbalance.

Anand A. Joshi, Soyoung Choi, Haleh Akrami et al.

Due to the spontaneous nature of resting-state fMRI (rs-fMRI) signals, cross-subject comparison and therefore, group studies of rs-fMRI are challenging. Most existing group comparison methods use features extracted from the fMRI time series, such as connectivity features, independent component analysis (ICA), and functional connectivity density (FCD) methods. However, in group studies, especially in the case of spectrum disorders, distances to a single atlas or a representative subject do not fully reflect the differences between subjects that may lie on a multi-dimensional spectrum. Moreover, there may not exist an individual subject or even an average atlas in such cases that is representative of all subjects. Here we describe an approach that measures pairwise distances between the synchronized rs-fMRI signals of pairs of subjects instead of to a single reference point. We also present a method for fMRI data comparison that leverages this generated pairwise feature to establish a radial basis function kernel matrix. This kernel matrix is used in turn to perform kernel regression of rs-fMRI to a clinical variable such as a cognitive or neurophysiological performance score of interest. This method opens a new pointwise analysis paradigm for fMRI data. We demonstrate the application of this method by performing a pointwise analysis on the cortical surface using rs-fMRI data to identify cortical regions associated with variability in ADHD index. While pointwise analysis methods are common in anatomical studies such as cortical thickness analysis and voxel- and tensor-based morphometry and its variants, such a method is lacking for rs-fMRI and could improve the utility of rs-fMRI for group studies. The method presented in this paper is aimed at filling this gap.