Improving Clinical Dataset Condensation with Mode Connectivity-based Trajectory Surrogates
This work addresses dataset condensation for clinical data, enabling more efficient and privacy-preserving access to regulated patient records for developing downstream clinical models, though it is incremental as it builds on existing trajectory alignment methods.
The paper tackled the problem of noisy and storage-intensive training trajectories in dataset condensation for clinical data by replacing full SGD trajectories with smooth quadratic Bézier curves, resulting in improved performance across five clinical datasets with stabilized gradients and faster convergence.
Dataset condensation (DC) enables the creation of compact, privacy-preserving synthetic datasets that can match the utility of real patient records, supporting democratised access to highly regulated clinical data for developing downstream clinical models. State-of-the-art DC methods supervise synthetic data by aligning the training dynamics of models trained on real and those trained on synthetic data, typically using full stochastic gradient descent (SGD) trajectories as alignment targets; however, these trajectories are often noisy, high-curvature, and storage-intensive, leading to unstable gradients, slow convergence, and substantial memory overhead. We address these limitations by replacing full SGD trajectories with smooth, low-loss parametric surrogates, specifically quadratic Bézier curves that connect the initial and final model states from real training trajectories. These mode-connected paths provide noise-free, low-curvature supervision signals that stabilise gradients, accelerate convergence, and eliminate the need for dense trajectory storage. We theoretically justify Bézier-mode connections as effective surrogates for SGD paths and empirically show that the proposed method outperforms state-of-the-art condensation approaches across five clinical datasets, yielding condensed datasets that enable clinically effective model development.