MedSteer: Counterfactual Endoscopic Synthesis via Training-Free Activation Steering
This work addresses the problem of generating causal training data for medical imaging, which is crucial for improving the robustness and interpretability of AI models in clinical settings, particularly for endoscopic diagnosis.
This paper introduces MedSteer, a training-free activation-steering framework for endoscopic image synthesis that generates counterfactual pairs where only a specific concept (e.g., pathology) differs. It achieves flip rates of 0.800, 0.925, and 0.950 for counterfactual generation, outperforms inversion-based baselines in concept flip rate and structural preservation, and improves polyp detection AUC to 0.9755 compared to 0.9083 for re-prompting.
Generative diffusion models are increasingly used for medical imaging data augmentation, but text prompting cannot produce causal training data. Re-prompting rerolls the entire generation trajectory, altering anatomy, texture, and background. Inversion-based editing methods introduce reconstruction error that causes structural drift. We propose MedSteer, a training-free activation-steering framework for endoscopic synthesis. MedSteer identifies a pathology vector for each contrastive prompt pair in the cross-attention layers of a diffusion transformer. At inference time, it steers image activations along this vector, generating counterfactual pairs from scratch where the only difference is the steered concept. All other structure is preserved by construction. We evaluate MedSteer across three experiments on Kvasir v3 and HyperKvasir. On counterfactual generation across three clinical concept pairs, MedSteer achieves flip rates of 0.800, 0.925, and 0.950, outperforming the best inversion-based baseline in both concept flip rate and structural preservation. On dye disentanglement, MedSteer achieves 75% dye removal against 20% (PnP) and 10% (h-Edit). On downstream polyp detection, augmenting with MedSteer counterfactual pairs achieves ViT AUC of 0.9755 versus 0.9083 for quantity-matched re-prompting, confirming that counterfactual structure drives the gain. Code is at link https://github.com/phamtrongthang123/medsteer