XAttnRes: Cross-Stage Attention Residuals for Medical Image Segmentation
For medical image segmentation, XAttnRes offers a lightweight, plug-and-play mechanism that enhances existing networks, though the gains are incremental over strong baselines.
XAttnRes improves medical image segmentation by replacing fixed skip connections with learned, selective aggregation across encoder-decoder stages, achieving consistent performance gains across four datasets and three imaging modalities, and matching baseline performance even without skip connections.
In the field of Large Language Models (LLMs), Attention Residuals have recently demonstrated that learned, selective aggregation over all preceding layer outputs can outperform fixed residual connections. We propose Cross-Stage Attention Residuals (XAttnRes), a mechanism that maintains a global feature history pool accumulating both encoder and decoder stage outputs. Through lightweight pseudo-query attention, each stage selectively aggregates from all preceding representations. To bridge the gap between the same-dimensional Transformer layers in LLMs and the multi-scale encoder-decoder stages in segmentation networks, XAttnRes introduces spatial alignment and channel projection steps that handle cross-resolution features with negligible overhead. When added to existing segmentation networks, XAttnRes consistently improves performance across four datasets and three imaging modalities. We further observe that XAttnRes alone, even without skip connections, achieves performance on par with the baseline, suggesting that learned aggregation can recover the inter-stage information flow traditionally provided by predetermined connections.