Attention mechanisms for physiological signal deep learning: which attention should we take?
This work addresses the selection of attention mechanisms for physiological signal deep learning, providing empirical guidance for researchers in medical AI, though it is incremental as it compares existing methods on specific tasks.
The study experimentally analyzed four attention mechanisms and three CNN architectures for physiological signal prediction tasks, finding that spatial attention performed best for hypotension classification and channel attention achieved the lowest error for cardiac output regression, with CNN-attention hybrids outperforming stand-alone self-attention models in performance and convergence.
Attention mechanisms are widely used to dramatically improve deep learning model performance in various fields. However, their general ability to improve the performance of physiological signal deep learning model is immature. In this study, we experimentally analyze four attention mechanisms (e.g., squeeze-and-excitation, non-local, convolutional block attention module, and multi-head self-attention) and three convolutional neural network (CNN) architectures (e.g., VGG, ResNet, and Inception) for two representative physiological signal prediction tasks: the classification for predicting hypotension and the regression for predicting cardiac output (CO). We evaluated multiple combinations for performance and convergence of physiological signal deep learning model. Accordingly, the CNN models with the spatial attention mechanism showed the best performance in the classification problem, whereas the channel attention mechanism achieved the lowest error in the regression problem. Moreover, the performance and convergence of the CNN models with attention mechanisms were better than stand-alone self-attention models in both problems. Hence, we verified that convolutional operation and attention mechanisms are complementary and provide faster convergence time, despite the stand-alone self-attention models requiring fewer parameters.