Time-Frequency-Based Attention Cache Memory Model for Real-Time Speech Separation
This addresses the challenge of real-time speech separation for applications requiring low latency, though it is incremental as it builds on existing causal models.
The paper tackled the problem of causal speech separation models underperforming due to difficulties in retaining historical information, proposing the TFACM model which achieved comparable performance to the SOTA TF-GridNet-Causal model with significantly lower complexity and fewer trainable parameters.
Existing causal speech separation models often underperform compared to non-causal models due to difficulties in retaining historical information. To address this, we propose the Time-Frequency Attention Cache Memory (TFACM) model, which effectively captures spatio-temporal relationships through an attention mechanism and cache memory (CM) for historical information storage. In TFACM, an LSTM layer captures frequency-relative positions, while causal modeling is applied to the time dimension using local and global representations. The CM module stores past information, and the causal attention refinement (CAR) module further enhances time-based feature representations for finer granularity. Experimental results showed that TFACM achieveed comparable performance to the SOTA TF-GridNet-Causal model, with significantly lower complexity and fewer trainable parameters. For more details, visit the project page: https://cslikai.cn/TFACM/.