Tune-In: Training Under Negative Environments with Interference for Attention Networks Simulating Cocktail Party Effect
This addresses the challenge of robust speech processing in noisy environments, offering a novel method with strong gains in efficiency and accuracy.
The paper tackles the cocktail party problem by proposing Tune-In, an attention network that learns speaker and speech representations in interfered conditions, achieving superior speaker verification and speech separation performance with lower memory and computational costs.
We study the cocktail party problem and propose a novel attention network called Tune-In, abbreviated for training under negative environments with interference. It firstly learns two separate spaces of speaker-knowledge and speech-stimuli based on a shared feature space, where a new block structure is designed as the building block for all spaces, and then cooperatively solves different tasks. Between the two spaces, information is cast towards each other via a novel cross- and dual-attention mechanism, mimicking the bottom-up and top-down processes of a human's cocktail party effect. It turns out that substantially discriminative and generalizable speaker representations can be learnt in severely interfered conditions via our self-supervised training. The experimental results verify this seeming paradox. The learnt speaker embedding has superior discriminative power than a standard speaker verification method; meanwhile, Tune-In achieves remarkably better speech separation performances in terms of SI-SNRi and SDRi consistently in all test modes, and especially at lower memory and computational consumption, than state-of-the-art benchmark systems.