A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation
This work addresses the problem of separating dialogue, music, and effects in cinematic audio, which is important for audio post-production and accessibility, representing an incremental improvement with specific gains.
The paper tackled cinematic audio source separation by developing a generalized bandsplit neural network that outperforms the ideal ratio mask for dialogue extraction on the Divide and Remaster dataset.
Cinematic audio source separation is a relatively new subtask of audio source separation, with the aim of extracting the dialogue, music, and effects stems from their mixture. In this work, we developed a model generalizing the Bandsplit RNN for any complete or overcomplete partitions of the frequency axis. Psychoacoustically motivated frequency scales were used to inform the band definitions which are now defined with redundancy for more reliable feature extraction. A loss function motivated by the signal-to-noise ratio and the sparsity-promoting property of the 1-norm was proposed. We additionally exploit the information-sharing property of a common-encoder setup to reduce computational complexity during both training and inference, improve separation performance for hard-to-generalize classes of sounds, and allow flexibility during inference time with detachable decoders. Our best model sets the state of the art on the Divide and Remaster dataset with performance above the ideal ratio mask for the dialogue stem.