Proactive Hearing Assistants that Isolate Egocentric Conversations
This work addresses the challenge of hearing assistance in noisy environments by enabling proactive, on-device operation without explicit user prompts, though it is incremental in improving existing hearing aid technologies.
The paper tackles the problem of automatically identifying and isolating conversation partners in multi-speaker settings using egocentric binaural audio, achieving generalization on real-world test sets with 2- and 3-speaker conversations totaling 6.8 hours from 11 participants.
We introduce proactive hearing assistants that automatically identify and separate the wearer's conversation partners, without requiring explicit prompts. Our system operates on egocentric binaural audio and uses the wearer's self-speech as an anchor, leveraging turn-taking behavior and dialogue dynamics to infer conversational partners and suppress others. To enable real-time, on-device operation, we propose a dual-model architecture: a lightweight streaming model runs every 12.5 ms for low-latency extraction of the conversation partners, while a slower model runs less frequently to capture longer-range conversational dynamics. Results on real-world 2- and 3-speaker conversation test sets, collected with binaural egocentric hardware from 11 participants totaling 6.8 hours, show generalization in identifying and isolating conversational partners in multi-conversation settings. Our work marks a step toward hearing assistants that adapt proactively to conversational dynamics and engagement. More information can be found on our website: https://proactivehearing.cs.washington.edu/