LipLearner: Customizable Silent Speech Interactions on Mobile Devices
This work addresses the need for more expressive and customizable silent speech interactions for users seeking private communication on mobile devices, representing a strong specific gain rather than a foundational advancement.
The paper tackles the problem of limited vocabulary in silent speech interfaces by using contrastive learning to enable few-shot command customization, achieving an F1-score of 0.8947 for 25-command classification with only one shot and demonstrating robustness in real-world conditions.
Silent speech interface is a promising technology that enables private communications in natural language. However, previous approaches only support a small and inflexible vocabulary, which leads to limited expressiveness. We leverage contrastive learning to learn efficient lipreading representations, enabling few-shot command customization with minimal user effort. Our model exhibits high robustness to different lighting, posture, and gesture conditions on an in-the-wild dataset. For 25-command classification, an F1-score of 0.8947 is achievable only using one shot, and its performance can be further boosted by adaptively learning from more data. This generalizability allowed us to develop a mobile silent speech interface empowered with on-device fine-tuning and visual keyword spotting. A user study demonstrated that with LipLearner, users could define their own commands with high reliability guaranteed by an online incremental learning scheme. Subjective feedback indicated that our system provides essential functionalities for customizable silent speech interactions with high usability and learnability.