Typing on Any Surface: A Deep Learning-based Method for Real-Time Keystroke Detection in Augmented Reality
This addresses the problem of poor text entry interfaces in AR for users, offering a more ergonomic and accurate alternative to existing methods, though it appears incremental as it builds on existing hand landmark extraction and neural network techniques.
The paper tackles the problem of frustrating text entry in augmented reality by proposing a deep learning method that enables real-time keystroke detection from user-perspective video, allowing typing on any flat surface without a physical or virtual keyboard, achieving 91.05% accuracy at 40 words per minute.
Frustrating text entry interface has been a major obstacle in participating in social activities in augmented reality (AR). Popular options, such as mid-air keyboard interface, wireless keyboards or voice input, either suffer from poor ergonomic design, limited accuracy, or are simply embarrassing to use in public. This paper proposes and validates a deep-learning based approach, that enables AR applications to accurately predict keystrokes from the user perspective RGB video stream that can be captured by any AR headset. This enables a user to perform typing activities on any flat surface and eliminates the need of a physical or virtual keyboard. A two-stage model, combing an off-the-shelf hand landmark extractor and a novel adaptive Convolutional Recurrent Neural Network (C-RNN), was trained using our newly built dataset. The final model was capable of adaptive processing user-perspective video streams at ~32 FPS. This base model achieved an overall accuracy of $91.05\%$ when typing 40 Words per Minute (wpm), which is how fast an average person types with two hands on a physical keyboard. The Normalised Levenshtein Distance also further confirmed the real-world applicability of that our approach. The promising results highlight the viability of our approach and the potential for our method to be integrated into various applications. We also discussed the limitations and future research required to bring such technique into a production system.