Reading Recognition in the Wild
This work addresses the need for contextual AI in smart glasses to record user interactions, extending reading understanding from constrained to diverse, realistic settings.
The paper tackles the problem of detecting when a user is reading in real-world scenarios using smart glasses, introducing a new task and a large-scale multimodal dataset with 100 hours of videos, and shows that combining RGB, eye gaze, and head pose modalities with a transformer model effectively solves this task.
To enable egocentric contextual AI in always-on smart glasses, it is crucial to be able to keep a record of the user's interactions with the world, including during reading. In this paper, we introduce a new task of reading recognition to determine when the user is reading. We first introduce the first-of-its-kind large-scale multimodal Reading in the Wild dataset, containing 100 hours of reading and non-reading videos in diverse and realistic scenarios. We then identify three modalities (egocentric RGB, eye gaze, head pose) that can be used to solve the task, and present a flexible transformer model that performs the task using these modalities, either individually or combined. We show that these modalities are relevant and complementary to the task, and investigate how to efficiently and effectively encode each modality. Additionally, we show the usefulness of this dataset towards classifying types of reading, extending current reading understanding studies conducted in constrained settings to larger scale, diversity and realism.