On- Device Information Extraction from Screenshots in form of tags
This work addresses privacy and usability issues for mobile users by enabling efficient on-device search of screenshots, though it is incremental in combining existing techniques for a specific application.
The paper tackles the problem of making mobile screenshots searchable by developing an on-device pipeline that extracts text and tags from images, addressing privacy concerns and supporting multiple languages, with results showing reduced user effort and improved search experience.
We propose a method to make mobile screenshots easily searchable. In this paper, we present the workflow in which we: 1) preprocessed a collection of screenshots, 2) identified script presentin image, 3) extracted unstructured text from images, 4) identifiedlanguage of the extracted text, 5) extracted keywords from the text, 6) identified tags based on image features, 7) expanded tag set by identifying related keywords, 8) inserted image tags with relevant images after ranking and indexed them to make it searchable on device. We made the pipeline which supports multiple languages and executed it on-device, which addressed privacy concerns. We developed novel architectures for components in the pipeline, optimized performance and memory for on-device computation. We observed from experimentation that the solution developed can reduce overall user effort and improve end user experience while searching, whose results are published.