BioVL-QR: Egocentric Biochemical Vision-and-Language Dataset Using Micro QR Codes
This addresses the problem of costly manual annotation in biochemical video analysis for researchers, though it is incremental by building on existing detection techniques.
The paper tackles the challenge of understanding biochemical videos by introducing BioVL-QR, a dataset with egocentric videos and protocols, and proposes a method combining Micro QR Code and hand object detection to improve object labeling, showing enhanced performance in localizing procedural steps.
This paper introduces BioVL-QR, a biochemical vision-and-language dataset comprising 23 egocentric experiment videos, corresponding protocols, and vision-and-language alignments. A major challenge in understanding biochemical videos is detecting equipment, reagents, and containers because of the cluttered environment and indistinguishable objects. Previous studies assumed manual object annotation, which is costly and time-consuming. To address the issue, we focus on Micro QR Codes. However, detecting objects using only Micro QR Codes is still difficult due to blur and occlusion caused by object manipulation. To overcome this, we propose an object labeling method combining a Micro QR Code detector with an off-the-shelf hand object detector. As an application of the method and BioVL-QR, we tackled the task of localizing the procedural steps in an instructional video. The experimental results show that using Micro QR Codes and our method improves biochemical video understanding. Data and code are available through https://nishi10mo.github.io/BioVL-QR/