Skeleton-Based Intake Gesture Detection With Spatial-Temporal Graph Convolutional Networks
This addresses dietary monitoring for individuals with unhealthy eating patterns, but it is incremental as it builds on existing spatial-temporal graph convolutional networks.
The study tackled automated detection of food intake gestures using skeleton data, achieving F1-scores up to 86.18% for eating and 74.84% for drinking on a lab dataset, and 85.40% and 67.80% on a smartphone dataset, demonstrating robustness in cross-dataset validation.
Overweight and obesity have emerged as widespread societal challenges, frequently linked to unhealthy eating patterns. A promising approach to enhance dietary monitoring in everyday life involves automated detection of food intake gestures. This study introduces a skeleton based approach using a model that combines a dilated spatial-temporal graph convolutional network (ST-GCN) with a bidirectional long-short-term memory (BiLSTM) framework, as called ST-GCN-BiLSTM, to detect intake gestures. The skeleton-based method provides key benefits, including environmental robustness, reduced data dependency, and enhanced privacy preservation. Two datasets were employed for model validation. The OREBA dataset, which consists of laboratory-recorded videos, achieved segmental F1-scores of 86.18% and 74.84% for identifying eating and drinking gestures. Additionally, a self-collected dataset using smartphone recordings in more adaptable experimental conditions was evaluated with the model trained on OREBA, yielding F1-scores of 85.40% and 67.80% for detecting eating and drinking gestures. The results not only confirm the feasibility of utilizing skeleton data for intake gesture detection but also highlight the robustness of the proposed approach in cross-dataset validation.