SBF: An Effective Representation to Augment Skeleton for Video-based Human Action Recognition
This work addresses limitations in skeleton-based action recognition for computer vision applications, offering an incremental improvement by enhancing existing pipelines without extra annotation.
The paper tackles the problem of video-based human action recognition by augmenting skeleton data with a new representation called Scale-Body-Flow (SBF) to capture depth, body contour, and human-object interactions, resulting in significantly higher accuracy compared to skeleton-only approaches.
Many modern video-based human action recognition (HAR) approaches use 2D skeleton as the intermediate representation in their prediction pipelines. Despite overall encouraging results, these approaches still struggle in many common scenes, mainly because the skeleton does not capture critical action-related information pertaining to the depth of the joints, contour of the human body, and interaction between the human and objects. To address this, we propose an effective approach to augment skeleton with a representation capturing action-related information in the pipeline of HAR. The representation, termed Scale-Body-Flow (SBF), consists of three distinct components, namely a scale map volume given by the scale (and hence depth information) of each joint, a body map outlining the human subject, and a flow map indicating human-object interaction given by pixel-wise optical flow values. To predict SBF, we further present SFSNet, a novel segmentation network supervised by the skeleton and optical flow without extra annotation overhead beyond the existing skeleton extraction. Extensive experiments across different datasets demonstrate that our pipeline based on SBF and SFSNet achieves significantly higher HAR accuracy with similar compactness and efficiency as compared with the state-of-the-art skeleton-only approaches.