Prompt When the Animal is: Temporal Animal Behavior Grounding with Positional Recovery Training
This work addresses the problem of sparse and uniformly distributed moments in animal behavior analysis for researchers in multimodal learning, representing an incremental improvement with a novel training method.
The paper tackles the challenge of temporal grounding in animal behavior data by proposing a Positional Recovery Training framework that prompts the model with start and end times during training, achieving an IoU@0.3 of 38.52 on the Animal Kingdom dataset.
Temporal grounding is crucial in multimodal learning, but it poses challenges when applied to animal behavior data due to the sparsity and uniform distribution of moments. To address these challenges, we propose a novel Positional Recovery Training framework (Port), which prompts the model with the start and end times of specific animal behaviors during training. Specifically, Port enhances the baseline model with a Recovering part to predict flipped label sequences and align distributions with a Dual-alignment method. This allows the model to focus on specific temporal regions prompted by ground-truth information. Extensive experiments on the Animal Kingdom dataset demonstrate the effectiveness of Port, achieving an IoU@0.3 of 38.52. It emerges as one of the top performers in the sub-track of MMVRAC in ICME 2024 Grand Challenges.