Entry-Flipped Transformer for Inference and Prediction of Participant Behavior
This addresses the challenge of error accumulation in spatio-temporal modeling for group interactions, with incremental improvements in specific domains like sports and pedestrian analysis.
The paper tackles the problem of inferring and predicting participant behavior in group activities like sports and dances by estimating how target participants react to others, proposing an Entry-Flipped Transformer that achieves best performance on datasets including tennis doubles and dance, with demonstrated ability to limit accumulated errors and recover from wrong estimations.
Some group activities, such as team sports and choreographed dances, involve closely coupled interaction between participants. Here we investigate the tasks of inferring and predicting participant behavior, in terms of motion paths and actions, under such conditions. We narrow the problem to that of estimating how a set target participants react to the behavior of other observed participants. Our key idea is to model the spatio-temporal relations among participants in a manner that is robust to error accumulation during frame-wise inference and prediction. We propose a novel Entry-Flipped Transformer (EF-Transformer), which models the relations of participants by attention mechanisms on both spatial and temporal domains. Unlike typical transformers, we tackle the problem of error accumulation by flipping the order of query, key, and value entries, to increase the importance and fidelity of observed features in the current frame. Comparative experiments show that our EF-Transformer achieves the best performance on a newly-collected tennis doubles dataset, a Ceilidh dance dataset, and two pedestrian datasets. Furthermore, it is also demonstrated that our EF-Transformer is better at limiting accumulated errors and recovering from wrong estimations.