Poly-Autoregressive Prediction for Modeling Interactions
This addresses the challenge of modeling complex interactions in multi-agent systems for applications like robotics and autonomous driving, though it appears incremental as it adapts existing transformer methods to new scenarios.
The paper tackles the problem of predicting agent behavior in multi-agent interactions by introducing Poly-Autoregressive (PAR) modeling, which outperforms autoregressive methods across human action forecasting, autonomous vehicle trajectory prediction, and object pose forecasting scenarios.
We introduce a simple framework for predicting the behavior of an agent in multi-agent settings. In contrast to autoregressive (AR) tasks, such as language processing, our focus is on scenarios with multiple agents whose interactions are shaped by physical constraints and internal motivations. To this end, we propose Poly-Autoregressive (PAR) modeling, which forecasts an ego agent's future behavior by reasoning about the ego agent's state history and the past and current states of other interacting agents. At its core, PAR represents the behavior of all agents as a sequence of tokens, each representing an agent's state at a specific timestep. With minimal data pre-processing changes, we show that PAR can be applied to three different problems: human action forecasting in social situations, trajectory prediction for autonomous vehicles, and object pose forecasting during hand-object interaction. Using a small proof-of-concept transformer backbone, PAR outperforms AR across these three scenarios. The project website can be found at https://neerja.me/PAR/.