Imitation Learning of Factored Multi-agent Reactive Models
This work addresses the problem of modeling uncoordinated biological agents for researchers in robotics and AI, offering an incremental improvement over existing deterministic methods.
The paper tackles imitation learning for multi-agent systems by learning stochastic policies directly from observational data without reward functions, achieving better predictive performance on a dataset of interacting flies and providing well-calibrated uncertainty estimates for future trajectories.
We apply recent advances in deep generative modeling to the task of imitation learning from biological agents. Specifically, we apply variations of the variational recurrent neural network model to a multi-agent setting where we learn policies of individual uncoordinated agents acting based on their perceptual inputs and their hidden belief state. We learn stochastic policies for these agents directly from observational data, without constructing a reward function. An inference network learned jointly with the policy allows for efficient inference over the agent's belief state given a sequence of its current perceptual inputs and the prior actions it performed, which lets us extrapolate observed sequences of behavior into the future while maintaining uncertainty estimates over future trajectories. We test our approach on a dataset of flies interacting in a 2D environment, where we demonstrate better predictive performance than existing approaches which learn deterministic policies with recurrent neural networks. We further show that the uncertainty estimates over future trajectories we obtain are well calibrated, which makes them useful for a variety of downstream processing tasks.