ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking
This work addresses the problem of physical reasoning for AI agents in robotics and simulation, though it is incremental as it builds on existing vision-based methods for stability prediction.
The paper tackled the problem of enabling intelligent agents to acquire and apply physical intuition for object stacking by creating a simulation-based dataset of 20,000 stack configurations and training visual classifiers for stability prediction. The result was state-of-the-art performance on a real-world benchmark for stability prediction and successful active construction of stable stacks, even in challenging conditions like exceeding training heights or stabilising unstable structures.
Physical intuition is pivotal for intelligent agents to perform complex tasks. In this paper we investigate the passive acquisition of an intuitive understanding of physical principles as well as the active utilisation of this intuition in the context of generalised object stacking. To this end, we provide: a simulation-based dataset featuring 20,000 stack configurations composed of a variety of elementary geometric primitives richly annotated regarding semantics and structural stability. We train visual classifiers for binary stability prediction on the ShapeStacks data and scrutinise their learned physical intuition. Due to the richness of the training data our approach also generalises favourably to real-world scenarios achieving state-of-the-art stability prediction on a publicly available benchmark of block towers. We then leverage the physical intuition learned by our model to actively construct stable stacks and observe the emergence of an intuitive notion of stackability - an inherent object affordance - induced by the active stacking task. Our approach performs well even in challenging conditions where it considerably exceeds the stack height observed during training or in cases where initially unstable structures must be stabilised via counterbalancing.