Beyond Static Perception: Integrating Temporal Context into VLMs for Cloth Folding
This addresses the challenge of manipulating clothes due to their complex dynamics and deformability, which is an incremental improvement in robotic cloth folding.
The paper tackles the problem of cloth folding by analyzing BiFold, a model that predicts language-conditioned pick-and-place actions from visual observations while implicitly encoding garment state through end-to-end learning. The result shows that leveraging temporal context improves state estimation for scenarios like crumpled garments or recovery from failed manipulations, with evidence of effective text-image alignment and temporal consistency.
Manipulating clothes is challenging due to their complex dynamics, high deformability, and frequent self-occlusions. Garments exhibit a nearly infinite number of configurations, making explicit state representations difficult to define. In this paper, we analyze BiFold, a model that predicts language-conditioned pick-and-place actions from visual observations, while implicitly encoding garment state through end-to-end learning. To address scenarios such as crumpled garments or recovery from failed manipulations, BiFold leverages temporal context to improve state estimation. We examine the internal representations of the model and present evidence that its fine-tuning and temporal context enable effective alignment between text and image regions, as well as temporal consistency.