Leveraging Foundation Models To learn the shape of semi-fluid deformable objects
This work addresses the challenge of manipulating semi-fluid deformable objects in robotics, offering a method to reduce reliance on manually labeled data, though it is incremental as it builds on existing knowledge distillation techniques.
The paper tackles the problem of characterizing deformable objects, specifically weld pools, by using foundation models as teachers in a knowledge distillation framework to extract keypoints without pre-training or labeled datasets, achieving an error of 13.4 pixels for keypoint retrieval and a mIoU of 75.26% for mask prediction.
One of the difficulties imposed on the manipulation of deformable objects is their characterization and the detection of representative keypoints for the purpose of manipulation. A keen interest was manifested by researchers in the last decade to characterize and manipulate deformable objects of non-fluid nature, such as clothes and ropes. Even though several propositions were made in the regard of object characterization, however researchers were always confronted with the need of pixel-level information of the object through images to extract relevant information. This usually is accomplished by means of segmentation networks trained on manually labeled data for this purpose. In this paper, we address the subject of characterizing weld pool to define stable features that serve as information for further motion control objectives. We achieve this by employing different pipelines. The first one consists of characterizing fluid deformable objects through the use of a generative model that is trained using a teacher-student framework. And in the second one we leverage foundation models by using them as teachers to characterize the object in the image, without the need of any pre-training and any dataset. The performance of knowledge distillation from foundation models into a smaller generative model shows prominent results in the characterization of deformable objects. The student network was capable of learning to retrieve the keypoitns of the object with an error of 13.4 pixels. And the teacher was evaluated based on its capacities to retrieve pixel level information represented by the object mask, with a mean Intersection Over Union (mIoU) of 75.26%.