Sim-to-real for high-resolution optical tactile sensing: From images to 3D contact force distributions
This work addresses the challenge of data collection for vision-based tactile sensors, which is a bottleneck for robotics and manipulation tasks.
This paper proposes a method to generate synthetic training data for vision-based tactile sensors, specifically for extracting 3D contact force distributions from images. By simulating material deformation and particle motion, an artificial neural network trained entirely on synthetic data achieved high accuracy on real-world tactile images and was transferable across multiple sensors.
The images captured by vision-based tactile sensors carry information about high-resolution tactile fields, such as the distribution of the contact forces applied to their soft sensing surface. However, extracting the information encoded in the images is challenging and often addressed with learning-based approaches, which generally require a large amount of training data. This article proposes a strategy to generate tactile images in simulation for a vision-based tactile sensor based on an internal camera that tracks the motion of spherical particles within a soft material. The deformation of the material is simulated in a finite element environment under a diverse set of contact conditions, and spherical particles are projected to a simulated image. Features extracted from the images are mapped to the 3D contact force distribution, with the ground truth also obtained via finite-element simulations, with an artificial neural network that is therefore entirely trained on synthetic data avoiding the need for real-world data collection. The resulting model exhibits high accuracy when evaluated on real-world tactile images, is transferable across multiple tactile sensors without further training, and is suitable for efficient real-time inference.