Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing
This work addresses data efficiency and generalization issues for vision-based tactile sensors in robotics, representing an incremental improvement over existing data-driven approaches.
The paper tackled the problem of data efficiency and generalization in vision-based tactile sensing by proposing a sim-to-real strategy that trains a deep neural network entirely on simulated data, achieving accurate predictions on real data and promising generalization to unseen contact conditions.
Data-driven approaches to tactile sensing aim to overcome the complexity of accurately modeling contact with soft materials. However, their widespread adoption is impaired by concerns about data efficiency and the capability to generalize when applied to various tasks. This paper focuses on both these aspects with regard to a vision-based tactile sensor, which aims to reconstruct the distribution of the three-dimensional contact forces applied on its soft surface. Accurate models for the soft materials and the camera projection, derived via state-of-the-art techniques in the respective domains, are employed to generate a dataset in simulation. A strategy is proposed to train a tailored deep neural network entirely from the simulation data. The resulting learning architecture is directly transferable across multiple tactile sensors without further training and yields accurate predictions on real data, while showing promising generalization capabilities to unseen contact conditions.