Heterogeneous Tactile Transformer
This work addresses the bottleneck of sensor heterogeneity in tactile sensing, enabling scalable learning of manipulation policies from diverse tactile data.
The authors propose the Heterogeneous Tactile Transformer (HTT) to learn shared representations across different tactile sensors, enabling transfer learning. On diverse perception and manipulation tasks, HTT achieves effective adaptation to new tasks and unseen sensors, leveraging a pretraining dataset of 1.6M paired frames.
Tactile sensors are inherently heterogeneous: a model trained on one sensor cannot be directly used on another, which limits learning contact-rich manipulation policies from diverse tactile data at scale. To bridge this gap, we propose the Heterogeneous Tactile Transformer (HTT), a framework that learns shared tactile representations across heterogeneous sensors. HTT consists of sensor-specific encoders and a shared transformer trunk, and is pretrained with per-modality masked reconstruction together with cross-modal alignment between paired sensors. Pretraining uses our novel Heterogeneous Paired Tactile (HPT) dataset, containing 1.6M synchronized paired frames across four vision- and array-based tactile sensors. Across distinct tactile perception and real-world manipulation tasks, HTT is shown to learn transferable representations that adapt to new tasks and previously unseen sensors. Dataset, code, and model checkpoints will be released upon publication at https://jxbi1010.github.io/htt-gh-page/.