Multimodal Material Classification for Robots using Spectroscopy and High Resolution Texture Imaging
This work addresses the problem of enabling robots to better interact with and manipulate objects by accurately recognizing materials, though it appears incremental as it builds on existing multimodal methods.
The paper tackles material recognition for robots by introducing a multimodal sensing technique combining near-infrared spectroscopy and high-resolution texture imaging, resulting in improved performance over prior state-of-the-art approaches for classifying materials of household objects.
Material recognition can help inform robots about how to properly interact with and manipulate real-world objects. In this paper, we present a multimodal sensing technique, leveraging near-infrared spectroscopy and close-range high resolution texture imaging, that enables robots to estimate the materials of household objects. We release a dataset of high resolution texture images and spectral measurements collected from a mobile manipulator that interacted with 144 household objects. We then present a neural network architecture that learns a compact multimodal representation of spectral measurements and texture images. When generalizing material classification to new objects, we show that this multimodal representation enables a robot to recognize materials with greater performance as compared to prior state-of-the-art approaches. Finally, we present how a robot can combine this high resolution local sensing with images from the robot's head-mounted camera to achieve accurate material classification over a scene of objects on a table.