Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors
This work is significant for robotics, particularly in applications requiring robust performance in environments where sensor corruption is a risk, by providing a method for detecting and compensating for faulty sensors.
This paper introduces the Crossmodal Compensation Model (CCM) to address the problem of corrupted sensor modalities in robots. CCM detects corrupted sensors using unimodal reconstruction loss and then compensates for the lost information using data from the remaining sensors, enabling robust contact-rich manipulation policies even with unseen corruption types.
Using sensor data from multiple modalities presents an opportunity to encode redundant and complementary features that can be useful when one modality is corrupted or noisy. Humans do this everyday, relying on touch and proprioceptive feedback in visually-challenging environments. However, robots might not always know when their sensors are corrupted, as even broken sensors can return valid values. In this work, we introduce the Crossmodal Compensation Model (CCM), which can detect corrupted sensor modalities and compensate for them. CMM is a representation model learned with self-supervision that leverages unimodal reconstruction loss for corruption detection. CCM then discards the corrupted modality and compensates for it with information from the remaining sensors. We show that CCM learns rich state representations that can be used for contact-rich manipulation policies, even when input modalities are corrupted in ways not seen during training time.