Bimodal SegNet: Instance Segmentation Fusing Events and RGB Frames for Robotic Grasping
This work addresses challenges like occlusion and motion blur for robotic grasping, representing an incremental advance in multimodal segmentation.
The paper tackles object segmentation for robotic grasping under dynamic conditions by fusing event-based and RGB frame data, achieving a 6-10% improvement in segmentation accuracy over state-of-the-art methods.
Object segmentation for robotic grasping under dynamic conditions often faces challenges such as occlusion, low light conditions, motion blur and object size variance. To address these challenges, we propose a Deep Learning network that fuses two types of visual signals, event-based data and RGB frame data. The proposed Bimodal SegNet network has two distinct encoders, one for each signal input and a spatial pyramidal pooling with atrous convolutions. Encoders capture rich contextual information by pooling the concatenated features at different resolutions while the decoder obtains sharp object boundaries. The evaluation of the proposed method undertakes five unique image degradation challenges including occlusion, blur, brightness, trajectory and scale variance on the Event-based Segmentation (ESD) Dataset. The evaluation results show a 6-10\% segmentation accuracy improvement over state-of-the-art methods in terms of mean intersection over the union and pixel accuracy. The model code is available at https://github.com/sanket0707/Bimodal-SegNet.git