Unsupervised Joint Learning of Optical Flow and Intensity with Event Cameras
This work addresses the problem of efficiently recovering motion and appearance from event cameras for robotics and computer vision applications, offering a novel joint approach that improves performance over separate methods.
The paper tackles the joint estimation of optical flow and image intensity from event camera data using an unsupervised learning framework, achieving a 20% reduction in EPE and 25% reduction in AE for optical flow compared to unsupervised approaches, with competitive intensity results in high dynamic range scenarios.
Event cameras rely on motion to obtain information about scene appearance. This means that appearance and motion are inherently linked: either both are present and recorded in the event data, or neither is captured. Previous works treat the recovery of these two visual quantities as separate tasks, which does not fit with the above-mentioned nature of event cameras and overlooks the inherent relations between them. We propose an unsupervised learning framework that jointly estimates optical flow (motion) and image intensity (appearance) using a single network. From the data generation model, we newly derive the event-based photometric error as a function of optical flow and image intensity. This error is further combined with the contrast maximization framework to form a comprehensive loss function that provides proper constraints for both flow and intensity estimation. Exhaustive experiments show our method's state-of-the-art performance: in optical flow estimation, it reduces EPE by 20% and AE by 25% compared to unsupervised approaches, while delivering competitive intensity estimation results, particularly in high dynamic range scenarios. Our method also achieves shorter inference time than all other optical flow methods and many of the image reconstruction methods, while they output only one quantity. Project page: https://github.com/tub-rip/E2FAI