Rebalancing gradient to improve self-supervised co-training of depth, odometry and optical flow predictions
This work addresses the problem of unbalanced learning in multi-task self-supervised learning for motion-aware predictions, offering an incremental improvement over existing co-training methods.
CoopNet dynamically rebalances gradient apportionment among co-trained networks (depth+odometry and optical flow) to ensure equitable learning, using a hybrid loss based on photo-metric reconstruction error disagreement. It achieves state-of-the-art or comparable results on KITTI and CityScapes for depth, odometry, and optical flow predictions.
We present CoopNet, an approach that improves the cooperation of co-trained networks by dynamically adapting the apportionment of gradient, to ensure equitable learning progress. It is applied to motion-aware self-supervised prediction of depth maps, by introducing a new hybrid loss, based on a distribution model of photo-metric reconstruction errors made by, on the one hand the depth + odometry paired networks, and on the other hand the optical flow network. This model essentially assumes that the pixels from moving objects (that must be discarded for training depth and odometry), correspond to those where the two reconstructions strongly disagree. We justify this model by theoretical considerations and experimental evidences. A comparative evaluation on KITTI and CityScapes datasets shows that CoopNet improves or is comparable to the state-of-the-art in depth, odometry and optical flow predictions.