Hallucinating Beyond Observation: Learning to Complete with Partial Observation and Unpaired Prior Knowledge
This addresses data completion challenges in computer vision, offering a more efficient training method, though it is incremental as it builds on existing encoder-decoder networks.
The paper tackles the problem of completing partially observed data, such as 2-D road layouts and 3-D vehicle shapes, by proposing a single-step training strategy that uses unpaired prior knowledge, achieving improvements like +12.2% F-measure on Cityscapes and +23.8% on KITTI.
We propose a novel single-step training strategy that allows convolutional encoder-decoder networks that use skip connections, to complete partially observed data by means of hallucination. This strategy is demonstrated for the task of completing 2-D road layouts as well as 3-D vehicle shapes. As input, it takes data from a partially observed domain, for which no ground truth is available, and data from an unpaired prior knowledge domain and trains the network in an end-to-end manner. Our single-step training strategy is compared against two state-of-the-art baselines, one using a two-step auto-encoder training strategy and one using an adversarial strategy. Our novel strategy achieves an improvement up to +12.2% F-measure on the Cityscapes dataset. The learned network intrinsically generalizes better than the baselines on unseen datasets, which is demonstrated by an improvement up to +23.8% F-measure on the unseen KITTI dataset. Moreover, our approach outperforms the baselines using the same backbone network on the 3-D shape completion benchmark by a margin of 0.006 Hamming distance.