Physics-informed Guided Disentanglement in Generative Networks
This addresses the issue of poor translation quality and controllability in image-to-image networks for applications involving physics-affected images, though it appears incremental as it builds on existing disentanglement concepts.
The paper tackles the problem of image-to-image translation networks suffering from entanglement effects due to physics-related phenomena like occlusions and fog, proposing a framework that uses physics models or neural networks to guide disentanglement, resulting in dramatically increased performance in challenging scenarios.
Image-to-image translation (i2i) networks suffer from entanglement effects in presence of physics-related phenomena in target domain (such as occlusions, fog, etc), lowering altogether the translation quality, controllability and variability. In this paper, we propose a general framework to disentangle visual traits in target images. Primarily, we build upon collection of simple physics models, guiding the disentanglement with a physical model that renders some of the target traits, and learning the remaining ones. Because physics allows explicit and interpretable outputs, our physical models (optimally regressed on target) allows generating unseen scenarios in a controllable manner. Secondarily, we show the versatility of our framework to neural-guided disentanglement where a generative network is used in place of a physical model in case the latter is not directly accessible. Altogether, we introduce three strategies of disentanglement being guided from either a fully differentiable physics model, a (partially) non-differentiable physics model, or a neural network. The results show our disentanglement strategies dramatically increase performances qualitatively and quantitatively in several challenging scenarios for image translation.