Analyzing and Improving Neural Networks by Generating Semantic Counterexamples through Differentiable Rendering
This addresses robustness issues in vision models for real-world applications, but it is incremental as it builds on existing adversarial attack and rendering techniques.
The paper tackles the brittleness of deep neural networks to semantic transformations like rotations and lighting changes by using differentiable rendering to generate realistic semantic counterexamples for data augmentation, showing improved generalization, robustness, and transferability in image classification and object detection tasks.
Even as deep neural networks (DNNs) have achieved remarkable success on vision-related tasks, their performance is brittle to transformations in the input. Of particular interest are semantic transformations that model changes that have a basis in the physical world, such as rotations, translations, changes in lighting or camera pose. In this paper, we show how differentiable rendering can be utilized to generate images that are informative, yet realistic, and which can be used to analyze DNN performance and improve its robustness through data augmentation. Given a differentiable renderer and a DNN, we show how to use off-the-shelf attacks from adversarial machine learning to generate semantic counterexamples -- images where semantic features are changed as to produce misclassifications or misdetections. We validate our approach on DNNs for image classification and object detection. For classification, we show that semantic counterexamples, when used to augment the dataset, (i) improve generalization performance (ii) enhance robustness to semantic transformations, and (iii) transfer between models. Additionally, in comparison to sampling-based semantic augmentation, our technique generates more informative data in a sample efficient manner.