Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations
This provides a more practical and model-agnostic solution for researchers and practitioners needing to invert complex vision models, though it is incremental in improving existing inversion techniques.
The paper tackles the problem of model inversion in vision by introducing Plug-In Inversion, which uses data augmentations to avoid hard-to-tune regularizers, enabling successful inversion of Vision Transformers and MLPs on ImageNet for the first time.
Existing techniques for model inversion typically rely on hard-to-tune regularizers, such as total variation or feature regularization, which must be individually calibrated for each network in order to produce adequate images. In this work, we introduce Plug-In Inversion, which relies on a simple set of augmentations and does not require excessive hyper-parameter tuning. Under our proposed augmentation-based scheme, the same set of augmentation hyper-parameters can be used for inverting a wide range of image classification models, regardless of input dimensions or the architecture. We illustrate the practicality of our approach by inverting Vision Transformers (ViTs) and Multi-Layer Perceptrons (MLPs) trained on the ImageNet dataset, tasks which to the best of our knowledge have not been successfully accomplished by any previous works.