Cross-modal Adversarial Reprogramming
This work addresses the challenge of cross-modal task adaptation for researchers and practitioners, offering a novel approach to leverage existing models, though it is incremental in extending adversarial reprogramming to new domains.
The paper tackles the problem of repurposing pre-trained image classification models for tasks in different data modalities, specifically NLP and sequence classification, by designing an adversarial program that maps discrete tokens into images. The result is competitive performance on various text and sequence benchmarks without retraining the network.
With the abundance of large-scale deep learning models, it has become possible to repurpose pre-trained networks for new tasks. Recent works on adversarial reprogramming have shown that it is possible to repurpose neural networks for alternate tasks without modifying the network architecture or parameters. However these works only consider original and target tasks within the same data domain. In this work, we broaden the scope of adversarial reprogramming beyond the data modality of the original task. We analyze the feasibility of adversarially repurposing image classification neural networks for Natural Language Processing (NLP) and other sequence classification tasks. We design an efficient adversarial program that maps a sequence of discrete tokens into an image which can be classified to the desired class by an image classification model. We demonstrate that by using highly efficient adversarial programs, we can reprogram image classifiers to achieve competitive performance on a variety of text and sequence classification benchmarks without retraining the network.