Whispering to a Blackbox: Bootstrapping Frozen OCR with Visual Prompts
This work addresses the problem of adapting frozen pre-trained models to specific tasks with mismatched data distributions, offering a sample-efficient solution for improving OCR performance without modifying the underlying model weights.
The Whisperer framework improves the performance of frozen Optical Character Recognition (OCR) models by learning diffusion-based preprocessors that adapt input images in pixel space. This method achieved an 8% absolute (10.6% relative) reduction in Character Error Rate (CER) on a dataset of 300k degraded synthetic text images.
In the landscape of modern machine learning, frozen pre-trained models provide stability and efficiency but often underperform on specific tasks due to mismatched data distributions. This paper introduces the Whisperer, a novel visual prompting framework that learns diffusion-based preprocessors to adapt inputs in pixel space, effectively "whispering" enhancements to frozen downstream models like EasyOCR. By framing the process as behavioral cloning of stochastically discovered improvement policies, our method achieves an 8% absolute (10.6% relative) reduction in Character Error Rate (CER) on a challenging dataset of 300k degraded synthetic text images, surpassing hand-engineered baselines such as CLAHE. The key innovation is a four-stage training curriculum that uses behavioral cloning to amplify "lucky" improvements discovered through the stochastic exploration of a partially trained diffusion model. This approach is highly sample-efficient and avoids the pitfalls of traditional reinforcement learning. Crucially, we frame this not as naive reinforcement learning, but as behavioral cloning of an exploration policy: we stochastically sample intermediate diffusion outputs, select those that improve CER by chance, and then train the model to reproduce them. This bootstrapping curriculum (4 stages over 60 GPU-hours) amplifies random successes into a systematic strategy. In summary, by whispering to the frozen OCR through its inputs, we improve an imperfect classifier without touching its weights.