Subspace Optimization for Backpropagation-Free Continual Test-Time Adaptation
This addresses the challenge of efficient and effective continual adaptation for machine learning models deployed in dynamic environments, representing an incremental improvement over existing backpropagation-free methods.
The paper tackles the problem of balancing runtime efficiency with learning capacity in backpropagation-free continual test-time adaptation by introducing PACE, which optimizes normalization layer parameters using a subspace optimization approach. The framework achieves state-of-the-art accuracy across multiple benchmarks under continual distribution shifts while reducing runtime by over 50% compared to existing methods.
We introduce PACE, a backpropagation-free continual test-time adaptation system that directly optimizes the affine parameters of normalization layers. Existing derivative-free approaches struggle to balance runtime efficiency with learning capacity, as they either restrict updates to input prompts or require continuous, resource-intensive adaptation regardless of domain stability. To address these limitations, PACE leverages the Covariance Matrix Adaptation Evolution Strategy with the Fastfood projection to optimize high-dimensional affine parameters within a low-dimensional subspace, leading to superior adaptive performance. Furthermore, we enhance the runtime efficiency by incorporating an adaptation stopping criterion and a domain-specialized vector bank to eliminate redundant computation. Our framework achieves state-of-the-art accuracy across multiple benchmarks under continual distribution shifts, reducing runtime by over 50% compared to existing backpropagation-free methods.