BoTTA: Benchmarking on-device Test Time Adaptation
This work addresses the underexplored challenge of adapting models in resource-constrained environments like mobile and edge devices, providing a benchmark for real-world deployments, though it is incremental as it builds on existing TTA research by focusing on specific constraints.
The paper tackles the problem of test-time adaptation (TTA) for deep learning models on mobile and edge devices by proposing BoTTA, a benchmark that evaluates TTA methods under practical constraints like limited samples and diverse distribution shifts, revealing that many algorithms struggle with small datasets and unseen categories while reporting device-specific resource use, such as SHOT improving accuracy by 2.25x with 512 adaptation samples but using 1.08x peak memory on Raspberry Pi.
The performance of deep learning models depends heavily on test samples at runtime, and shifts from the training data distribution can significantly reduce accuracy. Test-time adaptation (TTA) addresses this by adapting models during inference without requiring labeled test data or access to the original training set. While research has explored TTA from various perspectives like algorithmic complexity, data and class distribution shifts, model architectures, and offline versus continuous learning, constraints specific to mobile and edge devices remain underexplored. We propose BoTTA, a benchmark designed to evaluate TTA methods under practical constraints on mobile and edge devices. Our evaluation targets four key challenges caused by limited resources and usage conditions: (i) limited test samples, (ii) limited exposure to categories, (iii) diverse distribution shifts, and (iv) overlapping shifts within a sample. We assess state-of-the-art TTA methods under these scenarios using benchmark datasets and report system-level metrics on a real testbed. Furthermore, unlike prior work, we align with on-device requirements by advocating periodic adaptation instead of continuous inference-time adaptation. Experiments reveal key insights: many recent TTA algorithms struggle with small datasets, fail to generalize to unseen categories, and depend on the diversity and complexity of distribution shifts. BoTTA also reports device-specific resource use. For example, while SHOT improves accuracy by $2.25\times$ with $512$ adaptation samples, it uses $1.08\times$ peak memory on Raspberry Pi versus the base model. BoTTA offers actionable guidance for TTA in real-world, resource-constrained deployments.