Adaptive Estimation and Learning under Temporal Distribution Shift
This work addresses the challenge of adaptive learning in non-stationary environments, offering incremental improvements by generalizing existing methods and connecting to classical signal processing problems.
The paper tackles the problem of estimating a time-varying groundtruth sequence under temporal distribution shift, showing that a wavelet soft-thresholding estimator achieves optimal error bounds without prior knowledge of the shift level. It validates these findings with numerical experiments and applies the estimator to derive sparsity-aware risk bounds for binary classification and efficient training objectives.
In this paper, we study the problem of estimation and learning under temporal distribution shift. Consider an observation sequence of length $n$, which is a noisy realization of a time-varying groundtruth sequence. Our focus is to develop methods to estimate the groundtruth at the final time-step while providing sharp point-wise estimation error rates. We show that, without prior knowledge on the level of temporal shift, a wavelet soft-thresholding estimator provides an optimal estimation error bound for the groundtruth. Our proposed estimation method generalizes existing researches Mazzetto and Upfal (2023) by establishing a connection between the sequence's non-stationarity level and the sparsity in the wavelet-transformed domain. Our theoretical findings are validated by numerical experiments. Additionally, we applied the estimator to derive sparsity-aware excess risk bounds for binary classification under distribution shift and to develop computationally efficient training objectives. As a final contribution, we draw parallels between our results and the classical signal processing problem of total-variation denoising (Mammen and van de Geer,1997; Tibshirani, 2014), uncovering novel optimal algorithms for such task.