Positive Distribution Shift as a Framework for Understanding Tractable Learning
This work addresses a foundational issue in machine learning by reframing distribution shift as a tool for computational tractability, offering a new perspective for researchers and practitioners dealing with hard learning problems.
The paper tackles the problem of learning a target function under covariate shift by proposing Positive Distribution Shift (PDS), where a well-chosen training distribution makes learning easier, often computationally, and shows that certain hard classes become tractable under PDS.
We study a setting where the goal is to learn a target function f(x) with respect to a target distribution D(x), but training is done on i.i.d. samples from a different training distribution D'(x), labeled by the true target f(x). Such a distribution shift (here in the form of covariate shift) is usually viewed negatively, as hurting or making learning harder, and the traditional distribution shift literature is mostly concerned with limiting or avoiding this negative effect. In contrast, we argue that with a well-chosen D'(x), the shift can be positive and make learning easier -- a perspective called Positive Distribution Shift (PDS). Such a perspective is central to contemporary machine learning, where much of the innovation is in finding good training distributions D'(x), rather than changing the training algorithm. We further argue that the benefit is often computational rather than statistical, and that PDS allows computationally hard problems to become tractable even using standard gradient-based training. We formalize different variants of PDS, show how certain hard classes are easily learnable under PDS, and make connections with membership query learning.