It's Not What Machines Can Learn, It's What We Cannot Teach
This work addresses a foundational limitation in machine learning for practitioners, revealing that unbiased training data generation is infeasible for hard problems, making it incremental by challenging existing empirical approaches.
The paper tackles the problem of whether deep neural networks can learn to solve computationally hard tasks, proving that under standard complexity assumptions, polynomial-time sample generators for NP-hard problems only sample from easier sub-problems, and empirically showing that biased data generation leads to overestimated model accuracy.
Can deep neural networks learn to solve any task, and in particular problems of high complexity? This question attracts a lot of interest, with recent works tackling computationally hard tasks such as the traveling salesman problem and satisfiability. In this work we offer a different perspective on this question. Given the common assumption that $\textit{NP} \neq \textit{coNP}$ we prove that any polynomial-time sample generator for an $\textit{NP}$-hard problem samples, in fact, from an easier sub-problem. We empirically explore a case study, Conjunctive Query Containment, and show how common data generation techniques generate biased datasets that lead practitioners to over-estimate model accuracy. Our results suggest that machine learning approaches that require training on a dense uniform sampling from the target distribution cannot be used to solve computationally hard problems, the reason being the difficulty of generating sufficiently large and unbiased training sets.