Meta-learning for mixed linear regression
This addresses the challenge of learning from many small-data tasks in fields like medical imaging and robotics, providing theoretical guarantees for when such meta-learning is feasible, though it is incremental as it builds on existing mixed linear regression frameworks.
The paper tackles the problem of meta-learning with many tasks that each have only small amounts of labeled data, focusing on a scenario where each task is drawn from a mixture of k linear regressions. It shows that abundant small-data tasks can compensate for the lack of big-data tasks, with the total number of examples scaling similarly, using a novel spectral approach that requires about Ω(k^{3/2}) medium-data tasks each with Ω(k^{1/2}) examples.
In modern supervised learning, there are a large number of tasks, but many of them are associated with only a small amount of labeled data. These include data from medical image processing and robotic interaction. Even though each individual task cannot be meaningfully trained in isolation, one seeks to meta-learn across the tasks from past experiences by exploiting some similarities. We study a fundamental question of interest: When can abundant tasks with small data compensate for lack of tasks with big data? We focus on a canonical scenario where each task is drawn from a mixture of $k$ linear regressions, and identify sufficient conditions for such a graceful exchange to hold; The total number of examples necessary with only small data tasks scales similarly as when big data tasks are available. To this end, we introduce a novel spectral approach and show that we can efficiently utilize small data tasks with the help of $\tildeΩ(k^{3/2})$ medium data tasks each with $\tildeΩ(k^{1/2})$ examples.