Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling
This work addresses the problem of inefficient resource allocation in AI development for researchers and practitioners, suggesting a more targeted approach to data acquisition.
The paper argues that not all AI tasks benefit equally from data scaling, proposing that the compositional and structural patterns of data should guide which tasks to prioritize for scaling and inform new compute paradigms for tasks where scaling is inefficient or insufficient.
While Large Language Models require more and more data to train and scale, rather than looking for any data to acquire, we should consider what types of tasks are more likely to benefit from data scaling. We should be intentional in our data acquisition. We argue that the shape of the data itself, such as its compositional and structural patterns, informs which tasks to prioritize in data scaling, and shapes the development of the next generation of compute paradigms for tasks where data scaling is inefficient, or even insufficient.