*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task
This work addresses the challenge of compositional generalization for machine learning systems, particularly Transformers, by providing a new benchmark and analysis of their scalability limitations.
The authors introduce *-CFQ, a dataset suite for analyzing machine learning scalability on compositional tasks. They found that Transformers struggle with compositional generalization at all training sizes, and increased natural language scope leads to higher error rates, only partially mitigated by more training data.
We present *-CFQ ("star-CFQ"): a suite of large-scale datasets of varying scope based on the CFQ semantic parsing benchmark, designed for principled investigation of the scalability of machine learning systems in a realistic compositional task setting. Using this suite, we conduct a series of experiments investigating the ability of Transformers to benefit from increased training size under conditions of fixed computational cost. We show that compositional generalization remains a challenge at all training sizes, and we show that increasing the scope of natural language leads to consistently higher error rates, which are only partially offset by increased training data. We further show that while additional training data from a related domain improves the accuracy in data-starved situations, this improvement is limited and diminishes as the distance from the related domain to the target domain increases.