Numeracy from Literacy: Data Science as an Emergent Skill from Large Language Models
This work addresses the challenge of enabling AI models to handle numerical reasoning, which could benefit data analysis applications, though it appears incremental in exploring emergent skills from existing models.
The research tackled the problem of whether large language models can translate literacy into numeracy by testing their ability to perform data science tasks like descriptive statistics and linear regression on complex datasets, showing capabilities in grouping, correlation, and prediction.
Large language models (LLM) such as OpenAI's ChatGPT and GPT-3 offer unique testbeds for exploring the translation challenges of turning literacy into numeracy. Previous publicly-available transformer models from eighteen months prior and 1000 times smaller failed to provide basic arithmetic. The statistical analysis of four complex datasets described here combines arithmetic manipulations that cannot be memorized or encoded by simple rules. The work examines whether next-token prediction succeeds from sentence completion into the realm of actual numerical understanding. For example, the work highlights cases for descriptive statistics on in-memory datasets that the LLM initially loads from memory or generates randomly using python libraries. The resulting exploratory data analysis showcases the model's capabilities to group by or pivot categorical sums, infer feature importance, derive correlations, and predict unseen test cases using linear regression. To extend the model's testable range, the research deletes and appends random rows such that recall alone cannot explain emergent numeracy.