Dataforge: A Data Agent Platform for Autonomous Data Engineering
This addresses scalability and expertise dependence in data engineering for fields like materials discovery and climate science, though it appears incremental as it builds on existing LLM and automation techniques.
The paper tackles the labor-intensive problem of data preparation for AI applications by introducing Data Agent, a fully autonomous system for tabular data that automatically performs cleaning, routing, and feature optimization, resulting in end-to-end reliability without human supervision.
The growing demand for AI applications in fields such as materials discovery, molecular modeling, and climate science has made data preparation an important but labor-intensive step. Raw data from diverse sources must be cleaned, normalized, and transformed to become AI-ready, while effective feature transformation and selection are essential for efficient training and inference. To address the challenges of scalability and expertise dependence, we present Data Agent, a fully autonomous system specialized for tabular data. Leveraging large language model (LLM) reasoning and grounded validation, Data Agent automatically performs data cleaning, hierarchical routing, and feature-level optimization through dual feedback loops. It embodies three core principles: automatic, safe, and non-expert friendly, which ensure end-to-end reliability without human supervision. This demo showcases the first practical realization of an autonomous Data Agent, illustrating how raw data can be transformed "From Data to Better Data."