Reasoning-Intensive Regression
This addresses a problem for AI researchers and practitioners working on tasks like rubric-based scoring or domain-specific retrieval where deep text analysis is needed with limited data and computation, though it is incremental as it builds on existing LLM and ensemble techniques.
The paper tackles reasoning-intensive regression (RiR), which involves deducing subtle numerical properties from text, by establishing a benchmark and showing that standard methods like prompting frozen LLMs and finetuning Transformer encoders struggle. It proposes MENTAT, a method combining batch-reflective prompt optimization with neural ensemble learning, achieving up to 65% improvement over baselines.
AI researchers and practitioners increasingly apply large language models (LLMs) to what we call reasoning-intensive regression (RiR), i.e. deducing subtle numerical properties from text. Unlike standard language regression tasks, e.g. for sentiment or similarity, RiR often appears instead in ad-hoc problems like rubric-based scoring or domain-specific retrieval, where much deeper analysis of text is required while only limited task-specific training data and computation are available. We cast three realistic problems as RiR tasks to establish an initial benchmark, and use that to test our hypothesis that prompting frozen LLMs and finetuning Transformer encoders via gradient descent will both often struggle in RiR. We then propose MENTAT, a simple and lightweight method that combines batch-reflective prompt optimization with neural ensemble learning. MENTAT achieves up to 65% improvement over both baselines, though substantial room remains for future advances in RiR.