From Residuals to Reasons: LLM-Guided Mechanism Inference from Tabular Data
For scientists using machine learning on tabular data, MARICL provides a method to generate interpretable corrections that reveal underlying mechanisms, validated by generalization across experimental conditions.
MARICL uses LLM agents to analyze residuals from a base model, hypothesize missing structure, and produce explicit correction terms via multi-turn textual gradient optimization, improving predictions over the base model on all nine benchmarks. Frozen formulas from one experimental batch generalize to held-out batches within the same protocol (92% improvement) but fail across protocols, indicating mechanistic generalization.
A persistent challenge in machine learning for scientific applications is jointly achieving prediction and understanding. Statistical models excel on structured data but operate as black boxes, while existing interpretability methods are largely inspective: they answer "which features matter?" but do not articulate how features interact or refine explanations iteratively alongside human understanding. Asking an LLM to predict the target directly forces it to search the entire output space; we instead anchor predictions with a base model and ask the LLM the narrower question of what that model is missing. We introduce Multi-Agent Residual In-Context Learning (MARICL), an agentic framework in which LLM agents analyze where a base-model fails, hypothesize missing structure from high-residual examples provided in context, and produce explicit correction terms refined through multi-turn textual gradient optimization. Across nine benchmarks spanning scientific, biomedical, socioeconomic, and synthetic settings, MARICL improves consistently over its base model on all datasets. To test whether these corrections reflect real structure or batch-specific noise, we freeze formulas learned on one experimental batch of the Cell-Free Protein dataset and apply them (with no retraining and no further LLM calls) to held-out batches. Within the same reagent protocol, the frozen formulas improve predictions in over 92% of cases; across a different protocol, they fail systematically. The success boundary aligns with the biochemistry, not the batch count; direct evidence of mechanistic generalization.