Profit Mirage: Revisiting Information Leakage in LLM-based Financial Agents
This addresses a critical reliability issue for financial practitioners using LLM-based trading systems, though it appears incremental as it builds on existing techniques like counterfactual analysis and Monte Carlo Tree Search.
The paper tackles the problem of information leakage in LLM-based financial agents, which causes inflated back-tested returns that vanish post-training, and introduces FactFin, a framework using counterfactual perturbations to improve causal learning, resulting in superior out-of-sample generalization and risk-adjusted performance.
LLM-based financial agents have attracted widespread excitement for their ability to trade like human experts. However, most systems exhibit a "profit mirage": dazzling back-tested returns evaporate once the model's knowledge window ends, because of the inherent information leakage in LLMs. In this paper, we systematically quantify this leakage issue across four dimensions and release FinLake-Bench, a leakage-robust evaluation benchmark. Furthermore, to mitigate this issue, we introduce FactFin, a framework that applies counterfactual perturbations to compel LLM-based agents to learn causal drivers instead of memorized outcomes. FactFin integrates four core components: Strategy Code Generator, Retrieval-Augmented Generation, Monte Carlo Tree Search, and Counterfactual Simulator. Extensive experiments show that our method surpasses all baselines in out-of-sample generalization, delivering superior risk-adjusted performance.