History Is Not Enough: An Adaptive Dataflow System for Financial Time-Series Synthesis
This work addresses the critical obstacle of poor generalization in quantitative finance due to non-stationary data, offering an adaptive solution for building more reliable data-driven systems in dynamic markets.
The paper tackles the problem of concept drift and distributional non-stationarity in financial time-series data, which causes models trained on static historical data to overfit and perform poorly in dynamic markets. The result is a drift-aware dataflow system that enhances model robustness and improves risk-adjusted returns, as demonstrated in forecasting and reinforcement learning trading tasks.
In quantitative finance, the gap between training and real-world performance-driven by concept drift and distributional non-stationarity-remains a critical obstacle for building reliable data-driven systems. Models trained on static historical data often overfit, resulting in poor generalization in dynamic markets. The mantra "History Is Not Enough" underscores the need for adaptive data generation that learns to evolve with the market rather than relying solely on past observations. We present a drift-aware dataflow system that integrates machine learning-based adaptive control into the data curation process. The system couples a parameterized data manipulation module comprising single-stock transformations, multi-stock mix-ups, and curation operations, with an adaptive planner-scheduler that employs gradient-based bi-level optimization to control the system. This design unifies data augmentation, curriculum learning, and data workflow management under a single differentiable framework, enabling provenance-aware replay and continuous data quality monitoring. Extensive experiments on forecasting and reinforcement learning trading tasks demonstrate that our framework enhances model robustness and improves risk-adjusted returns. The system provides a generalizable approach to adaptive data management and learning-guided workflow automation for financial data.