Has the Deep Neural Network learned the Stochastic Process? An Evaluation Viewpoint
This provides a new evaluation perspective for DNNs modeling complex stochastic systems, addressing a gap in assessing model fidelity to underlying processes, though it is incremental in focusing on evaluation rather than model development.
The paper tackles the problem that traditional evaluation methods for deep neural networks (DNNs) forecasting stochastic systems fail to measure learning of the underlying stochastic process, and it proposes a new criterion called Fidelity to Stochastic Process (F2SP), showing empirically that Expected Calibration Error (ECE) uniquely captures this on synthetic and real-world datasets like wildfire data.
This paper presents the first systematic study of evaluating Deep Neural Networks (DNNs) designed to forecast the evolution of stochastic complex systems. We show that traditional evaluation methods like threshold-based classification metrics and error-based scoring rules assess a DNN's ability to replicate the observed ground truth but fail to measure the DNN's learning of the underlying stochastic process. To address this gap, we propose a new evaluation criterion called Fidelity to Stochastic Process (F2SP), representing the DNN's ability to predict the system property Statistic-GT--the ground truth of the stochastic process--and introduce an evaluation metric that exclusively assesses F2SP. We formalize F2SP within a stochastic framework and establish criteria for validly measuring it. We formally show that Expected Calibration Error (ECE) satisfies the necessary condition for testing F2SP, unlike traditional evaluation methods. Empirical experiments on synthetic datasets, including wildfire, host-pathogen, and stock market models, demonstrate that ECE uniquely captures F2SP. We further extend our study to real-world wildfire data, highlighting the limitations of conventional evaluation and discuss the practical utility of incorporating F2SP into model assessment. This work offers a new perspective on evaluating DNNs modeling complex systems by emphasizing the importance of capturing the underlying stochastic process.