Evaluating and Characterizing Incremental Learning from Non-Stationary Data
This provides a common evaluation basis for researchers in incremental learning, addressing a lack of standardized testing practices, though it is incremental as it builds on existing algorithms.
The paper tackles the problem of evaluating incremental learning algorithms on non-stationary data by proposing a testbed with synthetic datasets, and it demonstrates effectiveness by characterizing the strengths and weaknesses of well-known algorithms.
Incremental learning from non-stationary data poses special challenges to the field of machine learning. Although new algorithms have been developed for this, assessment of results and comparison of behaviors are still open problems, mainly because evaluation metrics, adapted from more traditional tasks, can be ineffective in this context. Overall, there is a lack of common testing practices. This paper thus presents a testbed for incremental non-stationary learning algorithms, based on specially designed synthetic datasets. Also, test results are reported for some well-known algorithms to show that the proposed methodology is effective at characterizing their strengths and weaknesses. It is expected that this methodology will provide a common basis for evaluating future contributions in the field.