Relevance As a Metric for Evaluating Machine Learning Algorithms
This work addresses the need for domain-specific evaluation metrics in machine learning, though it appears incremental as it focuses on a specific application class.
The authors tackled the problem of evaluating machine learning algorithms by proposing a novel probability-based performance metric called Relevance Score, which they found to be more appropriate than Classification Accuracy for a certain class of applications through empirical analysis on a dataset from an intelligent lighting pilot installation.
In machine learning, the choice of a learning algorithm that is suitable for the application domain is critical. The performance metric used to compare different algorithms must also reflect the concerns of users in the application domain under consideration. In this work, we propose a novel probability-based performance metric called Relevance Score for evaluating supervised learning algorithms. We evaluate the proposed metric through empirical analysis on a dataset gathered from an intelligent lighting pilot installation. In comparison to the commonly used Classification Accuracy metric, the Relevance Score proves to be more appropriate for a certain class of applications.