Evaluating structure learning algorithms with a balanced scoring function
This addresses a methodological issue for researchers in machine learning and statistics by providing an unbiased assessment tool, though it is incremental as it builds on existing metrics.
The paper tackles the problem of biased evaluation metrics for structure learning algorithms in causal or Bayesian Network graphs, proposing a Balanced Scoring Function (BSF) that adjusts rewards based on edge occurrence rates to eliminate bias towards underfitted graphs.
Several structure learning algorithms have been proposed towards discovering causal or Bayesian Network (BN) graphs. The validity of these algorithms tends to be evaluated by assessing the relationship between the learnt and the ground truth graph. However, there is no agreed scoring metric to determine this relationship. Moreover, this paper shows that some of the commonly used metrics tend to be biased in favour of graphs that minimise edges. While graphs that are less complex are desirable, some of the metrics favour underfitted graphs, thereby encouraging limited propagation of evidence. This paper proposes the Balanced Scoring Function (BSF) that eliminates this bias by adjusting the reward function based on the difficulty of discovering an edge, or no edge, proportional to their occurrence rate in the ground truth graph. The BSF score can be used in conjunction with other traditional metrics to provide an alternative and unbiased assessment about the capability of a structure learning algorithm in discovering causal or BN graphs.