A partition-based similarity for classification distributions
This work provides a principled similarity measure for classification distributions, aiding machine learning practitioners in tasks like transfer learning, but it appears incremental as it builds on existing concepts of optimal representations and decision rules.
The authors tackled the problem of measuring similarity between classification distributions by proposing a novel 'task similarity' measure, which quantifies how well an optimal representation for a source distribution performs on a target distribution, and demonstrated in simulations that it correlates with transfer efficiency and semantic similarity.
Herein we define a measure of similarity between classification distributions that is both principled from the perspective of statistical pattern recognition and useful from the perspective of machine learning practitioners. In particular, we propose a novel similarity on classification distributions, dubbed task similarity, that quantifies how an optimally-transformed optimal representation for a source distribution performs when applied to inference related to a target distribution. The definition of task similarity allows for natural definitions of adversarial and orthogonal distributions. We highlight limiting properties of representations induced by (universally) consistent decision rules and demonstrate in simulation that an empirical estimate of task similarity is a function of the decision rule deployed for inference. We demonstrate that for a given target distribution, both transfer efficiency and semantic similarity of candidate source distributions correlate with empirical task similarity.