Sample complexity of learning Mahalanobis distance metrics
This work addresses the sample efficiency challenge in metric learning for machine learning practitioners, offering theoretical insights and practical regularization methods to enhance generalization, though it is incremental in building on existing PAC-style analysis.
The paper tackles the problem of determining sample complexity rates for supervised metric learning, providing matching lower and upper bounds that scale with representation dimension under no distributional assumptions, and shows that leveraging data structure and adding norm-based regularization can adapt to intrinsic complexity and improve generalization, with experiments validating these findings on benchmark datasets.
Metric learning seeks a transformation of the feature space that enhances prediction quality for the given task at hand. In this work we provide PAC-style sample complexity rates for supervised metric learning. We give matching lower- and upper-bounds showing that the sample complexity scales with the representation dimension when no assumptions are made about the underlying data distribution. However, by leveraging the structure of the data distribution, we show that one can achieve rates that are fine-tuned to a specific notion of intrinsic complexity for a given dataset. Our analysis reveals that augmenting the metric learning optimization criterion with a simple norm-based regularization can help adapt to a dataset's intrinsic complexity, yielding better generalization. Experiments on benchmark datasets validate our analysis and show that regularizing the metric can help discern the signal even when the data contains high amounts of noise.