Margin-Based Transfer Bounds for Meta Learning with Deep Feature Embedding
This work provides theoretical generalization bounds for meta-learning models, which is a significant contribution for researchers and practitioners seeking to understand the generalization ability of these models.
This paper establishes three margin-based transfer bounds for meta-learning based multiclass classification (MLMC), demonstrating that the expected error on a future task can be estimated using the average empirical error from previous tasks. They show that models trained with a multi-margin loss achieve competitive performance on three benchmarks, validating the practical relevance of their theoretical analysis.
By transferring knowledge learned from seen/previous tasks, meta learning aims to generalize well to unseen/future tasks. Existing meta-learning approaches have shown promising empirical performance on various multiclass classification problems, but few provide theoretical analysis on the classifiers' generalization ability on future tasks. In this paper, under the assumption that all classification tasks are sampled from the same meta-distribution, we leverage margin theory and statistical learning theory to establish three margin-based transfer bounds for meta-learning based multiclass classification (MLMC). These bounds reveal that the expected error of a given classification algorithm for a future task can be estimated with the average empirical error on a finite number of previous tasks, uniformly over a class of preprocessing feature maps/deep neural networks (i.e. deep feature embeddings). To validate these bounds, instead of the commonly-used cross-entropy loss, a multi-margin loss is employed to train a number of representative MLMC models. Experiments on three benchmarks show that these margin-based models still achieve competitive performance, validating the practical value of our margin-based theoretical analysis.