G2L: A Geometric Approach for Generating Pseudo-labels that Improve Transfer Learning
This addresses the challenge of expensive and limited human annotations in deep learning, offering an incremental improvement in pseudo-label generation for transfer learning.
The paper tackles the problem of generating pseudo-labels for transfer learning when human-annotated labels are limited, using a geometric approach based on the Cayley-Menger determinant, resulting in models with similar or better transferability than those trained on ImageNet1K labels, with an overall error decrease of 0.43%.
Transfer learning is a deep-learning technique that ameliorates the problem of learning when human-annotated labels are expensive and limited. In place of such labels, it uses instead the previously trained weights from a well-chosen source model as the initial weights for the training of a base model for a new target dataset. We demonstrate a novel but general technique for automatically creating such source models. We generate pseudo-labels according to an efficient and extensible algorithm that is based on a classical result from the geometry of high dimensions, the Cayley-Menger determinant. This G2L (``geometry to label'') method incrementally builds up pseudo-labels using a greedy computation of hypervolume content. We demonstrate that the method is tunable with respect to expected accuracy, which can be forecast by an information-theoretic measure of dataset similarity (divergence) between source and target. The results of 280 experiments show that this mechanical technique generates base models that have similar or better transferability compared to a baseline of models trained on extensively human-annotated ImageNet1K labels, yielding an overall error decrease of 0.43\%, and an error decrease in 4 out of 5 divergent datasets tested.