An Algorithm for Learning Smaller Representations of Models With Scarce Data
This work addresses the problem of limited data availability for machine learning practitioners, offering a formal theoretical foundation for data expansion techniques, though it appears incremental as it builds on existing hyperparameter and pruning methods.
The paper tackles binary classification with scarce, unrepresentative data by introducing an algorithm that reconstructs the underlying distribution's manifold up to homology, combining hyperparameter search, pruning, and data generation. It achieves up to 2(1 - 2^{-sizeΘ}) times better performance than exhaustive search under ideal conditions and extends to deep neural networks.
We present an algorithm for solving binary classification problems when the dataset is not fully representative of the problem being solved, and obtaining more data is not possible. It relies on a trained model with loose accuracy constraints, an iterative hyperparameter searching-and-pruning procedure over a search space $Θ$, and a data-generating function. Our algorithm works by reconstructing up to homology the manifold on which lies the support of the underlying distribution. We provide an analysis on correctness and runtime complexity under ideal conditions and an extension to deep neural networks. In the former case, if $\sizeΘ$ is the number of hyperparameter sets in the search space, this algorithm returns a solution that is up to $2(1 - {2^{-\sizeΘ}})$ times better than simply training with an enumeration of $Θ$ and picking the best model. As part of our analysis we also prove that an open cover of a dataset has the same homology as the manifold on which lies the support of the underlying probability distribution, if and only said dataset is learnable. This latter result acts as a formal argument to explain the effectiveness of data expansion techniques.