On minimizing the training set fill distance in machine learning regression
This addresses computational and cost limitations in machine learning regression by improving data selection efficiency, though it is incremental as it builds on existing sampling approaches.
The paper tackles the problem of selecting small training sets from large unlabeled data pools for regression tasks by studying Farthest Point Sampling (FPS) to minimize fill distance, deriving an error bound that depends linearly on this distance, and empirically showing it significantly reduces maximum prediction error and outperforms other sampling methods.
For regression tasks one often leverages large datasets for training predictive machine learning models. However, using large datasets may not be feasible due to computational limitations or high data labelling costs. Therefore, suitably selecting small training sets from large pools of unlabelled data points is essential to maximize model performance while maintaining efficiency. In this work, we study Farthest Point Sampling (FPS), a data selection approach that aims to minimize the fill distance of the selected set. We derive an upper bound for the maximum expected prediction error, conditional to the location of the unlabelled data points, that linearly depends on the training set fill distance. For empirical validation, we perform experiments using two regression models on three datasets. We empirically show that selecting a training set by aiming to minimize the fill distance, thereby minimizing our derived bound, significantly reduces the maximum prediction error of various regression models, outperforming alternative sampling approaches by a large margin. Furthermore, we show that selecting training sets with the FPS can also increase model stability for the specific case of Gaussian kernel regression approaches.