Adaptive Kernel Density Estimation with Pre-training
This work addresses the challenge of high-dimensional density estimation for statisticians and machine learning practitioners, but the benefit is conditional on distribution similarity, making it an incremental improvement.
The paper introduces a pre-training strategy for kernel density estimation that uses a neural network to recommend location-adaptive kernels, improving accuracy in high dimensions when the target distribution is close to the pre-training distribution family. Numerical experiments show effectiveness, with fine-tuning reactivating benefits for dissimilar distributions.
Density estimation in high-dimensional settings is an important and challenging statistical problem.Traditional methods based on kernel smoothing are inefficient in high dimensions due to the difficulties in specifying appropriate location-adaptive kernels. In this work, we introduce pre-training, a key idea behind many cutting-edge AI technologies, to the context of non-parametric density estimation. By establishing a pre-trained neural network that can recommend an appropriate location-adaptive kernel for each sample point, efficient density estimation with adaptive kernels is achieved in high dimensions. A wide range of numerical experiments show that this strategy is highly effective for improving density-estimation accuracy, when the target distribution is close to the distribution family for pre-training. When the target distribution is substantially different from the pre-training distribution family, the benefit from the proposed pre-training strategy may be diluted, but can be reactivated by an additional fine-tuning procedure.