Aspects of importance sampling in parameter selection for neural networks using ridgelet transform
This work addresses the challenge of parameter initialization in neural networks, offering a method to potentially improve training efficiency, though it appears incremental as it builds on existing ridgelet transform concepts.
The study tackled the problem of selecting initial parameters for neural networks by using an oracle distribution derived from the ridgelet transform, which connects parameter distributions to integral representations of target functions and allows network construction via linear regression instead of backpropagation in simple cases. The results indicated that the magnitude of weight parameters might be more critical than intercept parameters, as demonstrated through one-dimensional and high-dimensional examples.
The choice of parameters in neural networks is crucial in the performance, and an oracle distribution derived from the ridgelet transform enables us to obtain suitable initial parameters. In other words, the distribution of parameters is connected to the integral representation of target functions. The oracle distribution allows us to avoid the conventional backpropagation learning process; only a linear regression is enough to construct the neural network in simple cases. This study provides a new look at the oracle distributions and ridgelet transforms, i.e., an aspect of importance sampling. In addition, we propose extensions of the parameter sampling methods. We demonstrate the aspect of importance sampling and the proposed sampling algorithms via one-dimensional and high-dimensional examples; the results imply that the magnitude of weight parameters could be more crucial than the intercept parameters.