Principal Components for Neural Network Initialization
This work addresses a problem for researchers and practitioners in machine learning who use PCA and XAI, offering an incremental improvement in interpretability and training efficiency.
The paper tackles the issue that using PCA on data before training neural networks complicates explainable AI (XAI) interpretations, and proposes PCsInit, a method to initialize the first layer with principal components, resulting in simpler and more direct explanations while improving training performance compared to using PCA as a preprocessing step.
Principal Component Analysis (PCA) is a commonly used tool for dimension reduction and denoising. Therefore, it is also widely used on the data prior to training a neural network. However, this approach can complicate the explanation of eXplainable Artificial Intelligence (XAI) methods for the decision of the model. In this work, we analyze the potential issues with this approach and propose Principal Components-based Initialization (PCsInit), a strategy to incorporate PCA into the first layer of a neural network via initialization of the first layer in the network with the principal components, and its two variants PCsInit-Act and PCsInit-Sub. We will show that explanations using these strategies are more simple, direct and straightforward than using PCA prior to training a neural network on the principal components. We also show that the proposed techniques possess desirable theoretical properties. Moreover, as will be illustrated in the experiments, such training strategies can also allow further improvement of training via backpropagation compared to training neural networks on principal components.