Sensitivity Analysis On Loss Landscape
This work addresses sensitivity analysis for researchers in machine learning, but appears incremental as it builds on existing gradient-based methods.
The paper tackles the problem of understanding which independent variables impact dependent variables by performing sensitivity analysis on the loss landscape using first, second, and third derivatives through automatic differentiation. The result shows that second-order gradients can provide information similar to Spearman's rank correlation coefficient for visualizing relationships.
Gradients can be employed for sensitivity analysis. Here, we leverage the advantages of the Loss Landscape to comprehend which independent variables impact the dependent variable. We seek to grasp the loss landscape by utilizing first, second, and third derivatives through automatic differentiation. we know that Spearman's rank correlation coefficient can detect the monotonic relationship between two variables. However, I have found that second-order gradients, with certain configurations and parameters, provide information that can be visualized similarly to Spearman results, In this approach, we incorporate a loss function with an activation function, resulting in a non-linear pattern. Each exploration of the loss landscape through retraining yields new valuable information. Furthermore, the first and third derivatives are also beneficial, as they indicate the extent to which independent variables influence the dependent variable.