APEX: Probing Neural Networks via Activation Perturbation
This work addresses the limitation of existing probing methods for neural networks, offering a novel approach to understand model behavior beyond input-space analysis, which is incremental but provides practical insights for researchers and practitioners in machine learning.
The paper tackles the problem of probing neural networks by introducing APEX, an inference-time method that perturbs hidden activations to access structural information in intermediate representations, demonstrating advantages such as measuring sample regularity, distinguishing structured models, and exposing training-induced biases like in backdoored models.
Prior work on probing neural networks primarily relies on input-space analysis or parameter perturbation, both of which face fundamental limitations in accessing structural information encoded in intermediate representations. We introduce Activation Perturbation for EXploration (APEX), an inference-time probing paradigm that perturbs hidden activations while keeping both inputs and model parameters fixed. We theoretically show that activation perturbation induces a principled transition from sample-dependent to model-dependent behavior by suppressing input-specific signals and amplifying representation-level structure, and further establish that input perturbation corresponds to a constrained special case of this framework. Through representative case studies, we demonstrate the practical advantages of APEX. In the small-noise regime, APEX provides a lightweight and efficient measure of sample regularity that aligns with established metrics, while also distinguishing structured from randomly labeled models and revealing semantically coherent prediction transitions. In the large-noise regime, APEX exposes training-induced model-level biases, including a pronounced concentration of predictions on the target class in backdoored models. Overall, our results show that APEX offers an effective perspective for exploring, and understanding neural networks beyond what is accessible from input space alone.