Connecting Concept Convexity and Human-Machine Alignment in Deep Neural Networks
It addresses the problem of making AI more interpretable and reliable by exploring alignment with human cognition, but it is incremental as it presents a first step with complex and inconsistent results.
This study investigated the relationship between convexity in neural network representations and human-machine alignment using vision transformer models, finding a correlation but noting that increasing convexity through fine-tuning inconsistently affects alignment.
Understanding how neural networks align with human cognitive processes is a crucial step toward developing more interpretable and reliable AI systems. Motivated by theories of human cognition, this study examines the relationship between \emph{convexity} in neural network representations and \emph{human-machine alignment} based on behavioral data. We identify a correlation between these two dimensions in pretrained and fine-tuned vision transformer models. Our findings suggest that the convex regions formed in latent spaces of neural networks to some extent align with human-defined categories and reflect the similarity relations humans use in cognitive tasks. While optimizing for alignment generally enhances convexity, increasing convexity through fine-tuning yields inconsistent effects on alignment, which suggests a complex relationship between the two. This study presents a first step toward understanding the relationship between the convexity of latent representations and human-machine alignment.