Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces
This work addresses interpretability and efficiency issues in language models for AI researchers, but appears incremental as it builds on existing geometric and categorical approaches.
The paper tackles the polysemantic nature of neurons in AI language models by proposing a geometric definition of neurons as categorical vector spaces with non-orthogonal bases, enabling identification of a critical categorical zone for improved model efficiency.
The polysemantic nature of synthetic neurons in artificial intelligence language models is currently understood as the result of a necessary superposition of distributed features within the latent space. We propose an alternative approach, geometrically defining a neuron in layer n as a categorical vector space with a non-orthogonal basis, composed of categorical sub-dimensions extracted from preceding neurons in layer n-1. This categorical vector space is structured by the activation space of each neuron and enables, via an intra-neuronal attention process, the identification and utilization of a critical categorical zone for the efficiency of the language model - more homogeneous and located at the intersection of these different categorical sub-dimensions.