AILA--First Experiments with Localist Language Models
This provides a practical framework for regulated domains requiring both transparency and capability in language models, though it appears incremental as it builds on existing transformer architectures.
The paper tackles the problem of balancing interpretability and performance in language models by introducing a transformer architecture with a tunable locality parameter that enables continuous control between localist and distributed representations. Results show that localist configurations achieve dramatically lower attention entropy (5.36 bits vs. 7.18 bits) while maintaining high pointer fidelity, with intermediate values optimizing the tradeoff (test perplexity 4.65, accuracy 84.7%).
This paper presents the first empirical demonstration of controllable locality in transformer language models, a novel architectural framework that enables continuous control over the degree of representation localization through a tunable locality dial parameter. Unlike traditional language models that rely exclusively on distributed representations, our approach allows dynamic interpolation between highly interpretable localist encodings and efficient distributed representations without requiring model retraining. We conducted experiments on the WikiText corpus using a two-layer transformer architecture, systematically varying the locality parameter λ across the full spectrum from 1.0 (fully localist) to 0.0 (fully distributed). Our results demonstrate that localist configurations achieve dramatically lower attention entropy, with λ = 1.0 yielding 5.36 bits compared to 7.18 bits at λ = 0.0, while maintaining substantially higher pointer fidelity scores reflecting stronger alignment with rule-specified targets. Prediction experiments reveal that intermediate locality values optimize the tradeoff between interpretability and performance, with λ = 0.6 achieving test perplexity of 4.65 and accuracy of 84.7%. These findings establish that localist language models provide a practical framework for applications in regulated domains requiring both transparency and capability, offering precise mathematical control over the interpretability-performance spectrum through explicit penalty thresholds and information-theoretic design principles.