Towards Vector Optimization on Low-Dimensional Vector Symbolic Architecture
This work addresses efficiency and accuracy issues in machine learning for VSA applications, but it appears incremental as it builds on existing LDC methods.
The paper tackles the problem of hyperdimensionality and accuracy in Vector Symbolic Architecture by investigating Low-Dimensional Computing optimization, revealing that batch normalization and knowledge distillation boost accuracy and inference confidence without adding computational overhead during inference.
Vector Symbolic Architecture (VSA) is emerging in machine learning due to its efficiency, but they are hindered by issues of hyperdimensionality and accuracy. As a promising mitigation, the Low-Dimensional Computing (LDC) method significantly reduces the vector dimension by ~100 times while maintaining accuracy, by employing a gradient-based optimization. Despite its potential, LDC optimization for VSA is still underexplored. Our investigation into vector updates underscores the importance of stable, adaptive dynamics in LDC training. We also reveal the overlooked yet critical roles of batch normalization (BN) and knowledge distillation (KD) in standard approaches. Besides the accuracy boost, BN does not add computational overhead during inference, and KD significantly enhances inference confidence. Through extensive experiments and ablation studies across multiple benchmarks, we provide a thorough evaluation of our approach and extend the interpretability of binary neural network optimization similar to LDC, previously unaddressed in BNN literature.