Positional Encoding in the Context of Memristor-Based Analog Computation for Automatic Speech Recognition
This work addresses a practical bottleneck in memristor-based analog computation for neural models, offering specific improvements for automatic speech recognition.
The paper identifies that large output values from transformed positional encodings cause significant degradation in analog-to-digital conversion during memristor-based computation for automatic speech recognition. By adjusting ADC weight and precision bits, they reduce degradation by ~50% relative, and by removing encoding-related linear transformations, they achieve ~30% relative reduction when ADC cannot be modified.
Memristors provide a new chance for resource-efficient computation of neural models for natural language processing by enabling analog execution of vector-matrix-multiplication. Yet, computations on these devices are currently subject to larger distortion, both in weight programming and execution. In this work, we identify large output values of transformed positional encodings to cause major degradation within analog-to-digital conversion (ADC) as part of memristor-based computation. By adjusting the proportion of weight and precision bits of the ADC of specific memristor layers, we reduce the degradation of the execution by ~50% relative, while keeping the estimated energy consumption stable. Additionally, we investigate scenarios where the ADC cannot be modified. In that case the degradation can be reduced by ~30% relative after removing encoding-related linear transformations.