WEST: Word Encoded Sequence Transducers
This addresses memory constraints in on-device training and inference, such as in federated learning and ASR, offering a solution that bridges the gap between larger unit and sub-unit models.
The paper tackles the memory bottleneck of embedding and softmax layers in large vocabulary models for on-device applications like federated learning and ASR by proposing WEST, an algorithm that encodes categorical features and output classes with sequences of sub-units, achieving significant compression without performance loss.
Most of the parameters in large vocabulary models are used in embedding layer to map categorical features to vectors and in softmax layer for classification weights. This is a bottle-neck in memory constraint on-device training applications like federated learning and on-device inference applications like automatic speech recognition (ASR). One way of compressing the embedding and softmax layers is to substitute larger units such as words with smaller sub-units such as characters. However, often the sub-unit models perform poorly compared to the larger unit models. We propose WEST, an algorithm for encoding categorical features and output classes with a sequence of random or domain dependent sub-units and demonstrate that this transduction can lead to significant compression without compromising performance. WEST bridges the gap between larger unit and sub-unit models and can be interpreted as a MaxEnt model over sub-unit features, which can be of independent interest.