An Algorithm for Routing Vectors in Sequences
This work addresses the challenge of efficient sequence processing for tasks like natural language and visual classification, with incremental improvements in optimization.
The authors tackled the problem of routing sequences of vectors to maximize 'bang per bit' by predicting input vectors, resulting in an algorithm that reduces parameter count, computation, and memory use by orders of magnitude and achieves competitive or state-of-the-art accuracy on natural language and visual classification tasks.
We propose a routing algorithm that takes a sequence of vectors and computes a new sequence with specified length and vector size. Each output vector maximizes "bang per bit," the difference between a net benefit to use and net cost to ignore data, by better predicting the input vectors. We describe output vectors as geometric objects, as latent variables that assign credit, as query states in a model of associative memory, and as agents in a model of a Society of Mind. We implement the algorithm with optimizations that reduce parameter count, computation, and memory use by orders of magnitude, enabling us to route sequences of greater length than previously possible. We evaluate our implementation on natural language and visual classification tasks, obtaining competitive or state-of-the-art accuracy and end-to-end credit assignments that are interpretable.