Efficient Sparsely Activated Transformers
This work addresses inference efficiency for users of large Transformer models, offering a practical tool for latency optimization, though it is incremental as it builds on existing MoE and Transformer techniques.
The paper tackles the problem of high inference latency in Transformer-based networks by introducing PLANER, a system that optimizes existing networks with mixture-of-expert layers to meet user-defined latency targets while maintaining accuracy, achieving over 2x latency reduction at iso-accuracy on language modeling tasks.
Transformer-based neural networks have achieved state-of-the-art task performance in a number of machine learning domains including natural language processing and computer vision. To further improve their accuracy, recent work has explored the integration of dynamic behavior into these networks in the form of mixture-of-expert (MoE) layers. In this paper, we explore the introduction of MoE layers to optimize a different metric: inference latency. We introduce a novel system named PLANER that takes an existing Transformer-based network and a user-defined latency target and produces an optimized, sparsely-activated version of the original network that tries to meet the latency target while maintaining baseline accuracy. We evaluate PLANER on two real-world language modeling tasks using the Transformer-XL network and achieve inference latency reductions of over 2x at iso-accuracy.