Enhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations
This addresses inference efficiency for users of large language models, but it is incremental as it builds on existing compression methods.
The paper tackles the problem of high inference costs in large language models by investigating model compression, finding that skipping latter attention sublayers in Transformer LLMs leads to a 21% speed increase in one-token generation for Llama 2 7B while improving performance on benchmarks.
Large Language Models are growing in size, and we expect them to continue to do so, as larger models train quicker. However, this increase in size will severely impact inference costs. Therefore model compression is important, to retain the performance of larger models, but with a reduced cost of running them. In this thesis we explore the methods of model compression, and we empirically demonstrate that the simple method of skipping latter attention sublayers in Transformer LLMs is an effective method of model compression, as these layers prove to be redundant, whilst also being incredibly computationally expensive. We observed a 21% speed increase in one-token generation for Llama 2 7B, whilst surprisingly and unexpectedly improving performance over several common benchmarks.