Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models
This work addresses the problem of slow inference speeds for LLM users by proposing an incremental optimization to reduce computational overhead.
The paper tackles the challenge of high inference latency in large language models (LLMs) by investigating the effect of dropping MLP and attention layers at inference time, finding that removing 33% of attention layers in a 13B Llama2 model results in only a 1.8% drop in average performance on the OpenLLM benchmark.
The inference demand for LLMs has skyrocketed in recent months, and serving models with low latencies remains challenging due to the quadratic input length complexity of the attention layers. In this work, we investigate the effect of dropping MLP and attention layers at inference time on the performance of Llama-v2 models. We find that dropping dreeper attention layers only marginally decreases performance but leads to the best speedups alongside dropping entire layers. For example, removing 33\% of attention layers in a 13B Llama2 model results in a 1.8\% drop in average performance over the OpenLLM benchmark. We also observe that skipping layers except the latter layers reduces performances for more layers skipped, except for skipping the attention layers.