Hardware Acceleration of LLMs: A comprehensive survey and comparison
This is an incremental survey that addresses the problem of inconsistent benchmarking for researchers and engineers working on hardware acceleration of LLMs.
The paper tackles the challenge of fairly comparing hardware acceleration frameworks for Large Language Models (LLMs) by extrapolating performance and energy efficiency results to the same process technology, implementing parts of LLMs on FPGA chips to enable a qualitative and quantitative comparison.
Large Language Models (LLMs) have emerged as powerful tools for natural language processing tasks, revolutionizing the field with their ability to understand and generate human-like text. In this paper, we present a comprehensive survey of the several research efforts that have been presented for the acceleration of transformer networks for Large Language Models using hardware accelerators. The survey presents the frameworks that have been proposed and then performs a qualitative and quantitative comparison regarding the technology, the processing platform (FPGA, ASIC, In-Memory, GPU), the speedup, the energy efficiency, the performance (GOPs), and the energy efficiency (GOPs/W) of each framework. The main challenge in comparison is that every proposed scheme is implemented on a different process technology making hard a fair comparison. The main contribution of this paper is that we extrapolate the results of the performance and the energy efficiency on the same technology to make a fair comparison; one theoretical and one more practical. We implement part of the LLMs on several FPGA chips to extrapolate the results to the same process technology and then we make a fair comparison of the performance.