LLM & HPC:Benchmarking DeepSeek's Performance in High-Performance Computing Tasks
This work addresses the problem of assessing LLM capabilities in HPC code generation for researchers and practitioners, but it is incremental as it benchmarks an existing model on new data.
This paper evaluated DeepSeek's performance in generating code for HPC benchmark tasks, such as conjugate gradient solvers and parallel matrix multiplication, and found that while it produces functional code, it lags behind GPT-4 in scalability and execution efficiency.
Large Language Models (LLMs), such as GPT-4 and DeepSeek, have been applied to a wide range of domains in software engineering. However, their potential in the context of High-Performance Computing (HPC) much remains to be explored. This paper evaluates how well DeepSeek, a recent LLM, performs in generating a set of HPC benchmark codes: a conjugate gradient solver, the parallel heat equation, parallel matrix multiplication, DGEMM, and the STREAM triad operation. We analyze DeepSeek's code generation capabilities for traditional HPC languages like Cpp, Fortran, Julia and Python. The evaluation includes testing for code correctness, performance, and scaling across different configurations and matrix sizes. We also provide a detailed comparison between DeepSeek and another widely used tool: GPT-4. Our results demonstrate that while DeepSeek generates functional code for HPC tasks, it lags behind GPT-4, in terms of scalability and execution efficiency of the generated code.