Bug In the Code Stack: Can LLMs Find Bugs in Large Python Code Stacks
This addresses the need to ensure LLMs can understand syntax for software development, though it is incremental as it builds on existing benchmarks for code-based evaluation.
The paper tackles the problem of evaluating Large Language Models (LLMs) in detecting syntax bugs within large Python code stacks, finding that code-based environments are more challenging than text-based ones, performance varies significantly among models, and longer context lengths correlate with degradation.
Recent research in Needle-in-a-Haystack (NIAH) benchmarks has explored the capabilities of Large Language Models (LLMs) in retrieving contextual information from large text documents. However, as LLMs become increasingly integrated into software development processes, it is crucial to evaluate their performance in code-based environments. As LLMs are further developed for program synthesis, we need to ensure that LLMs can understand syntax and write syntactically correct code. As a step in ensuring LLMs understand syntax, LLMs can be evaluated in their ability to find and detect syntax bugs. Our benchmark, Bug In The Code Stack (BICS), is designed to assess the ability of LLMs to identify simple syntax bugs within large source code. Our findings reveal three key insights: (1) code-based environments pose significantly more challenge compared to text-based environments for retrieval tasks, (2) there is a substantial performance disparity among different models, and (3) there is a notable correlation between longer context lengths and performance degradation, though the extent of this degradation varies between models.