NLD-LLM: A systematic framework for evaluating small language transformer models on natural language description
This work addresses the need for systematic evaluation frameworks in NLP for developers and researchers, but it is incremental as it applies existing methods to a new task.
The paper tackled the problem of evaluating small language transformer models on generating source code descriptions from natural language inputs, and found that prompt engineering significantly improves performance, with smaller models often competing effectively when using well-crafted prompts.
Natural Language Description (NLD) is a Natural Language Processing (NLP) task that requires models to generate structured and meaningful outputs from natural language inputs. In this work, we propose NLD-LLM, a systematic NLP framework to evaluate the performance of language models to generate accurate and concise source code descriptions. This framework incorporates a diverse set of transformer models, including Qwen, DeepSeek, Phi, LLaMA, and Mistral, spanning various sizes, architectures, and training approaches. Central to NLD-LLM is a comprehensive prompt design strategy that includes standardized formatting, clear task guidance, and NLD prompting, ensuring fair and consistent evaluation. Additionally, we apply an iterative refinement process to improve output's quality and assess the model's adaptability. Using semantic and structural metrics, our analysis demonstrates that prompt engineering significantly impacts the effectiveness of the model such that smaller models often performing competitively when supported by well-crafted prompts.