Evaluating LLMs and Prompting Strategies for Automated Hardware Diagnosis from Textual User-Reports
This work addresses automating hardware diagnosis for computer manufacturers to improve user experience, but it is incremental as it focuses on evaluating existing models and prompting strategies on a specific dataset.
This study tackled the problem of identifying faulty hardware components from ambiguous textual user reports, achieving an f1-score up to 0.76 by evaluating 27 open-source and 2 proprietary LLMs with four prompting strategies across 98,948 inferences.
Computer manufacturers offer platforms for users to describe device faults using textual reports such as "My screen is flickering". Identifying the faulty component from the report is essential for automating tests and improving user experience. However, such reports are often ambiguous and lack detail, making this task challenging. Large Language Models (LLMs) have shown promise in addressing such issues. This study evaluates 27 open-source models (1B-72B parameters) and 2 proprietary LLMs using four prompting strategies: Zero-Shot, Few-Shot, Chain-of-Thought (CoT), and CoT+Few-Shot (CoT+FS). We conducted 98,948 inferences, processing over 51 million input tokens and generating 13 million output tokens. We achieve f1-score up to 0.76. Results show that three models offer the best balance between size and performance: mistral-small-24b-instruct and two smaller models, llama-3.2-1b-instruct and gemma-2-2b-it, that offer competitive performance with lower VRAM usage, enabling efficient inference on end-user devices as modern laptops or smartphones with NPUs.