RECSIP: REpeated Clustering of Scores Improving the Precision
This addresses the problem of unreliable LLM responses for users in high-risk or industrial contexts, representing an incremental improvement.
The paper tackles the lack of reliability in Large Language Models (LLMs) by introducing RECSIP, a framework that improves precision through parallel querying, scoring, and clustering of responses, resulting in a 5.8 percentage point increase on the MMLU-Pro benchmark compared to the best individual model.
The latest research on Large Language Models (LLMs) has demonstrated significant advancement in the field of Natural Language Processing (NLP). However, despite this progress, there is still a lack of reliability in these models. This is due to the stochastic architecture of LLMs, which presents a challenge for users attempting to ascertain the reliability of a model's response. These responses may cause serious harm in high-risk environments or expensive failures in industrial contexts. Therefore, we introduce the framework REpeated Clustering of Scores Improving the Precision (RECSIP) which focuses on improving the precision of LLMs by asking multiple models in parallel, scoring and clustering their responses to ensure a higher reliability on the response. The evaluation of our reference implementation recsip on the benchmark MMLU-Pro using the models GPT-4o, Claude and Gemini shows an overall increase of 5.8 per cent points compared to the best used model.