DFPE: A Diverse Fingerprint Ensemble for Enhancing LLM Performance
This addresses the challenge of enhancing LLM performance in multi-faceted language understanding tasks, though it is incremental as it builds on existing ensemble techniques.
The paper tackles the problem of LLMs struggling in diverse or complex domains by proposing DFPE, an ensemble method that clusters models based on response fingerprints, filters underperforming ones, and assigns adaptive weights, resulting in a 3% overall accuracy improvement on the MMLU benchmark.
Large Language Models (LLMs) have shown remarkable capabilities across various natural language processing tasks but often struggle to excel uniformly in diverse or complex domains. We propose a novel ensemble method - Diverse Fingerprint Ensemble (DFPE), which leverages the complementary strengths of multiple LLMs to achieve more robust performance. Our approach involves: (1) clustering models based on response "fingerprints" patterns, (2) applying a quantile-based filtering mechanism to remove underperforming models at a per-subject level, and (3) assigning adaptive weights to remaining models based on their subject-wise validation accuracy. In experiments on the Massive Multitask Language Understanding (MMLU) benchmark, DFPE outperforms the best single model by 3% overall accuracy and 5% in discipline-level accuracy. This method increases the robustness and generalization of LLMs and underscores how model selection, diversity preservation, and performance-driven weighting can effectively address challenging, multi-faceted language understanding tasks.