Majority Rules: LLM Ensemble is a Winning Approach for Content Categorization
This provides a scalable and reliable solution for taxonomy-based classification, potentially reducing dependence on human expert labeling, though it is incremental as it builds on existing LLM methods.
The study tackled content categorization by introducing an ensemble framework using multiple large language models (eLLM), which improved F1-score by up to 65% over the best single model and achieved near human-expert-level performance on a dataset of 8,660 samples.
This study introduces an ensemble framework for unstructured text categorization using large language models (LLMs). By integrating multiple models, the ensemble large language model (eLLM) framework addresses common weaknesses of individual systems, including inconsistency, hallucination, category inflation, and misclassification. The eLLM approach yields a substantial performance improvement of up to 65\% in F1-score over the strongest single model. We formalize the ensemble process through a mathematical model of collective decision-making and establish principled aggregation criteria. Using the Interactive Advertising Bureau (IAB) hierarchical taxonomy, we evaluate ten state-of-the-art LLMs under identical zero-shot conditions on a human-annotated corpus of 8{,}660 samples. Results show that individual models plateau in performance due to the compression of semantically rich text into sparse categorical representations, while eLLM improves both robustness and accuracy. With a diverse consortium of models, eLLM achieves near human-expert-level performance, offering a scalable and reliable solution for taxonomy-based classification that may significantly reduce dependence on human expert labeling.