Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling
This addresses a bottleneck in the open-source LLM community for improving model accuracy and efficiency, though it is an incremental advance over existing ensembling methods.
The paper tackles the problem of underutilizing token-level probability information in ensembling large language models (LLMs) by treating token generation as classification, which prevents early errors and improves performance; experiments on benchmarks like exams and mathematics show it breaks existing performance ceilings and achieves better results with lower latency by ensembling only key tokens.
Ensembling multiple models has always been an effective approach to push the limits of existing performance and is widely used in classification tasks by simply averaging the classification probability vectors from multiple classifiers to achieve better accuracy. However, in the thriving open-source Large Language Model (LLM) community, ensembling methods are rare and typically limited to ensembling the full-text outputs of LLMs, such as selecting the best output using a ranker, which leads to underutilization of token-level probability information. In this paper, we treat the Generation of each token by LLMs as a Classification (GaC) for ensembling. This approach fully exploits the probability information at each generation step and better prevents LLMs from producing early incorrect tokens that lead to snowballing errors. In experiments, we ensemble state-of-the-art LLMs on several benchmarks, including exams, mathematics and reasoning, and observe that our method breaks the existing community performance ceiling. Furthermore, we observed that most of the tokens in the answer are simple and do not affect the correctness of the final answer. Therefore, we also experimented with ensembling only key tokens, and the results showed better performance with lower latency across benchmarks.