SparseDoctor: Towards Efficient Chat Doctor with Mixture of Experts Enhanced Large Language Models
This work addresses efficiency issues in medical LLMs for healthcare applications, though it appears incremental as it builds on existing methods like LoRA and MoE.
The paper tackles the high training cost of fine-tuning large language models for medical applications by introducing SparseDoctor, a sparse model with a contrastive learning-enhanced LoRA-MoE architecture, which outperforms strong baselines like HuatuoGPT on medical benchmarks such as CMB, CMExam, and CMMLU-Med.
Large language models (LLMs) have achieved great success in medical question answering and clinical decision-making, promoting the efficiency and popularization of the personalized virtual doctor in society. However, the traditional fine-tuning strategies on LLM require the updates of billions of parameters, substantially increasing the training cost, including the training time and utility cost. To enhance the efficiency and effectiveness of the current medical LLMs and explore the boundary of the representation capability of the LLMs on the medical domain, apart from the traditional fine-tuning strategies from the data perspective (i.e., supervised fine-tuning or reinforcement learning from human feedback), we instead craft a novel sparse medical LLM named SparseDoctor armed with contrastive learning enhanced LoRA-MoE (low rank adaptation-mixture of experts) architecture. To this end, the crafted automatic routing mechanism can scientifically allocate the computational resources among different LoRA experts supervised by the contrastive learning. Additionally, we also introduce a novel expert memory queue mechanism to further boost the efficiency of the overall framework and prevent the memory overflow during training. We conduct comprehensive evaluations on three typical medical benchmarks: CMB, CMExam, and CMMLU-Med. Experimental results demonstrate that the proposed LLM can consistently outperform the strong baselines such as the HuatuoGPT series.