Fine Tuning Large Language Models for Medicine: The Role and Importance of Direct Preference Optimization
This work addresses the underutilization of fine-tuning in medicine by providing guidance on when to use SFT vs. DPO, which is incremental but important for practitioners in the medical domain.
The study compared Supervised Fine Tuning (SFT) and Direct Preference Optimization (DPO) for fine-tuning large language models on five medical tasks, finding that SFT is sufficient for text classification, while DPO improves performance on more complex tasks like clinical reasoning, summarization, and clinical triage.
Large Language Model (LLM) fine tuning is underutilized in the field of medicine. Two of the most common methods of fine tuning are Supervised Fine Tuning (SFT) and Direct Preference Optimization (DPO), but there is little guidance informing users when to use either technique. In this investigation, we compare the performance of SFT and DPO for five common natural language tasks in medicine: Classification with text data, Classification with numeric data, Clinical Reasoning, Summarization, and Clinical Triage. We find that SFT alone is sufficient for Classification with text data, whereas DPO improves performance for the more complex tasks of Clinical Reasoning, Summarization and Clinical Triage. Our results establish the role and importance of DPO fine tuning within medicine, and consequently call attention to current software gaps that prevent widespread deployment of this technique.