SuryaKiran at MEDIQA-Sum 2023: Leveraging LoRA for Clinical Dialogue Summarization
This work addresses resource-intensive fine-tuning for clinical applications, but it is incremental as it applies an existing method to a specific domain.
The paper tackled clinical dialogue summarization by evaluating Low Rank Adaptation (LoRA) as a parameter-efficient fine-tuning method, showing it performs on par with end-to-end fine-tuning for large language models in tasks from ImageCLEFmedical.
Finetuning Large Language Models helps improve the results for domain-specific use cases. End-to-end finetuning of large language models is time and resource intensive and has high storage requirements to store the finetuned version of the large language model. Parameter Efficient Fine Tuning (PEFT) methods address the time and resource challenges by keeping the large language model as a fixed base and add additional layers, which the PEFT methods finetune. This paper demonstrates the evaluation results for one such PEFT method Low Rank Adaptation (LoRA), for Clinical Dialogue Summarization. The evaluation results show that LoRA works at par with end-to-end finetuning for a large language model. The paper presents the evaluations done for solving both the Subtask A and B from ImageCLEFmedical {https://www.imageclef.org/2023/medical}