Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing
This work addresses the need for comprehensive evaluation of LLMs in the financial domain, providing a foundation for developing more advanced models, though it is incremental as it focuses on benchmarking rather than introducing new methods.
The study evaluated the performance of large language models like ChatGPT on financial NLP tasks using the FinLMEval framework, finding that while some decoder-only models performed well with zero-shot prompting, they generally underperformed compared to fine-tuned expert models, particularly on proprietary datasets.
The emergence of Large Language Models (LLMs), such as ChatGPT, has revolutionized general natural language preprocessing (NLP) tasks. However, their expertise in the financial domain lacks a comprehensive evaluation. To assess the ability of LLMs to solve financial NLP tasks, we present FinLMEval, a framework for Financial Language Model Evaluation, comprising nine datasets designed to evaluate the performance of language models. This study compares the performance of encoder-only language models and the decoder-only language models. Our findings reveal that while some decoder-only LLMs demonstrate notable performance across most financial tasks via zero-shot prompting, they generally lag behind the fine-tuned expert models, especially when dealing with proprietary datasets. We hope this study provides foundation evaluations for continuing efforts to build more advanced LLMs in the financial domain.