Distinguishing Scams and Fraud with Ensemble Learning
This work addresses a data gap for scam defense in LLM-enabled chatbots, but it is incremental as it applies existing methods to a new dataset.
The researchers tackled the problem of distinguishing scam from non-scam fraud in the Consumer Financial Protection Bureau's complaints database, using an LLM ensemble approach to evaluate LLM performance on user scam queries, with initial findings on strengths and weaknesses.
Users increasingly query LLM-enabled web chatbots for help with scam defense. The Consumer Financial Protection Bureau's complaints database is a rich data source for evaluating LLM performance on user scam queries, but currently the corpus does not distinguish between scam and non-scam fraud. We developed an LLM ensemble approach to distinguishing scam and fraud CFPB complaints and describe initial findings regarding the strengths and weaknesses of LLMs in the scam defense context.