FinNuE: Exposing the Risks of Using BERTScore for Numerical Semantic Evaluation in Finance
This exposes a critical limitation for financial NLP applications where numerical precision is essential, highlighting risks in using existing metrics and motivating new evaluation frameworks.
The paper tackled the problem of BERTScore's low sensitivity to numerical variation in financial texts, demonstrating that it fails to distinguish semantically critical differences like a 2% gain versus a 20% loss, which can lead to high similarity scores for financially divergent pairs.
BERTScore has become a widely adopted metric for evaluating semantic similarity between natural language sentences. However, we identify a critical limitation: BERTScore exhibits low sensitivity to numerical variation, a significant weakness in finance where numerical precision directly affects meaning (e.g., distinguishing a 2% gain from a 20% loss). We introduce FinNuE, a diagnostic dataset constructed with controlled numerical perturbations across earnings calls, regulatory filings, social media, and news articles. Using FinNuE, demonstrate that BERTScore fails to distinguish semantically critical numerical differences, often assigning high similarity scores to financially divergent text pairs. Our findings reveal fundamental limitations of embedding-based metrics for finance and motivate numerically-aware evaluation frameworks for financial NLP.