Learning Robust Negation Text Representations
This addresses a specific semantic limitation in text encoders for NLP applications, but is incremental as it builds on existing contrastive learning and distillation techniques.
The paper tackles the problem of text encoders failing to properly capture negation semantics, which affects downstream applications relying on text embeddings, by proposing a strategy that distills data from large language models using diverse negation and hedging patterns to improve negation robustness. The result shows large improvement in negation understanding capabilities while maintaining competitive performance on general benchmarks, with the method also adaptable to LLMs for improved performance on negation benchmarks.
Despite rapid adoption of autoregressive large language models, smaller text encoders still play an important role in text understanding tasks that require rich contextualized representations. Negation is an important semantic function that is still not properly captured by such methods, affecting many downstream applications relying on text embeddings. We propose a strategy to improve negation robustness of text encoders, by distilling data from large language models using diverse patterns of negation and hedging. We adopt a standard contrastive learning strategy to finetune a strong BERT-based model, and observe large improvement in negation understanding capabilities while maintaining competitive performance on general benchmarks. In addition, we also show that our method can be adapted to LLMs, leading to improved performance on negation benchmarks.