Learning Invariant Representations for Sentiment Analysis: The Missing Material is Datasets
This work addresses the problem of evaluating and mitigating nuisance factors in NLP tasks like sentiment analysis, but it is incremental as it builds on existing datasets and methods.
The paper tackles the challenge of learning invariant representations in NLP by introducing two generalization metrics to assess model robustness to nuisance factors, and applies a data filtering method to create biased datasets, showing that a simple sentiment analysis baseline can be significantly affected by product ID as a nuisance factor.
Learning representations which remain invariant to a nuisance factor has a great interest in Domain Adaptation, Transfer Learning, and Fair Machine Learning. Finding such representations becomes highly challenging in NLP tasks since the nuisance factor is entangled in a raw text. To our knowledge, a major issue is also that only few NLP datasets allow assessing the impact of such factor. In this paper, we introduce two generalization metrics to assess model robustness to a nuisance factor: \textit{generalization under target bias} and \textit{generalization onto unknown}. We combine those metrics with a simple data filtering approach to control the impact of the nuisance factor on the data and thus to build experimental biased datasets. We apply our method to standard datasets of the literature (\textit{Amazon} and \textit{Yelp}). Our work shows that a simple text classification baseline (i.e., sentiment analysis on reviews) may be badly affected by the \textit{product ID} (considered as a nuisance factor) when learning the polarity of a review. The method proposed is generic and applicable as soon as the nuisance variable is annotated in the dataset.