Analyzing Text Representations by Measuring Task Alignment
This work addresses a fundamental question in NLP for researchers and practitioners, but it is incremental as it builds on existing representation methods.
The paper tackled the problem of understanding what makes text representations effective for classification, hypothesizing that task alignment is key, and validated this by showing that their proposed alignment score explains classification performance.
Textual representations based on pre-trained language models are key, especially in few-shot learning scenarios. What makes a representation good for text classification? Is it due to the geometric properties of the space or because it is well aligned with the task? We hypothesize the second claim. To test it, we develop a task alignment score based on hierarchical clustering that measures alignment at different levels of granularity. Our experiments on text classification validate our hypothesis by showing that task alignment can explain the classification performance of a given representation.