Sentence Smith: Controllable Edits for Evaluating Text Embeddings
This work addresses the problem of opaque benchmarking in NLP by enabling fine-grained evaluation of text embedding models through controllable generation, though it is incremental as it builds on existing parsing and generation techniques.
The paper tackles the challenge of controllable text generation for evaluating text embeddings by proposing the Sentence Smith framework, which parses sentences into semantic graphs, applies manipulation rules, and generates text to create hard negative pairs, resulting in a resource-efficient method that produces high-quality texts validated by humans.
Controllable and transparent text generation has been a long-standing goal in NLP. Almost as long-standing is a general idea for addressing this challenge: Parsing text to a symbolic representation, and generating from it. However, earlier approaches were hindered by parsing and generation insufficiencies. Using modern parsers and a safety supervision mechanism, we show how close current methods come to this goal. Concretely, we propose the Sentence Smith framework for English, which has three steps: 1. Parsing a sentence into a semantic graph. 2. Applying human-designed semantic manipulation rules. 3. Generating text from the manipulated graph. A final entailment check (4.) verifies the validity of the applied transformation. To demonstrate our framework's utility, we use it to induce hard negative text pairs that challenge text embedding models. Since the controllable generation makes it possible to clearly isolate different types of semantic shifts, we can evaluate text embedding models in a fine-grained way, also addressing an issue in current benchmarking where linguistic phenomena remain opaque. Human validation confirms that our transparent generation process produces texts of good quality. Notably, our way of generation is very resource-efficient, since it relies only on smaller neural networks.