Visualizing Linguistic Diversity of Text Datasets Synthesized by Large Language Models
This tool helps researchers and practitioners analyze the linguistic diversity of LLM-synthesized datasets, addressing a known bottleneck in understanding failure modes, but it is incremental as it builds on existing visualization and clustering techniques.
The authors tackled the problem of evaluating LLM-generated text datasets, which can be repetitive in surprising ways, by developing LinguisticLens, an interactive visualization tool that clusters text along syntactic, lexical, and semantic axes, allowing users to quickly scan and inspect datasets.
Large language models (LLMs) can be used to generate smaller, more refined datasets via few-shot prompting for benchmarking, fine-tuning or other use cases. However, understanding and evaluating these datasets is difficult, and the failure modes of LLM-generated data are still not well understood. Specifically, the data can be repetitive in surprising ways, not only semantically but also syntactically and lexically. We present LinguisticLens, a novel inter-active visualization tool for making sense of and analyzing syntactic diversity of LLM-generated datasets. LinguisticLens clusters text along syntactic, lexical, and semantic axes. It supports hierarchical visualization of a text dataset, allowing users to quickly scan for an overview and inspect individual examples. The live demo is available at shorturl.at/zHOUV.