CMT-Bench: Cricket Multi-Table Generation Benchmark for Probing Robustness in Large Language Models
This work addresses robustness issues in LLMs for domain-specific tasks like sports summarization, though it is incremental as it focuses on benchmarking rather than proposing new methods.
The paper tackles the problem of brittleness in large language models for dynamic text-to-table generation by introducing CMT-Bench, a diagnostic benchmark from cricket commentary, and finds that models show large accuracy drops under extractive-cue ablation, temporal prefixing, and entity-form perturbations.
LLM Driven text-to-table (T2T) systems often rely on extensive prompt-engineering or iterative event extraction in code-parsable formats, which boosts scores but are computationally expensive and obscure how models actually reason over temporal evolving narratives to summarise key information. We present CMT-Bench, a diagnostic benchmark built from live cricket commentary that requires dynamic table generation across two evolving schemas under a dense, rule-governed policy. CMT-Bench is designed to probe robustness via three semantics-preserving dimensions: (i) extractive-cue ablation to separate extractive shortcuts from state tracking, (ii) temporal prefixing to test long-context stability, and (iii) entity-form perturbations (anonymization, outof-distribution substitutions, role-entangling paraphrases) to assess sensitivity to surface variation. Across diverse long-context stateof-the-art LLMs, we find large drops without extractive summaries, monotonic degradation with input length, and consistent accuracy drop under entity-form changes. Complementary distributional tests confirm significant shifts in numeric error patterns, indicating drift in reasoning rather than mere noise. Our results show that current LLMs are brittle in dynamic Textto-table generation, motivating robustness-first evaluation as a prerequisite for developing efficient and scalable approaches for this task.