Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents
This addresses a gap in evaluating embodied agents for fine-grained physical interactions, which is crucial for developing more capable agents in real-world environments, though it is incremental as it builds on existing MLLM and benchmark frameworks.
The paper tackles the lack of benchmarks for fine-grained action intelligence in embodied agents by introducing CFG-Bench, a dataset with 1,368 videos and 19,562 question-answer pairs, revealing that leading MLLMs struggle with detailed physical interactions and higher-order reasoning, but supervised fine-tuning on this data yields significant performance gains on established benchmarks.
Multimodal Large Language Models (MLLMs) show promising results as decision-making engines for embodied agents operating in complex, physical environments. However, existing benchmarks often prioritize high-level planning or spatial reasoning, leaving the fine-grained action intelligence required for embodied physical interaction underexplored. To address this gap, we introduce CFG-Bench, a new benchmark designed to systematically evaluate this crucial capability. CFG-Bench consists of 1,368 curated videos paired with 19,562 three-modalities question-answer pairs targeting four cognitive abilities: 1) Physical Interaction, 2) Temporal-Causal Relation, 3) Intentional Understanding, and 4) Evaluative Judgment. Together, these dimensions provide a systematic framework for assessing a model's ability to translate visual observations into actionable knowledge, moving beyond mere surface-level recognition. Our comprehensive evaluation on CFG-Bench reveals that leading MLLMs struggle to produce detailed instructions for physical interactions and exhibit profound limitations in the higher-order reasoning of intention and evaluation. Moreover, supervised fine-tuning (SFT) on our data demonstrates that teaching an MLLMs to articulate fine-grained actions directly translates to significant performance gains on established embodied benchmarks. Our analysis highlights these limitations and offers insights for developing more capable and grounded embodied agents. Project page: \href{https://cfg-bench.github.io/}{https://cfg-bench.github.io/}.