Benchmarking Single-Factor Physical Video-to-Audio Generation
For researchers in video-to-audio generation, this work highlights the need to move beyond perceptual quality toward learning physical processes from pixels, though it is incremental in proposing a new evaluation benchmark.
The paper introduces FlatSounds, a benchmark to evaluate physical reasoning in video-to-audio models using controlled counterfactual pairs and single-video pattern tests. Results show models rely more on text captions than visuals for physics, improving accuracy but degrading temporal alignment.
Generative video-to-audio (V2A) models produce highly plausible soundtracks, but it remains unclear whether they capture the underlying physical processes. Existing evaluations emphasize perceptual realism and overlook physical correctness under controlled interventions. In this paper, we introduce FlatSounds, a benchmark that audits the physical reasoning of V2A models through: 1) controlled counterfactual pairs in which a single physical factor is varied, and 2) single-video pattern tests that probe internal consistency and directional trends. These settings test whether the generated audio correctly reflects specific physical properties and timings. Our evaluation of state-of-the-art models reveals a consistent trade-off: models rely more on text captions than the visual stream to infer physics and semantics. Captions generally improve physical and semantic accuracy, but paradoxically degrade temporal alignment. Our results highlight the need to move beyond audio quality toward learning physical processes directly from pixels. Finally, we find that our physics-based metrics correlate strongly with human preference tests on our own data. Project webpage: https://research.nvidia.com/labs/cosmos-lab/flatsounds/