Conformal Prediction and MLLM aided Uncertainty Quantification in Scene Graph Generation
This addresses the need for reliable uncertainty estimation in SGG, which is crucial for practical applications like autonomous systems, but it is incremental as it adapts existing methods.
The paper tackles the problem of uncertainty quantification in Scene Graph Generation (SGG) by introducing a Conformal Prediction framework that constructs well-calibrated prediction sets with statistical coverage guarantees, and it uses an MLLM-based post-processing strategy to select plausible scene graphs, improving overall SGG performance.
Scene Graph Generation (SGG) aims to represent visual scenes by identifying objects and their pairwise relationships, providing a structured understanding of image content. However, inherent challenges like long-tailed class distributions and prediction variability necessitate uncertainty quantification in SGG for its practical viability. In this paper, we introduce a novel Conformal Prediction (CP) based framework, adaptive to any existing SGG method, for quantifying their predictive uncertainty by constructing well-calibrated prediction sets over their generated scene graphs. These scene graph prediction sets are designed to achieve statistically rigorous coverage guarantees. Additionally, to ensure these prediction sets contain the most practically interpretable scene graphs, we design an effective MLLM-based post-processing strategy for selecting the most visually and semantically plausible scene graphs within these prediction sets. We show that our proposed approach can produce diverse possible scene graphs from an image, assess the reliability of SGG methods, and improve overall SGG performance.