Alejandro Aparcedo

Alejandro Aparcedo, Akash Kumar, Aaryan Garg et al.

Existing benchmarks for Vision-Language Models (VLMs) primarily evaluate spatio-temporal understanding on simple single-action videos, closed attribute sets and restricted entity types, failing to capture the freeform, multi-action interactions between diverse entities which characterize real-world video understanding. Furthermore, the lack of a systematic framework for analyzing model failures across complementary spatio-temporal axes hinders comprehensive evaluation. To address these gaps, we introduce VISTA, a Video Interaction Spatio-Temporal Analysis benchmark designed for open-set, multi-entity and multi-action spatio-temporal understanding in VLMs. VISTA decomposes videos into interpretable entities, their associated actions, and relational dynamics, enabling multi-axis diagnostics and unified assessment of relational, spatial, and temporal understanding. Our benchmark integrates multiple datasets into a single interaction-aware taxonomy and comprises ~12K curated video-query pairs spanning diverse scenes and complexities. We systematically evaluate 11 state-of-the-art VLMs on VISTA, and break down aggregate performance across our taxonomy to reveal shortcomings and pronounced spatio-temporal biases obscured by traditional metrics. By providing detailed, taxonomy-driven diagnostics on a challenging dataset, VISTA offers a nuanced framework to guide advances in model design, pretraining strategies, and evaluation protocols. Overall, VISTA is the first, large-scale, interaction-aware diagnostic benchmark for spatio-temporal understanding in VLMs.

Alejandro Aparcedo, Christian Lopez, Abhinav Kotta et al.

Extreme weather events are increasingly common due to climate change, posing significant risks. To mitigate further damage, a shift towards renewable energy is imperative. Unfortunately, underrepresented communities that are most affected often receive infrastructure improvements last. We propose a novel visual spatiotemporal framework for predicting nighttime lights (NTL), power outage severity and location before and after major hurricanes. Central to our solution is the Visual-Spatiotemporal Graph Neural Network (VST-GNN), to learn spatial and temporal coherence from images. Our work brings awareness to underrepresented areas in urgent need of enhanced energy infrastructure, such as future photovoltaic (PV) deployment. By identifying the severity and localization of power outages, our initiative aims to raise awareness and prompt action from policymakers and community stakeholders. Ultimately, this effort seeks to empower regions with vulnerable energy infrastructure, enhancing resilience and reliability for at-risk communities.

Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang et al.

Recent advances in instruction tuning have led to the development of State-of-the-Art Large Multimodal Models (LMMs). Given the novelty of these models, the impact of visual adversarial attacks on LMMs has not been thoroughly examined. We conduct a comprehensive study of the robustness of various LMMs against different adversarial attacks, evaluated across tasks including image classification, image captioning, and Visual Question Answer (VQA). We find that in general LMMs are not robust to visual adversarial inputs. However, our findings suggest that context provided to the model via prompts, such as questions in a QA pair helps to mitigate the effects of visual adversarial inputs. Notably, the LMMs evaluated demonstrated remarkable resilience to such attacks on the ScienceQA task with only an 8.10% drop in performance compared to their visual counterparts which dropped 99.73%. We also propose a new approach to real-world image classification which we term query decomposition. By incorporating existence queries into our input prompt we observe diminished attack effectiveness and improvements in image classification accuracy. This research highlights a previously under-explored facet of LMM robustness and sets the stage for future work aimed at strengthening the resilience of multimodal systems in adversarial environments.