NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language
This addresses the limitation of VLMs in handling novel problems through compositional reasoning, offering a flexible and training-free solution for domains requiring visual and language integration.
The paper tackled the problem of compositional reasoning in Vision-Language Models (VLMs) by introducing NePTune, a neuro-symbolic framework that translates natural language queries into executable Python programs, resulting in significant improvements over strong base models on multiple visual reasoning benchmarks.
Modern Vision-Language Models (VLMs) have achieved impressive performance in various tasks, yet they often struggle with compositional reasoning, the ability to decompose and recombine concepts to solve novel problems. While neuro-symbolic approaches offer a promising direction, they are typically constrained by crisp logical execution or predefined predicates, which limit flexibility. In this work, we introduce NePTune, a neuro-symbolic framework that overcomes these limitations through a hybrid execution model that integrates the perception capabilities of foundation vision models with the compositional expressiveness of symbolic reasoning. NePTune dynamically translates natural language queries into executable Python programs that blend imperative control flow with soft logic operators capable of reasoning over VLM-generated uncertainty. Operating in a training-free manner, NePTune, with a modular design, decouples perception from reasoning, yet its differentiable operations support fine-tuning. We evaluate NePTune on multiple visual reasoning benchmarks and various domains, utilizing adversarial tests, and demonstrate a significant improvement over strong base models, as well as its effective compositional generalization and adaptation capabilities in novel environments.