Image Manipulation via Multi-Hop Instructions -- A New Dataset and Weakly-Supervised Neuro-Symbolic Approach
This work addresses image manipulation for AI applications, but it is incremental as it adapts an existing method (NSCL) from VQA to a new task.
The paper tackles the problem of image manipulation via natural language text, which requires complex multi-modal reasoning, by extending Neuro Symbolic Concept Learning (NSCL) to create NeuroSIM, a system that uses weak supervision from VQA data and achieves competitive or superior performance compared to state-of-the-art baselines using supervised data.
We are interested in image manipulation via natural language text -- a task that is useful for multiple AI applications but requires complex reasoning over multi-modal spaces. We extend recently proposed Neuro Symbolic Concept Learning (NSCL), which has been quite effective for the task of Visual Question Answering (VQA), for the task of image manipulation. Our system referred to as NeuroSIM can perform complex multi-hop reasoning over multi-object scenes and only requires weak supervision in the form of annotated data for VQA. NeuroSIM parses an instruction into a symbolic program, based on a Domain Specific Language (DSL) comprising of object attributes and manipulation operations, that guides its execution. We create a new dataset for the task, and extensive experiments demonstrate that NeuroSIM is highly competitive with or beats SOTA baselines that make use of supervised data for manipulation.