Towards Task Understanding in Visual Settings
This work addresses the need for precise task understanding in visual settings, which is incremental as it builds on existing image captioning methods by incorporating task ontologies.
The paper tackles the problem of understanding real-world tasks in images, moving beyond literal scene descriptions to identify the exact task being undertaken, and proposes a framework combining CNNs with a hierarchical task ontology to generate task descriptions, achieving efficacy in detailed experiments.
We consider the problem of understanding real world tasks depicted in visual images. While most existing image captioning methods excel in producing natural language descriptions of visual scenes involving human tasks, there is often the need for an understanding of the exact task being undertaken rather than a literal description of the scene. We leverage insights from real world task understanding systems, and propose a framework composed of convolutional neural networks, and an external hierarchical task ontology to produce task descriptions from input images. Detailed experiments highlight the efficacy of the extracted descriptions, which could potentially find their way in many applications, including image alt text generation.