Point and Instruct: Enabling Precise Image Editing by Unifying Direct Manipulation and Text Instructions
This addresses the challenge for users in specifying exact image edits with text alone, offering a more intuitive and accurate editing tool, though it is incremental as it builds on existing text-based editing methods.
The paper tackles the problem of precise image editing by combining direct manipulation with text instructions, enabling users to visually select objects and locations while using natural language for transformations, resulting in a system that improves precision in scenarios like moving specific objects in cluttered images.
Machine learning has enabled the development of powerful systems capable of editing images from natural language instructions. However, in many common scenarios it is difficult for users to specify precise image transformations with text alone. For example, in an image with several dogs, it is difficult to select a particular dog and move it to a precise location. Doing this with text alone would require a complex prompt that disambiguates the target dog and describes the destination. However, direct manipulation is well suited to visual tasks like selecting objects and specifying locations. We introduce Point and Instruct, a system for seamlessly combining familiar direct manipulation and textual instructions to enable precise image manipulation. With our system, a user can visually mark objects and locations, and reference them in textual instructions. This allows users to benefit from both the visual descriptiveness of natural language and the spatial precision of direct manipulation.