InstructBrush: Learning Attention-based Instruction Optimization for Image Editing
This addresses a bottleneck in image editing for users needing precise control over complex edits, though it is incremental as it builds on existing instruction-based methods.
The paper tackles the problem of instruction-based image editing methods struggling with tasks hard to describe in language by proposing InstructBrush, an inversion method that extracts editing instructions from exemplar image pairs, achieving superior performance with better semantic consistency in editing.
In recent years, instruction-based image editing methods have garnered significant attention in image editing. However, despite encompassing a wide range of editing priors, these methods are helpless when handling editing tasks that are challenging to accurately describe through language. We propose InstructBrush, an inversion method for instruction-based image editing methods to bridge this gap. It extracts editing effects from exemplar image pairs as editing instructions, which are further applied for image editing. Two key techniques are introduced into InstructBrush, Attention-based Instruction Optimization and Transformation-oriented Instruction Initialization, to address the limitations of the previous method in terms of inversion effects and instruction generalization. To explore the ability of instruction inversion methods to guide image editing in open scenarios, we establish a TransformationOriented Paired Benchmark (TOP-Bench), which contains a rich set of scenes and editing types. The creation of this benchmark paves the way for further exploration of instruction inversion. Quantitatively and qualitatively, our approach achieves superior performance in editing and is more semantically consistent with the target editing effects.