Automatic Photo to Ideophone Manga Matching
This addresses the need for more engaging and communicative photo annotation tools for users, though it is incremental as it applies existing methods to a new application area.
The paper tackles the problem of automatically recommending and positioning ideophones (mimetic words) on photos for annotation, contrasting with traditional object-based systems. The result shows that in Japanese, these annotations are strongly preferred and increase enjoyment and sharing likelihood compared to unannotated or object-annotated photos.
Photo applications offer tools for annotation via text and stickers. Ideophones, mimetic and onomatopoeic words, which are common in graphic novels, have yet to be explored for photo annotation use. We present a method for automatic ideophone recommendation and positioning of the text on photos. These annotations are accomplished by obtaining a list of ideophones with English definitions and applying a suite of visual object detectors to the image. Next, a semantic embedding maps the visual objects to the possible relevant ideophones. Our system stands in contrast to traditional computer vision-based annotation systems, which stop at recommending object and scene-level annotation, by providing annotations that are communicative, fun, and engaging. We test these annotations in Japanese and find they carry a strong preference and increase enjoyment and sharing likelihood when compared to unannotated and object-based annotated photos.