Charm: The Missing Piece in ViT fine-tuning for Image Aesthetic Assessment
This addresses a domain-specific bottleneck in image aesthetic assessment by enabling ViTs to handle variable-sized inputs more effectively without cropping or aspect ratio changes.
The paper tackles the problem of information loss in Vision Transformers (ViTs) when processing variable-sized images for aesthetic assessment by introducing Charm, a tokenization approach that preserves composition, high-resolution details, aspect ratio, and multi-scale information, resulting in performance improvements of up to 8.1% on various datasets.
The capacity of Vision transformers (ViTs) to handle variable-sized inputs is often constrained by computational complexity and batch processing limitations. Consequently, ViTs are typically trained on small, fixed-size images obtained through downscaling or cropping. While reducing computational burden, these methods result in significant information loss, negatively affecting tasks like image aesthetic assessment. We introduce Charm, a novel tokenization approach that preserves Composition, High-resolution, Aspect Ratio, and Multi-scale information simultaneously. Charm prioritizes high-resolution details in specific regions while downscaling others, enabling shorter fixed-size input sequences for ViTs while incorporating essential information. Charm is designed to be compatible with pre-trained ViTs and their learned positional embeddings. By providing multiscale input and introducing variety to input tokens, Charm improves ViT performance and generalizability for image aesthetic assessment. We avoid cropping or changing the aspect ratio to further preserve information. Extensive experiments demonstrate significant performance improvements on various image aesthetic and quality assessment datasets (up to 8.1 %) using a lightweight ViT backbone. Code and pre-trained models are available at https://github.com/FBehrad/Charm.