Hypergraph Vision Transformers: Images are More than Nodes, More than Edges
This work addresses computational bottlenecks in graph-based vision models for researchers and practitioners in computer vision, though it appears incremental as it builds on existing transformer and graph neural network frameworks.
The paper tackled the challenge of modeling higher-order relationships in vision tasks while balancing adaptability and computational efficiency, proposing the Hypergraph Vision Transformer (HgVT) which achieved strong performance on image classification and retrieval.
Recent advancements in computer vision have highlighted the scalability of Vision Transformers (ViTs) across various tasks, yet challenges remain in balancing adaptability, computational efficiency, and the ability to model higher-order relationships. Vision Graph Neural Networks (ViGs) offer an alternative by leveraging graph-based methodologies but are hindered by the computational bottlenecks of clustering algorithms used for edge generation. To address these issues, we propose the Hypergraph Vision Transformer (HgVT), which incorporates a hierarchical bipartite hypergraph structure into the vision transformer framework to capture higher-order semantic relationships while maintaining computational efficiency. HgVT leverages population and diversity regularization for dynamic hypergraph construction without clustering, and expert edge pooling to enhance semantic extraction and facilitate graph-based image retrieval. Empirical results demonstrate that HgVT achieves strong performance on image classification and retrieval, positioning it as an efficient framework for semantic-based vision tasks.