Multimodal Prediction based on Graph Representations
This addresses multimodal prediction problems for researchers and practitioners in AI/ML, offering a novel fusion method that is incremental over existing fusion techniques.
The paper tackles multimodal prediction tasks by proposing a rank-fusion graph learning model that encodes multiple descriptors and retrieval models into graphs, projecting them into vector spaces for fusion, and shows it outperforms early-fusion and late-fusion alternatives in experiments across various datasets.
This paper proposes a learning model, based on rank-fusion graphs, for general applicability in multimodal prediction tasks, such as multimodal regression and image classification. Rank-fusion graphs encode information from multiple descriptors and retrieval models, thus being able to capture underlying relationships between modalities, samples, and the collection itself. The solution is based on the encoding of multiple ranks for a query (or test sample), defined according to different criteria, into a graph. Later, we project the generated graph into an induced vector space, creating fusion vectors, targeting broader generality and efficiency. A fusion vector estimator is then built to infer whether a multimodal input object refers to a class or not. Our method is capable of promoting a fusion model better than early-fusion and late-fusion alternatives. Performed experiments in the context of multiple multimodal and visual datasets, as well as several descriptors and retrieval models, demonstrate that our learning model is highly effective for different prediction scenarios involving visual, textual, and multimodal features, yielding better effectiveness than state-of-the-art methods.