Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Task
This addresses a decision-making challenge for domains like medicine or robotics where acquiring multimodal data is costly, though it is incremental as it extends prior active feature acquisition to temporal settings.
The paper tackles the problem of actively selecting which input modalities to acquire at test time to balance cost and predictive performance, introducing the A2MT task and showing that agents can learn cost-reactive behavior on real-world datasets like Kinetics-700 and AudioSet, but fail to adapt strategies adaptively.
We introduce a challenging decision-making task that we call active acquisition for multimodal temporal data (A2MT). In many real-world scenarios, input features are not readily available at test time and must instead be acquired at significant cost. With A2MT, we aim to learn agents that actively select which modalities of an input to acquire, trading off acquisition cost and predictive performance. A2MT extends a previous task called active feature acquisition to temporal decision making about high-dimensional inputs. We propose a method based on the Perceiver IO architecture to address A2MT in practice. Our agents are able to solve a novel synthetic scenario requiring practically relevant cross-modal reasoning skills. On two large-scale, real-world datasets, Kinetics-700 and AudioSet, our agents successfully learn cost-reactive acquisition behavior. However, an ablation reveals they are unable to learn adaptive acquisition strategies, emphasizing the difficulty of the task even for state-of-the-art models. Applications of A2MT may be impactful in domains like medicine, robotics, or finance, where modalities differ in acquisition cost and informativeness.