Rethinking Foundation Model Collaboration: Enhancing Specialized Models through Proxy Task Reasoning
For embodied AI systems, this provides a general principle for effectively combining foundation and specialized models without replacing the latter.
Foundation models struggle with precise geometric/numerical tasks where specialized models excel. The authors propose FAT, a framework where specialists generate hypotheses and foundation models perform proxy reasoning (e.g., selection/verification), achieving consistent improvements across 2D/3D detection, trajectory prediction, and segmentation at lower cost than direct foundation-model regression.
Foundation models are increasingly integrated into embodied intelligence systems, but directly assigning them structured prediction tasks requires precise geometric and numerical estimation, where specialized models often remain stronger. This capability mismatch raises a key question: should foundation models replace task-specific predictors, or should they collaborate through tasks better aligned with their strengths? We propose FAT, a foundation-model-augmented task-specific reasoning framework that treats collaboration as task decomposition rather than model replacement. FAT decomposes structured prediction into specialist prediction, information-space reconstruction, and foundation-model proxy reasoning. The specialist generates geometrically and physically valid hypotheses in the native output space, while the foundation model performs a bounded proxy task, such as selection or verification, over reconstructed multimodal candidates. We instantiate this principle as ProxySelect with a vision--language model. Across 2D object detection, 3D object detection, trajectory prediction, and semantic segmentation, ProxySelect consistently improves specialized baselines and substantially outperforms direct foundation-model regression at lower computational cost. These results suggest a general collaboration principle: specialized models preserve task-specific structure, while foundation models refine their hypotheses through contextual proxy reasoning.