YOLOA: Real-Time Affordance Detection via LLM Adapter
This work addresses the 'what-where-how' challenge for embodied AI systems, offering a real-time solution that integrates object detection and affordance learning, though it appears incremental as it builds on existing YOLO and LLM adapter methods.
The paper tackles the problem of affordance detection in embodied AI by jointly addressing object detection and affordance learning, introducing YOLOA which achieves state-of-the-art accuracy (52.8 mAP on ADG-Det, 73.1 mAP on IIT-Heat) and real-time performance (up to 89.77 FPS).
Affordance detection aims to jointly address the fundamental "what-where-how" challenge in embodied AI by understanding "what" an object is, "where" the object is located, and "how" it can be used. However, most affordance learning methods focus solely on "how" objects can be used while neglecting the "what" and "where" aspects. Other affordance detection methods treat object detection and affordance learning as two independent tasks, lacking effective interaction and real-time capability. To overcome these limitations, we introduce YOLO Affordance (YOLOA), a real-time affordance detection model that jointly handles these two tasks via a large language model (LLM) adapter. Specifically, YOLOA employs a lightweight detector consisting of object detection and affordance learning branches refined through the LLM Adapter. During training, the LLM Adapter interacts with object and affordance preliminary predictions to refine both branches by generating more accurate class priors, box offsets, and affordance gates. Experiments on our relabeled ADG-Det and IIT-Heat benchmarks demonstrate that YOLOA achieves state-of-the-art accuracy (52.8 / 73.1 mAP on ADG-Det / IIT-Heat) while maintaining real-time performance (up to 89.77 FPS, and up to 846.24 FPS for the lightweight variant). This indicates that YOLOA achieves an excellent trade-off between accuracy and efficiency.