RoboCAS: A Benchmark for Robotic Manipulation in Complex Object Arrangement Scenarios
This addresses the problem of deploying foundation models in practical robotic manipulation for researchers, though it is incremental as it builds on existing benchmark concepts.
The paper tackles the lack of benchmarks for robotic manipulation in complex object arrangement scenarios by introducing RoboCAS, a benchmark that tests agents' abilities in long-horizon planning and spatial reasoning, revealing limitations in existing models.
Foundation models hold significant potential for enabling robots to perform long-horizon general manipulation tasks. However, the simplicity of tasks and the uniformity of environments in existing benchmarks restrict their effective deployment in complex scenarios. To address this limitation, this paper introduces the \textit{RoboCAS} benchmark, the first benchmark specifically designed for complex object arrangement scenarios in robotic manipulation. This benchmark employs flexible and concise scripted policies to efficiently collect a diverse array of demonstrations, showcasing scattered, orderly, and stacked object arrangements within a highly realistic physical simulation environment. It includes complex processes such as target retrieval, obstacle clearance, and robot manipulation, testing agents' abilities to perform long-horizon planning for spatial reasoning and predicting chain reactions under ambiguous instructions. Extensive experiments on multiple baseline models reveal their limitations in managing complex object arrangement scenarios, underscoring the urgent need for intelligent agents capable of performing long-horizon operations in practical deployments and providing valuable insights for future research directions. Project website: \url{https://github.com/notFoundThisPerson/RoboCAS-v0}.