UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI
This provides a comprehensive testing ground for embodied AI research, addressing the need for more realistic and varied virtual environments to develop systems for real-world deployment, though it is incremental as it builds on existing tools like UnrealCV.
The authors tackled the problem of limited realistic environments for embodied AI research by introducing UnrealZoo, a collection of over 100 photo-realistic 3D virtual worlds with diverse entities, which improved rendering efficiency and revealed that environmental diversity benefits generalizable reinforcement learning agents while highlighting persistent challenges in open-world scenarios.
We introduce UnrealZoo, a collection of over 100 photo-realistic 3D virtual worlds built on Unreal Engine, designed to reflect the complexity and variability of open-world environments. We also provide a rich variety of playable entities, including humans, animals, robots, and vehicles for embodied AI research. We extend UnrealCV with optimized APIs and tools for data collection, environment augmentation, distributed training, and benchmarking. These improvements achieve significant improvements in the efficiency of rendering and communication, enabling advanced applications such as multi-agent interactions. Our experimental evaluation across visual navigation and tracking tasks reveals two key insights: 1) environmental diversity provides substantial benefits for developing generalizable reinforcement learning (RL) agents, and 2) current embodied agents face persistent challenges in open-world scenarios, including navigation in unstructured terrain, adaptation to unseen morphologies, and managing latency in the close-loop control systems for interacting in highly dynamic objects. UnrealZoo thus serves as both a comprehensive testing ground and a pathway toward developing more capable embodied AI systems for real-world deployment.