Hierarchical Reinforcement Learning Framework for Stochastic Spaceflight Campaign Design
This work addresses the complex challenge of space mission design under uncertainty for space logistics and exploration planning, representing an incremental advancement by combining existing methods in a novel hierarchical structure.
The paper tackles the problem of designing multimission spaceflight campaigns under uncertainty by developing a hierarchical reinforcement learning framework that integrates reinforcement learning with mixed-integer linear programming, applied to human lunar exploration scenarios with uncertain resource utilization.
This paper develops a hierarchical reinforcement learning architecture for multimission spaceflight campaign design under uncertainty, including vehicle design, infrastructure deployment planning, and space transportation scheduling. This problem involves a high-dimensional design space and is challenging especially with uncertainty present. To tackle this challenge, the developed framework has a hierarchical structure with reinforcement learning and network-based mixed-integer linear programming (MILP), where the former optimizes campaign-level decisions (e.g., design of the vehicle used throughout the campaign, destination demand assigned to each mission in the campaign), whereas the latter optimizes the detailed mission-level decisions (e.g., when to launch what from where to where). The framework is applied to a set of human lunar exploration campaign scenarios with uncertain in situ resource utilization performance as a case study. The main value of this work is its integration of the rapidly growing reinforcement learning research and the existing MILP-based space logistics methods through a hierarchical framework to handle the otherwise intractable complexity of space mission design under uncertainty. This unique framework is expected to be a critical steppingstone for the emerging research direction of artificial intelligence for space mission design.