Resilient Computing with Reinforcement Learning on a Dynamical System: Case Study in Sorting
This addresses the problem of resilient computing for robots and autonomous agents with limited resources, though it appears incremental by applying existing RL methods to a new computational domain.
The paper formulates general computation as a feedback-control problem using reinforcement learning to achieve resilience to errors and early termination, with a case study on array sorting showing the RL agent makes steady progress, is resilient to faults, and performs fewer array manipulations than Quicksort and Bubble sort.
Robots and autonomous agents often complete goal-based tasks with limited resources, relying on imperfect models and sensor measurements. In particular, reinforcement learning (RL) and feedback control can be used to help a robot achieve a goal. Taking advantage of this body of work, this paper formulates general computation as a feedback-control problem, which allows the agent to autonomously overcome some limitations of standard procedural language programming: resilience to errors and early program termination. Our formulation considers computation to be trajectory generation in the program's variable space. The computing then becomes a sequential decision making problem, solved with reinforcement learning (RL), and analyzed with Lyapunov stability theory to assess the agent's resilience and progression to the goal. We do this through a case study on a quintessential computer science problem, array sorting. Evaluations show that our RL sorting agent makes steady progress to an asymptotically stable goal, is resilient to faulty components, and performs less array manipulations than traditional Quicksort and Bubble sort.