Preliminary Tests of the Anticipatory Classifier System with Hindsight Experience Replay
This addresses performance stagnation in Learning Classifier Systems for sparse-reward environments, but it is incremental as it combines existing mechanisms.
The paper tackles the problem of sparse rewards in Anticipatory Classifier Systems by integrating Hindsight Experience Replay, resulting in significantly accelerated knowledge acquisition and environmental mastery on benchmarks like Maze 6 and FrozenLake, though with increased computational overhead and classifier numerosity.
This paper introduces ACS2HER, a novel integration of the Anticipatory Classifier System (ACS2) with the Hindsight Experience Replay (HER) mechanism. While ACS2 is highly effective at building cognitive maps through latent learning, its performance often stagnates in environments characterized by sparse rewards. We propose a specific architectural variant that triggers hindsight learning when the agent fails to reach its primary goal, re-labeling visited states as virtual goals to densify the learning signal. The proposed model was evaluated on two benchmarks: the deterministic \texttt{Maze 6} and the stochastic \texttt{FrozenLake}. The results demonstrate that ACS2HER significantly accelerates knowledge acquisition and environmental mastery compared to the standard ACS2. However, this efficiency gain is accompanied by increased computational overhead and a substantial expansion in classifier numerosity. This work provides the first analysis of combining anticipatory mechanisms with retrospective goal-relabeling in Learning Classifier Systems.