Beyond ReLU: Chebyshev-DQN for Enhanced Deep Q-Networks
This work addresses performance limitations in reinforcement learning for tasks like CartPole, though it is incremental as it modifies an existing DQN framework with a specific basis function.
The paper tackled the problem of improving Deep Q-Networks (DQN) by introducing Chebyshev-DQN (Ch-DQN), which integrates Chebyshev polynomials for better feature representation, resulting in approximately 39% higher asymptotic performance on the CartPole-v1 benchmark compared to a standard DQN.
The performance of Deep Q-Networks (DQN) is critically dependent on the ability of its underlying neural network to accurately approximate the action-value function. Standard function approximators, such as multi-layer perceptrons, may struggle to efficiently represent the complex value landscapes inherent in many reinforcement learning problems. This paper introduces a novel architecture, the Chebyshev-DQN (Ch-DQN), which integrates a Chebyshev polynomial basis into the DQN framework to create a more effective feature representation. By leveraging the powerful function approximation properties of Chebyshev polynomials, we hypothesize that the Ch-DQN can learn more efficiently and achieve higher performance. We evaluate our proposed model on the CartPole-v1 benchmark and compare it against a standard DQN with a comparable number of parameters. Our results demonstrate that the Ch-DQN with a moderate polynomial degree (N=4) achieves significantly better asymptotic performance, outperforming the baseline by approximately 39\%. However, we also find that the choice of polynomial degree is a critical hyperparameter, as a high degree (N=8) can be detrimental to learning. This work validates the potential of using orthogonal polynomial bases in deep reinforcement learning while also highlighting the trade-offs involved in model complexity.